+ All Categories
Home > Technology > Upgrading to Oracle 11gR2

Upgrading to Oracle 11gR2

Date post: 05-Dec-2014
Category:
Upload: syed-hussain
View: 3,992 times
Download: 1 times
Share this document with a friend
Description:
A step-by-step procedure to upgrade clusterware followed by the databases from 10gR2 to 11gR2
144
Do I actually need to care about upgrading my existing environment to race with Oracle latest release? Is it mandatory? Presented by : Syed Jaffer Hussain red-gate/AllThingsOracle Slide # 1
Transcript
Page 1: Upgrading to Oracle 11gR2

Do I actually need to care about upgrading my existing environment to race with

Oracle latest release

Is it mandatory

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 1

Does it really matter which version of Oracle do we use

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 2

Upgrading to Oracle 11gR2

A Successful case study

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 3

We really had a roller coaster journeyhellip

A successful upgrade

A successful downgrade

De-installed failed GI home before start over

Manual upgrade

Tried out of the box optionshellip

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 4

Disclaimer

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5

This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied

This material should not be reproduced or used without the authors written permission

Foreword

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6

The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7

Know your presenter

Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters

EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8

Know your presenter

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9

Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 2: Upgrading to Oracle 11gR2

Does it really matter which version of Oracle do we use

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 2

Upgrading to Oracle 11gR2

A Successful case study

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 3

We really had a roller coaster journeyhellip

A successful upgrade

A successful downgrade

De-installed failed GI home before start over

Manual upgrade

Tried out of the box optionshellip

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 4

Disclaimer

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5

This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied

This material should not be reproduced or used without the authors written permission

Foreword

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6

The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7

Know your presenter

Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters

EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8

Know your presenter

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9

Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 3: Upgrading to Oracle 11gR2

Upgrading to Oracle 11gR2

A Successful case study

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 3

We really had a roller coaster journeyhellip

A successful upgrade

A successful downgrade

De-installed failed GI home before start over

Manual upgrade

Tried out of the box optionshellip

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 4

Disclaimer

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5

This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied

This material should not be reproduced or used without the authors written permission

Foreword

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6

The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7

Know your presenter

Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters

EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8

Know your presenter

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9

Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 4: Upgrading to Oracle 11gR2

We really had a roller coaster journeyhellip

A successful upgrade

A successful downgrade

De-installed failed GI home before start over

Manual upgrade

Tried out of the box optionshellip

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 4

Disclaimer

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5

This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied

This material should not be reproduced or used without the authors written permission

Foreword

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6

The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7

Know your presenter

Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters

EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8

Know your presenter

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9

Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 5: Upgrading to Oracle 11gR2

Disclaimer

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 5

This viewscontent in this slides are those of the author and do not necessarily reflect that of Oracle Corporation andor its affiliatessubsidiaries The material in this document is for informational purposes only and is published with no guarantee or warranty express or implied

This material should not be reproduced or used without the authors written permission

Foreword

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6

The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7

Know your presenter

Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters

EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8

Know your presenter

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9

Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 6: Upgrading to Oracle 11gR2

Foreword

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 6

The objective of this presentation is to explain you the best practices that we had adoptedfollowedpracticed to upgrade successfully our FOUR Oracle 10204 clusterware environments with over 100 databases to Oracle 11g R2 on HPUX Servers

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7

Know your presenter

Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters

EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8

Know your presenter

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9

Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 7: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 7

Know your presenter

Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters

EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8

Know your presenter

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9

Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 8: Upgrading to Oracle 11gR2

Know your presenter

Syed Jaffer HussainDatabase Support ManagerOver 20 years IT experience12 years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i9i10g amp 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1R2 Real Application Clusters

EssentialsTwitter sjaffarhussainhttpjaffardbablogspotcom

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 8

Know your presenter

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9

Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 9: Upgrading to Oracle 11gR2

Know your presenter

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 9

Technologist of the Year DBA 2011httpwwworaclecomtechnetworkissue-archive201212-jano12awards-tech-1403083html

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 10: Upgrading to Oracle 11gR2

What will be covered

o WHY upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 10

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 11: Upgrading to Oracle 11gR2

Why upgrade ndash A burning question

Do we really need to care which version are we using

Is it mandatory to upgrade an existing environment with every Oracle new release

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 11

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 12: Upgrading to Oracle 11gR2

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 12

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 13: Upgrading to Oracle 11gR2

Why upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 13

To best of my knowledge the following situations may contributes to oppose an upgrade

Legacy application Doesnrsquot want to take the risk Lack of skills and support

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 14: Upgrading to Oracle 11gR2

Why upgrade

Situations that are likely to invite you to embrace for an upgrade

Organization standards Policy standards by Oracle for the

old releases Tantalizing new features Bug fixes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 14

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 15: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility MATRIXo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 15

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 16: Upgrading to Oracle 11gR2

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 16

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 17: Upgrading to Oracle 11gR2

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 17

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 18: Upgrading to Oracle 11gR2

Upgrade compatibility matrix

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 18

Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9208 or higher 112x

10105 or higher 112x

10202 or higher 112x

11106 or higher 112x

The following database version will require an indirect upgrade path

Source Database ---gt Upgrade Path for Target Database---gtTarget Database733 (or lower)-----gt 734 ---gt 9208 ----gt112x

805 (or lower)----gt 806 ---gt 9208 ----gt112x

817 (or lower)----gt 8174---gt 10204----gt112x

9013 (or lower)----gt 9014-- -gt10204----gt112x

9207(or lower)----gt9208----gt112x

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 19: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [BEST PRACTICE]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 19

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 20: Upgrading to Oracle 11gR2

Upgrade pattern [BEST PRACTICE]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 20

UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 21: Upgrading to Oracle 11gR2

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 21

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 22: Upgrading to Oracle 11gR2

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 22

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 23: Upgrading to Oracle 11gR2

Upgrade pattern [best practice]

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 23

UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 24: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview about our environmento Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 24

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 25: Upgrading to Oracle 11gR2

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 25

HP Superdome Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 656

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 26: Upgrading to Oracle 11gR2

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 26

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 27: Upgrading to Oracle 11gR2

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 27

8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

bull31 databasesbull51 instances

bull4 logical processorsbull22GB Memory

bull11 databasesbull20 instances

bull3 logical processorsbull12GB Memory

bull60 databasesbull80 instances

bull3 logical processorsbull16GB Memory

bull12 databasesbull20 instances

bull3 logical processorsbull12GB Memory

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 28: Upgrading to Oracle 11gR2

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 28

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

The framework that we have followed for the upgrade

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 29: Upgrading to Oracle 11gR2

Overview about our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 29

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupdate

The framework that we have followed for the upgrade

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 30: Upgrading to Oracle 11gR2

Overview of our environment

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 30

V1123varopt

V1131varopt

HP UX upgrade from V1123 TO v1131

EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade

The framework that we have followed for the upgrade

10204 11202

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 31: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 31

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 32: Upgrading to Oracle 11gR2

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 32

Preparation is the headstone to every successful installation orupgrade projects

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 33: Upgrading to Oracle 11gR2

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 33

Verifyo OS certification for Oracle 11gR2o OS kernel packageso Additional OS groupso Oracle base locationo GIH location

Must set GI home different from OB Configure Single Client Access Name(SCAN)

o Maximum of 3 IPrsquos (same across nodes)o DNS round robin resolutiono Must be on the subnet as Public and VIP IPrsquoso Not recommended to configure in HOSTS file

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 34: Upgrading to Oracle 11gR2

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 34

Spaceo Additional space for Oracle software

home typically u00o Enough tmp space

Back upo Back up the Cluster and Oracle homeso Back up Voting disko Back up OCR

CRS versiono Remember the crs active and software version

must be the samecrsctl query crs activeversion

crsctl query crs softwareversion

Nodes readinessruncluvfysh stage ndashpre crsinst ndashn nodes_list

ndashfixup ndashfixupdir ltdirnamegt -verbose

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 35: Upgrading to Oracle 11gR2

Prepare for upgrade ndash prerequisites checks

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 35

Verify nodes readiness for upgrade

While upgrading from pre 11gR2 use the following syntax

runcluvfysh stage ndashpre crsinst ndashn nodes_list ndashfixup ndashfixupdir ltdirnamegt -verbose

While upgrading 11201 to 11202 or higher use the following syntax

runcluvfysh stage ndashpre crsinst -upgrade ndashn nodes_list ndashrolling

-scr_crshome existing_GI_HOME ndashdest_crshome new_GI_HOME

-dest_version 112030

ndashfixup ndashfixupdirpath ltdirnamegt -verbose

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 36: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Preparing to upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 36

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 37: Upgrading to Oracle 11gR2

Clusterware ASM upgrade ndash Whatrsquos new

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 37

Grid Infrastructure

Out-of-Place andIn-place upgrades

New GI home

Clusterware

Automatic Storage Management (ASM)

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 38: Upgrading to Oracle 11gR2

Clusterware ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 38

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 39: Upgrading to Oracle 11gR2

Clusterware ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 39

Clusteware +

ASM = no rolling upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 40: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 40

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 41: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 41

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 42: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 42

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 43: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 43

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 44: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 44

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 45: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 45

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 46: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 46

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 47: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 47

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 48: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 48

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 49: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 49

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 50: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 50

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 51: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 51

ORACLE BASE must be different from the SW

locationhelliphellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 52: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 52

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 53: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 53

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 54: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 54

Cause False warning generated by unpublished CVU bug 10024549

Solution The warning can be ignored

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 55: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 55

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 56: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 56

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 57: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 57

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 58: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 58

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 59: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 59

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 60: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 60

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 61: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 61

Can I run the rootupgradesh script in PARALLEL across nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 62: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 62

Can I run the rootupgradesh script in PARALLEL across nodes

The answer is YES howeverhellip

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 63: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 63

Always run the script on the LOCAL node first

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 64: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 64

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 65: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 65

Always run the script on the LOCAL node first

Run the script in parallel on all nodes except the LAST node

Run the script on the LAST node

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 66: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 66

Shuts down the existing clusterware stack

Brings up the new version clusterware stack

Up on successfully running the script on the LAST node the CRS activerelease versionwill set to the new version

rootupgradesh

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 67: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 67

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 68: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 68

Average rootupgradesh execution time on each node was just about 15 min

Took more time on couple nodes mostly while stopping the existing cluster stack

Look closely for the logs under u00app1120gridcfgtoollogs

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 69: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 69

rootupgradesh

execution failed

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 70: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 70

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixMOS [ID 9692541]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnixOUI FailureGrid Infrastructure Cluster rootupgradesh FailureGrid Infrastructure Standalone rootupgradesh Failure OUI Plug-in FailureOld Clusterware Home Removal After UpgradePatching Pre-upgrade Clusterware HomeScalability RAC Community

References

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 71: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 71

Use the ndashforce flag to force the upgrade to complete in case some nodes becomesunreachable whilst performing an upgrade

u01app1120gridrootupgrade ndashforce (gt=11202 only)

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 72: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 72

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 73: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 73

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 74: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 74

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 75: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 75

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 76: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 76

Internet

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 77: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 77

Internet MOS

Cried for HELP

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 78: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 78

ASM alertlog trace files amp log files

Internet MOS

Cried for HELP

Doesnrsquot help much

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 79: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain Slide 79

Life Savior

MOS [ID 3983741]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 80: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 80

SQLgt ORACLE instance shut downSQLgt Disconnected

[PRSR][1]prsr_get_enabled_key OCR api procr_open_key failed for key DATABASEASMnode01+asm1ENABLED with 4[ PRSR][1]OCR error code 4[ PRSR][1]OCR error msg PROC-4 The cluster registry key to be operated on does not exist2009-06-23 161226332 [ RACG][1] [28830][1][oranode01ASM1asm] prsr_fetch_asm_config() error rc = 8

2009-06-23 161227443 [ RACG][1] [28830][1][oranode01ASM1asm]SQLPlus Release 102040 - Production on Tue Jun 23 161226 2009

Copyright (c) 1982 2007 Oracle All Rights Reserved

Enter user-name Connected to an idle instance

SQLgt ORA-01506 missing or illegal database nameSQLgt Disconnected

cd $ORA_ASM_HOMEloghostnameracgoralthostnamegtASMasmlog

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 81: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 81

What might have gone wrong

ASM instance name registered as ASMninstead of +ASMn in the OCR

Could be due to a manual entry

GI is very particular about + in contrast toprevious cluster version

Bug 5644904 WRONG ASM KEY PROC-4 THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 82: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 82

Remedy

Stop ASM and dependent instances

Remove dependency on ASM instance

Remove ASM from OCR

Add ASM back in with lsquo+rsquo correct name

Add dependency to ASM

Startup ASM

Resume the failed step

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 83: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 83

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 84: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 84

In action

srvctl stop asm ndashn lthostnamegt

srvctl modify instance ndashd DBNAME ndashi ltinstance_namegt -f

[do it for all active instances on the node]

srvctl remove asm ndashn lthostnamegt -i ltASM_INSTANCE1gt

srvctl add asm ndashn lthostnamegt -i lt+ASM_INSTANCE1gt

-o ltORA_ASM_HOMEgt

srvctl modify instance ndashd ltDBNAMEgt -i ltinstance_namegt

-s +ASM_INSTANCE1

[do it for all active instances on the node]

srvctl start asm ndashn lthostnamegt

Repeat on all nodes

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 85: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 85

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 86: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 86

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 87: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 87

Post upgrade checks ndash activesoftware version

crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [112020]

crsctl query crs releaseversion

Oracle High Availability Services release version on the local node is [112020]

Ensure activerelease version reflect to 11202

for a successful upgrade

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 88: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 88

Post upgrade checks ndash cluster stats across nodes

crsctl check cluster -all

node01CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode02CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is onlinenode03CRS-4537 Cluster Ready Services is onlineCRS-4529 Cluster Synchronization Services is onlineCRS-4533 Event Manager is online

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 89: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 89

Post upgrade checks ndash query voting disk

ocrcheck

Status of Oracle Cluster Registry is as follows

Version 3

Total space (kbytes) 1051708

Used space (kbytes) 27048

Available space (kbytes) 1024660

ID 1837247566

DeviceFile Name devrdskoracleocrln1ora_ocr_001

DeviceFile integrity check succeeded

DeviceFile Name devrdskoracleocrln1ora_ocr_002

DeviceFile integrity check succeeded

DeviceFile not configured

DeviceFile not configured

DeviceFile not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 90: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 90

Post upgrade checks ndash query voting disk

crsctl query css votedisk

STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1 ONLINE 3ee7c55614f24fdcfff130e1abf94384 (devrdskoraclevoteln1ora_vote_001) []

2 ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (devrdskoraclevoteln1ora_vote_002) []

3 ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (devrdskoraclevoteln1ora_vote_003) []

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 91: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 91

Post upgrade checks ndash bg and daemon processes

ps -ef |grep dbin

oracle 14458 1 0 151412 041 u00app1120gridbinoraagentbin

oracle 14462 1 0 151413 000 u00app1120gridbinmdnsdbin

root 14539 1 0 151428 024 u00app1120gridbincssdagent

oracle 14549 1 0 151429 001 u00app1120gridbindiskmonbin -d -f

root 14541 1 0 151428 417 u00app1120gridbinorarootagentbin

root 14732 1 0 151548 735 u00app1120gridbincrsdbin reboot

oracle 14469 1 0 151415 011 u00app1120gridbingpnpdbin

oracle 14739 1 0 151549 321 u00app1120gridbinevmdbin

oracle 14535 1 0 151425 251 u00app1120gridbingipcdbin

oracle 14545 1 0 151429 442 u00app1120gridbinocssdbin

root 20675 1 0 153603 428 u00app1120gridbinorarootagentbin

root 14595 1 0 151450 306 u00app1120gridbinoctssdbin

root 13999 1 0 151320 247 u00app1120gridbinohasdbin reboot

root 14533 1 0 151424 026 u00app1120gridbincssdmonitor

oracle 20671 1 0 153603 008 u00app1120gridbinappagentbin

oracle 20685 1 0 153603 026 u00app1120gridbinoraagentbin

ps -ef |grep initd

root 66 0 0 Feb 9 000 pagetable_init_daemon

oracle 8645 15234 0 193102 pts0 000 grep initd

root 13977 1 0 151319 000 binsh sbininitdinitohasd run

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 92: Upgrading to Oracle 11gR2

Clusterware and ASM upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 92

Remember

You can easily resumedowngrade from a failed or successful upgrade

Can fall back to the pre 11gR2 version

You hardly need a 3-step procedure to do that

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 93: Upgrading to Oracle 11gR2

Remove OLD homes

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 93

Removing OLD cluster home

Itrsquos a best practice to keep theOLD_HOME for a while after a successfulUpgrade The following procedure can be used to remove the OLD_HOMES

export ORACLE_HOME=$OLD_HOME

detach OLD_HOME

$OLD_HOMEouibinrunInstaller -detachHome

-silent -local ORACLE_HOME=$OLD_HOME

confirm $OLD_HOME is removed from central inventory

$OLD_HOMEOPatchopatch lsinventory -all

remove files in OLD_HOME manually

binrm -rf $OLD_HOME

unset ORACLE_HOME

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 94: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 94

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 95: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 95

cluvfy stage -pre dbinst -n

node01node02node03 -d

$ORACLE_HOME ndashverbose

Verify the READINESS of the nodes for DB software installation

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 96: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 96

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 97: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 97

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 98: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 98

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 99: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 99

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 100: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 100

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 101: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 101

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 102: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 102

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 103: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 103

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 104: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 104

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 105: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 105

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 106: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 106

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 107: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 107

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 108: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 108

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 109: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 109

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 110: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 110

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 111: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 111

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 112: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 112

How to manage the pre 11gR2 databases under 11gR2 GIH

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 113: Upgrading to Oracle 11gR2

11gR2 RDBMS software installation

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 113

rootnode01 u00app1120gridbin

crsctl pin css -n node01 node02 node03

CRS-4664 Node node01 successfully pinned

CRS-4664 Node node02 successfully pinned

CRS-4664 Node node03 successfully pinned

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 114: Upgrading to Oracle 11gR2

What will be covered

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 114

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 115: Upgrading to Oracle 11gR2

Database upgrade ndash Caution

Ensure a valid database backup BEFORE you kick off the db upgrade procedure

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 115

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 116: Upgrading to Oracle 11gR2

Database upgrade ndash upgrade paths

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 116

UPGRADE

SQLgt catupgradesql

bull RAC aware and most recommended for RAC dbsbull Also can be run in ndashsilent mode

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 117: Upgrading to Oracle 11gR2

Database upgrade

For a smooth rise follow the best practice run OH_11Grdbmsadminutlu112isql

Tablespaces [make adjustments in the current environment]

--gt SYSTEM tablespace is adequate for the upgrade

minimum required size 788 MB

--gt UNDOTBS1 tablespace is adequate for the upgrade

minimum required size 43 MB

--gt SYSAUX tablespace is adequate for the upgrade

minimum required size 566 MB

--gt TEMP tablespace is adequate for the upgrade

minimum required size 61 MB

PURGE DBA_RECYCLEBIN

must be executed immediately prior to executing your upgrade

Recommendations

Oracle recommends gathering dictionary statistics prior to

upgrading the database

To gather dictionary statistics execute the following command

while connected as SYSDBA

EXECUTE dbms_statsgather_dictionary_stats

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 117

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 118: Upgrading to Oracle 11gR2

Database upgrade

As a best practice run OH_11Grdbmsadminutlu112isql

Components [The following database components will be upgraded or installed]

--gt Oracle Catalog Views [upgrade] VALID

--gt Oracle Packages and Types [upgrade] VALID

--gt JServer JAVA Virtual Machine [upgrade] VALID

--gt Oracle XDK for Java [upgrade] VALID

--gt Real Application Clusters [upgrade] VALID

--gt Oracle Workspace Manager [upgrade] VALID

--gt OLAP Analytic Workspace [upgrade] VALID

--gt OLAP Catalog [upgrade] VALID

--gt EM Repository [upgrade] VALID

--gt Oracle Text [upgrade] VALID

--gt Oracle XML Database [upgrade] VALID

--gt Oracle Java Packages [upgrade] VALID

--gt Oracle interMedia [upgrade] VALID

--gt Spatial [upgrade] VALID

--gt Data Mining [upgrade] VALID

--gt Expression Filter [upgrade] VALID

--gt Rule Manager [upgrade] VALID

--gt Oracle OLAP API [upgrade] VALID

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 118

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 119: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 119

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 120: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 120

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 121: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 121

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 122: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 122

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 123: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 123

Listener will be created

on all nodes

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 124: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 124

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 125: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 125

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 126: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 126

Note 4539031 - Enabling and Disabling Oracle Database Vault in UNIX Note 4539021 - Enabling and Disabling Oracle Database Vault in WINDOWS

$ cd $ORACLE_HOMErdbmslib

$ make -f ins_rdbmsmk dv_off

$ cd $ORACLE_HOMEbin

$ relink all

(For RAC installations run these commands on all nodes)

cd u00apporacleproduct1120db_1rdbmslib

make -f ins_rdbmsmk dv_off usrccsbinar d

u00apporacleproduct1120db_1rdbmsliblibknlopta kzvidvo

usrccsbinar cr

u00apporacleproduct1120db_1rdbmsliblibknlopta

u00apporacleproduct1120db_1rdbmslibkzvndvo

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 127: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 127

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 128: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 128

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 129: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 129

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 130: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 130

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 131: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 131

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 132: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 132

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 133: Upgrading to Oracle 11gR2

Database upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 133

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 134: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 134

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 135: Upgrading to Oracle 11gR2

Database post upgrade ndash list

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 135

Run utlrpsql to compile ANY invalid objects

Run utlu112ssql to check post upgrade components status

Adjust time zone data in the database

Gather statistics for system and fixed objects

Amend scripts [if any] to affect the new OH settings

Verify the database upgradesrvctl config database ndashd ltDBNAMEgt

Back up the database

Modify the COMPATIBLE init Parameter to 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 136: Upgrading to Oracle 11gR2

Database upgrade ndash measuring upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 136

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 137: Upgrading to Oracle 11gR2

Database upgrade ndash trivia

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 137

Upgraded over 100 databases successfully using MANUAL and DBUA methods

Avg database upgrade time was just about 2 hrs

Upgrade using DBUA for RAC DB proved a wise decision

Nearly identical duration between CLI and DBUAupgrade methods

Duration is typically proportional to the number ofinstalled database components

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 138: Upgrading to Oracle 11gR2

Database upgrade ndash CLI upgrade steps

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 138

Courtesy from Oracle CorporationThanks to Roy Swonger amp Mike Dietrich

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 139: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 139

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 140: Upgrading to Oracle 11gR2

Upgrade aftermath stories

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 140

Snapshot Controlfile must be placed on a shared storage for a RAC database

ORA-24247 network access denied by access control list (ACL)Solution UTL_TCP UTL_HTTP UTL_SMTP UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMINCREATE_ACL

ADD_PRIVILEGEASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution crsctl modify resource ldquooraasmrdquo ndashattr ldquoCHECK_TIMEOUT=132rdquoMOS [ID 13236791]

IMPDP fails with ORA-29913 31011 19202 LPX-00217[BUG 11877267]Solution no solutionhellip workaround exists

ORA-07445 exception core dump [evaopn3()]Solution _simple_view_merge _complex_view_merging=false

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 141: Upgrading to Oracle 11gR2

Very important apply the latest PSU patch right after

your upgrade

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 141

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 142: Upgrading to Oracle 11gR2

What will be covered

o Why upgradeo Upgrade compatibility matrixo Upgrade pattern [best practices]o Overview of our setupo Prepare for upgradeo Clusterware and ASM upgradeo 11gR2 RDBMS software installationo Database upgradeo Post DB upgrade stepso How to proceed from failed upgrade o Upgrade aftermath storieso References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 142

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 143: Upgrading to Oracle 11gR2

References

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 143

Oracle Database Upgrade Path Reference List [ID 7303651] Complete Checklist for Manual Upgrades to 11gR2 [ID 8375701] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 8708141]Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 10969521] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on LinuxUnix [ID 9692541] Oracle Grid Infrastructure 11202 Installation or Upgrade may fail due to Multicasting Requirement [ID 12127031] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System [ID 7436491]

Oracle 11g R1R2 Real Application Clusters Essentials

Move to Oracle Database 11g ndash The whole Story blogsoraclecomUPGRADE httpdownloadoraclecomdocscdE11882_01server112e17222tochtm httpwwworaclecomtechnetworkdatabaseupgradeindexhtml httpoukcoraclecomstatic09opncore09oracle9i_database95960060311_95960indexhtm

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom

Page 144: Upgrading to Oracle 11gR2

A big thank to red-gate

and you all

for listening

Presented by Syed Jaffer Hussain red-gateAllThingsOracle Slide 144

You can write me at sjaffarhussaingmailcom


Recommended