All Change

transcript

All Change:The Story of a MigrationDr. Jason Arneil

• Introduction

• The Installation

• The Migration

• Conclusion

AGENDA

Introduction

• Jason Arneil

• Oracle DBA/System & Storage Administrator

• Using Oracle since 1998

• At Nominet since 2001

About Me

Introduction

• Nominet is the internet registry for .uk domain names

• Nominet has just celebrated its 11th anniversary

• Nominet is run as a not-for-profit company

• Nominet is owned by its members

• There are almost 6 Million .uk domain names

About Nominet

Introduction

Hardware/Software Stack

Oracle10.1

Veritas

Powerpath

Solaris 8

Introduction

Motivation

• Replace ageing SPARC hardware

• New 4-way Sun SPARC servers ~ £50K EACH

• Equivalent 4-way AMD Opteron cost £20K in TOTAL

• Reduce complexity in the software stack

• Ability to store accented characters

Introduction

Changes

10.1.0.3 10.2.0.2

Solaris 8 RedHat AS 4

VxFS ASM

US7ASCII UTF8

SPARC AMD

Introduction

Issues

• This was not just a case of running an upgrade script

• Database was some 100GB in size

• Change in endianess an additional complication

• Downtime allowable - but needed to be the minimum possible

The Installation

Initial Server Install

• Server installation kept to as minimal rpm’s as possible

• Recommended install rpms have many package dependencies

• Both 32-bit and 64-bit versions of certain packages are required– rpm --query --queryformat “%(NAME)-%(VERSION).%(RELEASE)

(%(ARCH))\n” rpm name

• Binutils rpm incorrect with RedHat Update 2 – rpm -Uvh --force binutils-2.15.92.0.213.0.0.0.2.x86_64.rpm

The Installation

ASMLib

• Eliminates need to bind raw devices

• Need to install 3 rpms:

– oracleasm-support

– oracleasm-your-kernel

– oracleasmlib

• Must get ASMLib from

http://www.oracle.com/technology/tech/linux/asmlib/index.html

The Installation

Other tips

• Follow recommendations in the Oracle Validated Linux configurations:

http://www.oracle.com/technology/tech/linux/validated-configurations/index.html

• Ensure DNS resolving works for hostnames, VIPs

• Use Cluster Verification Utility

• Bond the Interconnect

The Installation

Hugepages

• Translation Lookaside Buffer maps virtual -> physical memory

• Normal memory page size is 4K

• Hugepage size is 2MB resulting in more efficient memory access

• Method of moving tablespaces between databases

• Export metadata then copy the actual datafiles

• Plug in the metadata into the target database

• Transporting across platforms new in 10g

• A change of endianess means an rman convert is needed

The Migration

Transportable Tablespaces

• Check your platform is supported for transporting:

– sql> select * from V$TRANSPORTABLE_PLATFORM

PLATFORM_ID PLATFORM_NAME ENDIAN_FORMAT

----------- -------------------------------- --------------

1 Solaris[tm] OE (32-bit) Big

2 Solaris[tm] OE (64-bit) Big

3 HP-UX (64-bit) Big

4 HP-UX IA (64-bit) Big

5 HP Tru64 UNIX Little

6 AIX-Based Systems (64-bit) Big

7 Microsoft Windows IA (32-bit) Little

The Migration

Transportable Tablespaces

The Migration

Transport

• Must have a self-contained set of tablespaces to move

– sql> EXECUTE DBMS_TTS.TRANSPORT_SET_CHECK('TBNAME', TRUE);

– sql> select * from TRANSPORT_SET_VIOLATIONS;

• You must make the tablespaces to be moved read only

• Convert the endianess of the datafiles

– rman> convert tablespace TBNAME to platform 'Linux 64-bit for AMD’ format='/home/oracle/%U' ;

The Migration

Plugging In• Now you need the metadata:

– exp ‘as sys’ transport_tablespace=y file=tbname.dmp tablespaces=tbname

• Plug tablespace in as a “normal” filesystem tablespace:

– imp ‘as sys’ transport_tablespace=y file=‘path to export’ datafiles=‘path to filesystem datafile’

• Convert from filesystem datafile to ASM datafile:

– rman> convert datafile ‘path to filesystem datafile’ format ‘ASM PATH’

The Migration

The Plan

Solaris Linux

Copy datafiles

metadata

The Migration

A problem

• Started to notice space calculations incorrect

• Particular problem with 1 critical tablespace

• Assumed we had problems with our created types & custom indexing

• Clearing database down had catastrophic problem:

– sql> drop user TRANSPORTED cascade;

ERROR at line 1:

ORA-00600: internal error code, arguments: [ktecgetsh-inc], [2], [], [], [], [], [], []

The Migration

Oracle Response

The Migration

The cause of the problem

• The cause of the instance crashing was the following:

– ORA-00600: internal error code, arguments: [ktecgetsh-inc], [2], [], [], [], [], [], []

– Non-fatal internal error happened while SMON was doing temporary segment drop.

– Instance terminated by PMON

• Internal table SEG$ info wrong after TTS

• Fixed by patch 5186784

The Migration

Data Pump to the Rescue!• Could not wait for a fix to the Transportable Tablespace problems

- did not know how long it would take.

• Data Pump new in 10g - export/import on steroids

• Oracle claim Data Pump is 2x faster than export and 15-45 times faster than import

• Restartable with detailed job monitoring

• Automatic tuning and parallel server side execution

• direct import across the network

The Migration

Data Pump Architecture

Client Process

Status Queue

Master table LOGMaster Control Process

Command and Control Queue

Worker Worker Dump file set

Data, Metadata & Master Table

The Migration

Data Pump Basics

• Create dumpdir:

– sql> create directory dumpdir as '/home/oracle/dumpdir’

– sql> grant read, write on directory dumpdir to USER;

• Actually do a Data Pump export:

– expdp user/password schemas=user directory=dumpdir dumpfile=output.dmp

• Do the Data Pump import:

– impdp user/password directory=dumpdir dumpfile=output.dmp

The Migration

Data Pump Issues

• Data Pump expdp did not run trouble free:

– ORA-31693: Table data object ”USER"."DOMAINS" failed to

load/unload and is being skipped due to error: ORA-06502: PL/SQL: numeric or value error

LPX-00210: expected '<' instead of '?'

• Apparently the xml/xsl stylesheets were not loaded into our 10.1 database

• The fix was running $Oracle_Home/rdbms/admin/initmeta.sql

The Migration

Data Pump Issues

• Data Pump impdp also was not trouble free:

– ORA-31693: Table data object ”USER"."DOMAINS" failed to load/unload and is being skipped due to error:ORA-44002: invalid object nameORA-06512: at "SYS.DBMS_ASSERT", line 283ORA-06512: at "SYS.KUPD$DATA", line 1336ORA-12801: error signaled in parallel query server P004,instance lr2:NOM2 (2)ORA-29913: error in executing ODCIEXTTABLEOPEN calloutORA-31626: job does not existORA-06512: at "SYS.ORACLE_DATAPUMP", line 19

• Seems like Data Pump import running in parallel across RAC nodes does not work. Shutdown of 2nd node cleared this problem

The Migration

Data Pump Issues

• We encountered even more problems during an import:

– ORA-02374: conversion error loading table ”USER"."INVOICES” ORA-12899: value too large for column CONTACT (actual: 162, maximum: 100)ORA-02372: data for row: CONTACT0X'C3A1C2B3CA98C3C5C28DC2A3C283C2A1C3A3C283C2B3C3'

• This time it was due to changing charactersetUS7ASCII -> UTF8

• Fortunately not too many rows affected so data munging was possible

The Migration

Data Pump Issues

• Tried playing with importing directly across network:

– ORA-31693: Table data object "SNOWDROP"."EMPLOYEE_OLE" failed to load/unload and is being skipped due to error:ORA-06502: PL/SQL: numeric or value error

• Seems like this is either a Bug:

– 3369197 - Dpimp : Network Mode Hr.Per_Addresses Ora-00997: Illegal Use Of Long Datatype Errors

– Metalink Note 313496.1

• Or maybe this is a feature - the above bug & Metalink Note are NOT publicly visible

The Migration

Final Data Pump Issues

• When we came to the day of the migration we hit this problem when exporting:

– expdp system parfile=param.exp

ORA-31626: job does not existORA-31637: cannot create job SYS_EXPORT_SCHEMA_01 for

user SYSTEM ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95 ORA-06512: at "SYS.KUPV$FT_INT", line 601

ORA-12850: Could not allocate slaves on all specified instances: 2 needed, 1 allocated

• We managed to export 1 schema without issue but second schema was starting Parallel slaves on both nodes

The Migration

Data Pump Timings

• Use a parameter file:

– dumpfile=user.dmpdirectory=dumpdirparallel=4logfile=user.logschemas=user

• Some timings:

(time mins)

network

(time mins)

35 11 55 75

49 13 95 240

Tips & Conclusions

ODBC Issue• Rule based Optimizer meant to be de-supported in 10g - see

Metalink Note: 189702.1

• ODBC driver was adding /*+ RULE */ hint to certain queries - still performing ok in 10.1.0.3

• /*+ RULE */ hinted queries in 10.2.0.2 were performing x 100 slower

• This is covered in Metalink Note:373129.1

• Added logon trigger for application to ignore optimizer hints _optimizer_ignore_hints

Tips & Conclusions

Post-Migration Issues

• There were quite a few niggling issues post migration

• perl-dbi applications silently dying

– update DBD driver to 1.17

• Pro*C applications silently dying

– update Oracle Client to 10.2

Tips & Conclusions

Post-Migration Issues

• Java JDBC problem:

– java.sql.SQLException: OALL8 is in an inconsistent state.

• Character set problem with domain registration system

– ORA-01756: quoted string not properly terminated

• Developers finding counting characters “harder”

– Can no longer guarantee 1 character == 1 byte

Tips & Conclusions

Migration Benefits

• Load on servers is greatly reduced

• Most ran query has gone from 1.6ms to 0.8ms

• RAC related waits decreased

• Stable for 6 months +

Tips & Conclusions

Conclusion

• TEST, TEST, TEST

• Transportable tablespaces is quickest way of moving data

– changing platforms really complicates & slows it down

• Data Pump is much, much faster than old export/import

– Roughly 25GB per hour

Questions?

All Change:The Story of a Migration

Contact:

• jason@nominet.org.uk

• http://blog.nominet.org.uk

All Change

Technology