Oracle Database Service HA with Data Guard · Our company. 3 23.11.2017 Trivadis DOAG17: Oracle...

Oracle Database Service High Availability with Data Guard?Robert BialekSenior Principal Consultant

@RobertPBialek doag2017

Who Am I

Trivadis DOAG17: Oracle Database Service High Availability with Data Guard?2 23.11.2017

Senior Principal Consultant and Trainer at Trivadis GmbH in Munich.

– Master of Science in Computer Engineering.

– At Trivadis since 2004.

– Trivadis Partner since 2012.

Focus:

– Data and Service High Availability, Disaster Recovery.

– Architecture Design, Optimization, Automation.

– New Technologies (Trivadis Technology Center).

– Open Source.

– Technical Project Leadership.

– Trainer: O-GRINF, O-RAC, O-DG.

Our company.


Trivadis is a market leader in IT consulting, system integration, solution engineering

and the provision of IT services focusing on and

technologies

in Switzerland, Germany, Austria and Denmark. We offer our services in the following

strategic business fields:

Trivadis Services takes over the interacting operation of your IT systems.

O P E R A T I O N

COPENHAGEN

MUNICH

LAUSANNE

BERN

ZURICH

BRUGG

GENEVA

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

VIENNA

With over 600 specialists and IT experts in your region.


14 Trivadis branches and more than

600 employees.

200 Service Level Agreements.

Over 4,000 training participants.

Research and development budget:

CHF 5.0 million.

Financially self-supporting and

sustainably profitable.

Experience from more than 1,900

projects per year at over 800

customers.

23.11.2017 Trivadis DOAG17: Oracle Database Service High Availability with Data Guard?5

Technology on its own won't help you.You need to know how to use it properly.

Database Service High Availability – Goal


Increase database service uptime, by:

– eliminating any single point of failure to avoid unplanned outages.

– minimizing the effect of an unplanned outage on the end user (automatic failover).

– reducing downtimes during planned outages.

Consider the whole SW/HW stack. Find the best cost/risk ratio.

StorageDatabase

Server(s)Application

Server(s)Clients

Best cost/risk

ratio

Downtime

Effort

Costs

Complexity

Availability

Database Service High Availability – Options


Cluster – primarily used option for service high availability:

– Real Application Clusters.

– RAC One Node.

– Cold Failover Cluster.

Data Replication – used mostly for data, rather than

service high availability:

– Data Guard (Fast-Start Failover/Global Data Services).

– GoldenGate (Global Data Services).

– Other replication technologies.

SPOF

HA

HA

HA

Agenda


1. Introduction

2. Configuration

3. Special Cases

4. Conclusions


Introduction

Data Guard

Database Service HA with Data Guard? – Introduction


Yes, it can also be used for service high availability:

– Planned downtimes – manual switchover.

– Unplanned downtimes – fast-start failover or manual failover.

Why might we consider Data Guard for service high availability:

– Less complex than a cluster installation.

– Infrastructure requirements not that high (even local storage is sufficient).

– Not subject to additional license fees (EE license assumed).

– Additionally, many other advantages: data high availability, snapshot standby,

potentially rolling upgrade capability, ...

But, with some restrictions we need to consider...

Primary Standby

FSFO Configuration

Database Service HA with Data Guard? – Big Picture


Primary Target

Failover StandbyCandidate Target Failover Standbys

(optional, 12.2)

Master Observer

Backup Observers

(optional, 12.2)

...

Database Clients

Primary

RW Service

Transparency

required

(failover/

switchover)

Ping Ping Ping

Required

Primary Target

Failover Standby

Database Service HA with Data Guard? – Monitoring


Observer

DGMGRL – Threads

W000 B001

P001 S001

Connect

Enter PING State

PING

SLEEP

~ 3sec.

ObserverReconnect

property set and reached

Reconnect interval expired

PRIMARY

Logoff

PING

SLEEP

~ 3sec.

TARGET STANDBY

Failover condition

detected

FailoverConnect

re-tries

Timeout


Configuration

Data Guard Protection Modes with FSFO – Prerequisites


▪ LogXptMode=SYNC or

FASTSYNC (12.1+)

▪ FastStartFailoverTarget(*)

▪ Flashback Database

▪ LogXptMode=ASYNC


▪ FastStartFailoverLagLimit


▪ LogXptMode=SYNC



▪ Recommended: at least 2

STDBY DBs (protection

mode downgrade!)

MaxAvailability (10.2+) MaxPerformance (11.1+) MaxProtection (12.2)

DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverThreshold = <xy>;

DGMGRL> ENABLE FAST_START FAILOVER;

All Protection Modes

FSFO: Guaranted zero data

loss.

FSFO: Data loss possible. FSFO: Guaranted zero data

loss.

Mostly used

protection mode

Value in seconds

Fast-Start Failover – Observer (1)


Monitoring component, initiates a failover procedure.

In 12.2, up to 3 observers (in background) can be started:

– One master and up to two backup (standby) observers.

In older releases:

– Only one running observer (HA needs to be adressed).

DGMGRL> START OBSERVER OBS1.TRIVADIS.COM IN BACKGROUND

FILE IS '$ADMIN_SID/fsfo_$ORACLE_SID.dat'

LOGFILE IS '$ADMIN_SID/fsfo_$ORACLE_SID.log'

CONNECT IDENTIFIER IS <Alias1>.TRIVADIS.COM;

W000 B001

P001 S001

Oracle wallet

required

nohup dgmgrl -logfile $ADMIN_SID/fsfo_$ORACLE_SID.log <<EOD &

CONNECT $CONNECT_DATA

START OBSERVER FILE='$ADMIN_SID/fsfo_$ORACLE_SID.dat';

EOD

Observer

PRIMARY TARGET

Failover Standby

Ping

Fast-Start Failover – Observer (2)


Fast start-failover is initiated by the master observer to

the target standby database, if one of the following

conditions is detected:

– observer and the target standby database cannot reach

the primary database (default: ObserverOverride=‘FALSE‘).

– user-configurable condition is met.

– DBMS_DG.INITIATE_FS_FAILOVER function has been executed.

Additonally, other pre-condidtions enforced by a protection mode need to be fulfilled:

– MaxProtection/MaxAvailability: target failover standby is in SYNC.

– MaxPerformance: FastStartFailoverLagLimit not reached for the target failover

standby.

W000 B001

P001 S001

Observer

PRIMARY TARGET

Failover Standby

Ping

Data Guard: Role-Based Services


For a Data Guard system, we need a role-based service,

that is running only if database has a specific role:

– Read-write service on a primary database.

– Optionally, a service on standby databases for reporting.

– Optionally, a service on snapshot standby databases.

To accomplish this task:

– Use Oracle Grid Infrastructure role-based services.

– Create your own AFTER STARTUP ON DATABASE trigger.

Database Clients

Service

R/WService

R/O

[SNAP]

?

Data Guard: Example Role-Based Services


Example role-based services with Grid Infrastructure.

Services are started only if database and service role match.

Depending on used client HA features (TAF, FAN/FCF, AC) additional service

properties need to be specified.

srvctl add service -db DB_SITE1 –service SRV_RW.trivadis.com -role PRIMARY

srvctl add service -db DB_SITE1 –service SRV_RO.trivadis.com -role PHYSICAL_STANDBY

srvctl add service -db DB_SITE1 -service SRV_SP.trivadis.com -role SNAPSHOT_STANDBY

SvcAgent::start 680 query_db_role

SvcAgent::start 710 not starting service srv_rw Role mismatch - Service

role:PRIMARY, current DB role:PHYSICAL_STANDBY

Client-Side Configuration – Main Problems To Address

New network session (connect) Already established network session

Database Clients

1 IP not reachable (server/network/… issue)

2 Connect attempts

3 Wait for

connect timeout

4 Client failover

Problem

Database Clients

2 IP not reachable (server/network/… issue)

1 Connected

3 Re-connect attempts

4 Wait for

re-connect timeout

Problem

5 Client failover

ProblemProblem

CASE 1 CASE 2


New Oracle Net Session – Connect Timeout (1)

sqlnet.ora parameters (OCI, ODP.net)

– Applies to each IP that a host name resolves to!

– All Oracle client versions supported.

For clients >=11.2:

TCP.CONNECT_TIMEOUT=3 #default 60 sec.

SQLNET.OUTBOUND_CONNECT_TIMEOUT=5 #no default

LSNR LSNR

Three-way handshake

Oracle Net

1

2

3


OLTP.trivadis.com =

(DESCRIPTION =

(FAILOVER=ON) (LOAD_BALANCE=OFF)

(CONNECT_TIMEOUT=5)(RETRY_COUNT=3)(RETRY_DELAY=1)(TRANSPORT_CONNECT_TIMEOUT=3)

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP )(HOST = italy )(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP )(HOST = sweden )(PORT = 1521)))

(CONNECT_DATA = (SERVICE_NAME = OLTP_RW.trivadis.com)))

Introduced in 12.1.0.2

New Oracle Net Session – Connect Timeout (2)

JDBC Thin driver

– TRANSPORT_CONNECT_TIMEOUT is available beginning with 12.2 version

– To use RETRY_COUNT with 12.1.0.2, patch is required (BUG 19154304)

JDBC Thin clients can alternatively use the following driver property (ms)

– Overrides CONNECT_TIMEOUT from address description parameters

pds.setURL("jdbc:oracle:thin:@(DESCRIPTION =(FAILOVER=ON)(LOAD_BALANCE=OFF)" +

"(CONNECT_TIMEOUT=3)(RETRY_COUNT=10)(RETRY_DELAY=1)" +

"(ADDRESS_LIST = " +

"(ADDRESS = (PROTOCOL = TCP )(HOST = blue.trivadis.com )(PORT = 1521)) " +

"(ADDRESS = (PROTOCOL = TCP )(HOST = brown.trivadis.com )(PORT = 1521))) " +

"(CONNECT_DATA = (SERVICE_NAME = sales_rw.trivadis.com)))");

Properties prop = new Properties();

prop.put(oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR, ""+3000);

ods.setConnectionProperties(prop);


Established Oracle Net Session – Re-Connect Timeout

Break established network connection

without waiting for long TCP timeouts (>15 min.)

– In most cases no VIPs in use!

Using the following parameters is not a good idea:

Better solution:

– If possible use: Fast Application Notification/Fast Connection Failover.

– Tuning OS kernel parameter tcp_retries2 might be also an alternative.

LSNR LSNROracle Net

1P1

2

Client failover

4

SQLNET.RECV_TIMEOUT=30 #no default value, OCI driver

SQLNET.SEND_TIMEOUT=30 #no default value, OCI driver

prop.put ("oracle.jdbc.ReadTimeout", "5000"); //5000ms, JDBC Thin driver

Timeout

3


Client HA Features – Overview

Transparent Application Failover:

– Can be used with Data Guard the same way (advantages/disadvantages) as with

a cluster.

Fast Application Notification/Fast Connection Failover:

– Oracle Grid Infrastructure is required to register with ONS.

– Comparing to RAC “only” rapid notification about up/down events, no workload

balancing.

Application Continuity can be used with Data Guard the same way as with a cluster:

– But requires RAC or RAC One Node or ADG (GG) option.

More about this topic:

– DOAG 2016 presentation: „Oracle Client Failover - Under the Hood”



Special Cases

Special Cases with FSFO: Candidate Targets Standby


Starting with 12.2, many candidate fast-start failover target databases can be

specified, but switchover or FSFO works only to the current target standby database.

Threshold: 60 seconds

Target: db_site2

Candidate Targets: db_site2,db_site3

Observers: (*) obs1.trivadis.com

obs2.trivadis.com

DGMGRL> EDIT DATABASE db_site1 SET PROPERTY FastStartFailoverTarget =

'db_site2,db_site3';

DGMGRL> SWITCHOVER TO db_site3;

Error: ORA-16655: specified standby database not the current fast-start

failover target standby.

db_site1 db_site2 db_site3

Current target depends

on many conditions

FSFO

Switchover

Special Cases: Switchover – DelayMins >0


Version 11.2.0.X: recovery re-started with NODELAY option.

Version 12.1.0.X: recovery waits until DelayMins reached!

– OPEN_MODE

• Primary – CLOSED BY SWITCHOVER

• Standby – MOUNTED

– Application RW service outage within DelayMins time-frame! Application connect

attemps fail with ORA-16456: switchover to standby in progress or completed

Version 12.2.0.1: Switchover is not possible.

Error: ORA-16672: switchover not permitted to standby database with non-zero

DelayMinsFailed.

PRIMARY TARGET

Failover Standby

DelayMins>0

Special Cases with FSFO: Master Observer Failure


After about 31 sec. the master observer is changed, i.e.:

available backup observer is promoted to the master role.

– Note: to perform a master change, the primary database

needs to be available!

For maintenance, a master change can be performed manually.

Data Guard Broker initiated a master observer switch since

the current master observer cannot reach the primary database

Master Observer Backup Observers

DGMGRL> SET MASTEROBSERVER TO obs2.trivadis.com;

Sent the proposed master observer to the data guard broker configuration.

Please run SHOW OBSERVER to see if master observer switch actually happens.

PRIMARY

Logged on the

primary database

Special Cases with FSFO: Failover Target Failure


After about 10 sec. the target failover standby is changed, i.e.: a candidate

target failover standby observer is promoted to the current target role.

Note: to perform the target failover standby change, the primary database and the

master observer need to be available!

– If the master observer fails at the same time:

Permission granted to the primary database for target

switch.

The primary database returned to SYNC/NOT LAGGING state

with the standby database db_site3.db_site1 db_site2 db_site3

Observer

LGWR: FSFO SetState("UNSYNC", 0x2) operation requires an ack

Primary database will shutdown within 30 seconds if permission

is not granted from Observer or FSFO target standby to proceed

Special Cases with FSFO: Master Observer/Primary


If the primary and the master observer fail:

– No failover is initiated to a candidate standby.

– From a backup observer log file:

If the master observer is started at a later time, it waits until

FastStartFailoverThreshold timeout is reached again and fails over to the current

target standby.

Ready to failover check on standby returned

RFS_NON_MSTOB.

Command READY_TO_FSFO to thread S024 returned status=0

Fast-Start Failover is not possible because this

observer is not the master.

Master Observer Backup Observers


Special Cases with FSFO: Private Redo Network


If the public network on the primary server fails:

– Broker configuration property: ObserverOverride=FALSE.

– No failover (HB over private network still works!).

In this network configuration consider using:

Fast-Start Failover is not possible because

primary last contacted the standby within

FastStartFailoverThreshold seconds


Master Observer

Private Network

DGMGRL> EDIT CONFIGURATION SET PROPERTY ObserverOverride='TRUE';

Public Network

HB

Master Observer, Primary/Standby DB: Location? (1)


For DR HA service protection, do not place the primary and the master observer in

the same data center!

Data Center 1 Data Center 2

Master Observer

Primary

Backup Observer

Target Standby

No automatic

failover!

No RW application service available!

Master Observer, Primary/Standby DB: Location? (2)


For DR HA service protection, do not place the primary and the master observer in

the same data center!

Data Center 1 Data Center 2

Backup Observer

Target Standby

Automatic

failover!

RW application service available!

Master ObserverMaster observer placement correction monitoring:

DGMGRL> SET MASTEROBSERVER TO …

Candidate

Target StandbyPotential placement

problem!

Primary

To relocate:

Disable & Enable

FSFO

Conclusions


Conclusions (1)

Can Data Guard be a good solution for database service high availability?

– Yes, with a fast-start failover configuration.

– Anyway, it is not a replacement for a cluster but rather an alternative.

– Careful business requirements analysis is necessary.

Advantages:

– It offers a good service high availability, in addition to excellent data high

availability and some other features.

– Fairly simple solution (setup and operation).

– Not subject to additional license fees (EE license assumed).

– Infrastructure requirements not that high as for a cluster.

– Most client HA features can be used the same way as with a cluster.


Conclusions (2)

Disadvantages:

– Component placement is critical and requires customized monitoring scripts.

– Some technical restrictions like network latencies (SYNC), flashback database or

force logging might limit Data Guard in this area.

– Re-connect timeouts without FAN/FCF (no VIPs).


Trivadis @ DOAG 2017#opencompany

Booth: 3rd Floor – next to the escalator

We share our Know how!Just come across, Live-Presentationsand documents archive

T-Shirts, Contest and much more

We look forward to your visit


Date post:	04-Mar-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	1 times

Oracle Database Service HA with Data Guard · Our company. 3 23.11.2017 Trivadis DOAG17: Oracle...

Documents