Post on 08-Apr-2018
transcript
8/6/2019 Beyond 2 Node RAC
1/49
Beyond
2 Node RACWorkload Management
in the GridErik Peterson
Real Application Clusters Development
8/6/2019 Beyond 2 Node RAC
2/49
Agenda
A Gradual Approach to the Grid
An Architectural Mindset Change
Automatic Workload Management
More Customer Examples
8/6/2019 Beyond 2 Node RAC
3/49
Operates over3,700 retail stores
Presence in over910 cities across 18 countries/regions
13 retail brands
Manufactures a full range of leading beverages from
bottled water, fruit juices, soft drinks and tea products
Employs over64,000 staff
2003 turnoverHK$60bn
8/6/2019 Beyond 2 Node RAC
4/49
Phase 1 EBIS Single Node DB and Application server.
Database Node
Application server Node
Users
RAC being designed.
DONE
!
8/6/2019 Beyond 2 Node RAC
5/49
Phase 2 EBIS Dual Node DB and Application server.
Database Nodes with Veritas VCS
in failover mode. Failover node is
usually idle.
Application server Node
Users
RAC being released.
DONE
!
8/6/2019 Beyond 2 Node RAC
6/49
Phase 3 EBIS Dual Node RAC DB and Application server.
Database Nodes with Veritas DBAC cluster
and RAC. Load is balanced between bothnodes.
Application server Node
Users
RAC being adopted.
DONE
!
8/6/2019 Beyond 2 Node RAC
7/49
Phase 4 EBIS + PNS Dual Node RAC DB and Application server.
Database Nodes with
Veritas DBAC cluster
and RAC. Load is
balanced between both
nodes.
Application
server Node
RAC being proved.
10G being released.
EBIS
Application
server Node
Park n Shop
Retek RMS
8/6/2019 Beyond 2 Node RAC
8/49
Phase 5 EBIS + PNS Four Node RAC DB and Application server + PNS D/W.
Application
server Node
RAC being adopted.10G being proved.
EBIS
Application
server Node
Park n Shop
Retek RMS
Park n Shop + Fortress RDWBusiness benefit is that we can use spare capacity on EBIS for PnS
8/6/2019 Beyond 2 Node RAC
9/49
Phase 6 Multi-Node RAC DB and Application servers + RAC Development.
EBIS, PnS Retek RMS, PnS RDW, Fortress RDW Production Cluster.
Application
server Nodes
Development
cluster.
8/6/2019 Beyond 2 Node RAC
10/49
Phase 7 Full development and Production Cluster.
Production Cluster
Development Cluster
Intel/Linux based blade
server for applications
and web front end.
LOAD
BALANCER
LOAD
BALANCER
LOADBALANCER
Nodes can be moved from Development to
Production to cope with seasonal demands.
User Population
8/6/2019 Beyond 2 Node RAC
11/49
An Architectural Mindset Change
8/6/2019 Beyond 2 Node RAC
12/49
Monday Morning
Email Payroll OE DW
100% 20% 50% 40%
8/6/2019 Beyond 2 Node RAC
13/49
Payroll Run (night)
Email Payroll OE DW
10% 100% 10% 80%
8/6/2019 Beyond 2 Node RAC
14/49
Xmas Orders
Email Payroll OE DW
40% 20% 100% 40%
8/6/2019 Beyond 2 Node RAC
15/49
Xmas Orders Desire
PayrollOE
DW
8/6/2019 Beyond 2 Node RAC
16/49
Xmas Orders:
Before Failure
EmailPayroll
OEDW
8/6/2019 Beyond 2 Node RAC
17/49
Xmas Orders:
After Failure
Payroll
OE
DW
8/6/2019 Beyond 2 Node RAC
18/49
Paradigm Shift
0
500
1000
1500
2000
2500
3000
3500
4000
60~70% Grid
ComputingTraditional
Computing
8/6/2019 Beyond 2 Node RAC
19/49
Requirements
Location Transparency
Workload Balancing
Resource Management
Manageable as a Single Environment
8/6/2019 Beyond 2 Node RAC
20/49
Automatic Workload Management
Application workloads can be defined as
Services Individually managed and controlled
Assigned to instances during normal startup
On instance failure, automatic re-assignment Service performance individually tracked
Finer grained control with Resource Manager
Integrated with other Oracle tools / facilities
8/6/2019 Beyond 2 Node RAC
21/49
Services are a unit for
performance A new level dimension for performance tuning
workloads are visible and measurable tuning by service and SQL replaces session and
SQL in most systems where sessions are shared.
Performance measures for real transactions
Alerts and actions when performance goals areviolated
8/6/2019 Beyond 2 Node RAC
22/49
Automatic Workload ManagementProvides Visibility
8/6/2019 Beyond 2 Node RAC
23/49
Useful Service Views
Service status in V$ACTIVE_SERVICES,
DBA_SERVICES, V$SESSION,V$ACTIVE_SESSION_HISTORY
Service performance in V$SERVICE_STATS,V$SERVICE_EVENTS,V$SERVICE_WAIT_CLASSES,V$SERVICEMETRIC,V$SERVICEMETRIC_HISTORY
Service, MODULE, and ACTION performance inV$SERV_MOD_ACT_STATS.
8/6/2019 Beyond 2 Node RAC
24/49
Example Service Metrics
Service time - current NAME ELA(s)/CALL CPU(s)/CALL
--------------------------- ------------ ------------
ERP 0.1940 0.0082
Service time - history every 60 seconds
NAME ELA(s)/CALL CPU (s)/CALL
--------------------------- ------------ ------------ERP 0.1940 0.0082
0.2046 0.0085
0.2154 0.0093
0.2248 0.0105
0.2160 0.00970.2185 0.0104
0.2211 0.0104
8/6/2019 Beyond 2 Node RAC
25/49
Service Thresholds and Alerts
DBMS_SERVER_ALERT.SET_THRESHOLD
SERVICE_ELAPSED_TIME SERVICE_CPU_TIME
Warning and critical levels for observed periods
Import from EM baselines
Comparison of response time againstaccepted minimum levels
a desire for the wall clock time to be, at most, acertain value.
8/6/2019 Beyond 2 Node RAC
26/49
30 service, module, action statistics
user calls DB time response time
DB CPU CPU/service
parse count (total)
parse time elapsed parse time cpu
execute count
sql execute elapsed time
sql execute cpu time
opened cursors cumulative
session logical reads
physical reads
physical writes
redo size
user commits
workarea executions - optimal workarea executions - onepass
workarea executions - multipass
session cursor cache hits
user rollbacks
db block changes
gc cr blocks received
gc cr block receive time
gc current blocks received
gc current block receive time
cluster wait time concurrency wait time
application wait time
user I/O wait time
8/6/2019 Beyond 2 Node RAC
27/49
Automatic Workload ManagementProvides Visibility
0
5
10
15
20
25
30
9 10 11 12 13 14 15 16 17 18
D
C
B
A
8/6/2019 Beyond 2 Node RAC
28/49
AWR Automatically Measures
Service AWR measures response time, resource used
Automatically for work done in every service
AWR monitors thresholds, sends AWR alerts
response time, cpu used
maintains runtime history every 30 minutes
Statistics collection and tracing are HA
persistent for service location / instance restart. enabled and disabled globally for RAC.
8/6/2019 Beyond 2 Node RAC
29/49
Service High Availability using
RAC 10g Focus is on protecting the application services
more flexible/more cost effective than other HAthat focus on availability of single physical systems
Services are available continuously with load
shared across one or more instances. Any server in the RAC can offer services
in response to failures
in response to planned maintenance.
in response to runtime demands
8/6/2019 Beyond 2 Node RAC
30/49
Automatic Workload ManagementIntegration with Other Tools
Job Scheduler
Job classes mapped to services
Parallel Query / DML
Query coordinator connects to a service like any
other client
Parallel slaves inherit service from coordinator
Oracle Streams Advanced Queuing
Queues accessed by service
Achieves location transparency
8/6/2019 Beyond 2 Node RAC
31/49
Failure Notification (FaNTM) with Oracle
Application Server 10g
Fast, coordinated
recovery without humanintervention
Oracle Database 10gRAC signals JDBC
Fast Connect Failoverwhen instances failuresoccur
Immediate recovery formid-tiers
< 4 seconds from15 minutes
self correcting
Oracle Clusterware
AppServer
10g
RAC
8/6/2019 Beyond 2 Node RAC
32/49
Failure Notification (FaN)JDBC Fast Connection Failover Processing
Supports multipleconnection caches
Datasource for eachcache mapped to aservice
Keeps track of serviceand instance for eachconnection
Distributes new workrequests acrossavailable instances
SERVICE 1
SERVICE 2
SERVICE 3
JDBC / Mid-Tier Database Tier
INSTANCE X
INSTANCE Z
INSTANCE Y
CACHES
8/6/2019 Beyond 2 Node RAC
33/49
Initial State
C2
C3
C4
C5C6
C1
Instance 1
Instance 2
Service across RAC
Connection Pool
8/6/2019 Beyond 2 Node RAC
34/49
Instance Join
C2
C3
C4
C5C6
C1
Instance 1
Instance 2
Instance 3
Service across RAC
Connection Pool
8/6/2019 Beyond 2 Node RAC
35/49
Instance Join Undesired CasesOption 1: Nothing Happens
C2
C3
C4
C5C6
C1
Instance 1
Instance 2
Instance 3
Service across RAC
Connection Pool
Instance Join Undesired Cases
8/6/2019 Beyond 2 Node RAC
36/49
Instance Join Undesired CasesOption 2: Add New Connections
Randomly
C2
C3
C4
C5C6
C7
C8
C9
C1
Instance 1
Instance 2
Instance 3
Service across RAC
Connection Pool
8/6/2019 Beyond 2 Node RAC
37/49
Instance Join Desired Result
C2
C3
C4
C5C6
C7
C8
C9
C1
Instance 1
Instance 2
Instance 3
Service across RAC
Connection Pool
8/6/2019 Beyond 2 Node RAC
38/49
Rebalancing upon Instance
Join Desire: Create new connections to new
instances & potentially disable some old ones. Method:
10gR1: Fast Application Notification (FAN)
Using the HA up and down events for services inRAC
9i: DBMS session disconnect (Manual steps)
8/6/2019 Beyond 2 Node RAC
39/49
Node Leaves
C2
C3
C4
C5C6
C7
C8
C9
C1
Instance 1
Instance 2
Service across RAC
Connection Pool
8/6/2019 Beyond 2 Node RAC
40/49
Work
request
C2
C3
C4
C5C6
C7
C8
C9
C1
Instance 1
Instance 2
Instance 2
Service across RAC
Connection Pool
Bad
OK
Good
1 2
10gR2 Connection Pool Load Balancing
8/6/2019 Beyond 2 Node RAC
41/49
Customer Examples
8/6/2019 Beyond 2 Node RAC
42/49
Critical 4
Node-4
New Distribution Possible
Chicago Stock Exchange
Batch
Low
Critical 1
Node-3Node-2Node-1
Critical 2 Critical 3
Medium 1 Medium 2
Chicago Stock Exchange
8/6/2019 Beyond 2 Node RAC
43/49
Chicago Stock Exchange
Resource Management Plan -
Daytime
Critical1
50%
Remainder
25%
Remainder
30%
Low
80%
Medium 1
30%
P1
P2
P3
Critical 2
25%
Medium 2
40%
Need flexibility to change plan to meet changing demands
Batch
20%
D ll
8/6/2019 Beyond 2 Node RAC
44/49
Gains: Manages Priorities, Visibility of Use,
Can turn off reporting during aggregation
Node-1 Node-2 Node-3 Node-4
Adhoc Reports P3
Loads/Aggregation P2 0-100%0-100%0-100%0-100%
Dell
European Operations DW
0-100%0-100%
0-100%0-100%0-100%0-100%Critical Reports P1
0-100%0-100%
8/6/2019 Beyond 2 Node RAC
45/49
Talk America
DW
OLTP 1
OLTP 2
OLTP 3
OLTP 4
Node-4Node-3Node-2Node-1 Node-6Node-5
BatchRepor
ting
8/6/2019 Beyond 2 Node RAC
46/49
Consolidated eCommerce Project
Retailer
eCommerce
Backups
Order Management
Real Time
Reporting
Spare Capacity
DW
8/6/2019 Beyond 2 Node RAC
47/49
Large Node Proofpoints
Amazon.com 16 node Clickstream DW (17 TB), 8node ETL DW on Linux
a US Telecom 16 node Grid on Solaris
Telstra 8 Node - 18 TB DW on Linux
Mercadolibre 8 Node OLTP on Linux/Itanium Gas Natural 8 Node DW, 7 Node SAP BI system
AC3 - 63 Node POC on Linux)
OpenWorld Japan w/ Sun & Cisco - 128 NodeCluster
8/6/2019 Beyond 2 Node RAC
48/49
Closing Remarks
Use 4+ Nodes Greater HA & Scalability
Connect by Service: not SID, not Instance, notHost
Instrument your code to use Module & Action
Implement Services weather your DBconsists of one application or many
8/6/2019 Beyond 2 Node RAC
49/49
Q U E S T I O N SQ U E S T I O N S
A N S W E R SA N S W E R S