Diagnostics and Verification of Oracle Real Application Clusters 10g
Jack Cai
Principal Product Manager
Oracle Corporation
40248
Agenda
Real Application Clusters 10g is for Everyone
Benefits of RAC Diagnostics and Verification
Diagnostics Framework Cluster Verification Framework Q&A
Oracle RAC 10g for Everyone Foundation of Oracle grid computing Much easier to use and manage
– Full stack reduces complication Clusterware Cluster filesystem Automatic storage manager
– Integral to Enterprise Manager– Workload management with policy based
service– Cluster diagnostics and verification
Diagnostics & Verification Benefits• Improve problem resolution time
Reduce need to reproduce problems Eliminate problems related to incorrect
cluster configuration Improve quality of services through a
more systematic approach
Diagnostics Framework
Trace instrumentation of Database Server Clusterization of oradebug, hang analyzer Trace loader Navigator Equally useful on single instance database For database server internal diagnostics ONLY
– Not for diagnosis of application problems/tuning– Use them ONLY under Oracle support supervisions
Diagnostics Framework Goals Generate enough diagnostic data for
first pass analysis Minimal overhead to database server Easy to use
– Single system view of a cluster– Tool for data gathering, conversion and
repository– Tool for display, navigate and analysis
DIAG Process
Instance 1
SGA
Diagnostics Architecture (Cluster View)
Off-line Tools
On-line Tools
Trace Files
OradebugX$VIEWs
Trace Navigation ToolTrace Loader
DIAG Process
Instance 2
SGA
DIAG Process
Instance 3
SGA
Diagnostics Architecture (Instance)
Off-line Tools
On-line Tools
X$ Views
Trace Buffer Pi
Trace Buffer Pj
Process Pi
Process Pj
DIAG Process
SGA
Trace Files
Instance
Trace Navigational ToolTrace Navigational Tool
Trace LoaderTrace Loader
OradebugOradebug
Flexible Trace Instrumentation Focus on issues encountered by customers
– Instrument trace on events Oracle support handles most frequently
– Foundation laid in 9i, more component instrumentation in 10g
Trace control– Trace is event based, events can be anything within a
program– Trace is controllable on a per process basis– 1000 event id (10000-10999), 256 opcodes, 256 levels– Trace is generated to SGA buffer rather than directly to
files– X$Trace, X$Trace_events can be queried during run time
“Always-on” minimal tracing Optional trace archiving
Trace Control Example
Dynamic control– ALTER TRACING <cmd>
ON OFF ENABLE <event-string> DISABLE <event-spec> FLUSH <proc-spec>
– Examples alter tracing enable <event>12345:<level>5:<pid>32 alter tracing disable 12345:5:BGS alter tracing flush 32
oradebug Is Cluster Aware Single system image of the cluster Commands are clusterized DIAG based implementation
– Damen process– Minimal overhead
HANGANALYZE analyzes the entire cluster
ExamplesSQL> oradebug -g <inst_list>/ALL <cmd>SQL> oradebug setinst “3 4”
Offline Trace Loader
• Offline trace loader converts binary trace files into text format, and loads files into Oracle databases for archiving
• Better data analysis through power of database queries
• Build a repository for problem tracking
• Facilitate trace data exchange in cross-platform environments
Offline Trace Loader Extracts trace data from archived trace
files Converts binary trace data to text Loads trace data into database as trace
repository Simple command line interface
– trcldr [option,...]– trcldr mode=append ifile=f1.trw userid=scott/tiger
Raw Trace Files(in binary or text format)
Converted Trace Files(in text format)
Target Database
Data Conversion Data Loading
TraceRepository
TraceFile Trace
LoaderTrace
Loader
TraceFile
Trace Navigation Tool
Navigate traces with links and highlights across cluster
– send/recc, malloc/free– User Defined Protocol Navigation
Coloring of traces based upon attributes Well-defined queries against keywords Cluster wide time-ordered event
diagram
Navigation Tool Protocol Trace
Trace Navigation
Navigation Tool Event Diagram
Summary of Diagnostics
Diagnostics facilities allow support to resolve issues more quickly
– First pass failure diagnosis
Allows developers to find bugs faster Tools are not meant for end users End users should only use the tools
under guidance from Oracle support personnel
Cluster Verification Goal
Eliminate errors through: – Pre/post validation of installation steps
and/or configuration changes– Ongoing cluster validation
Flexible– API invoked by other programs
OUI, OEM, DBCA, ASM– Command line interface– Can be run anytime
To be released post Oracle Database 10g
Cluster Verification Framework Verification categories
– Stage verification– Component verification
Diagnostic mode API for vendor verification tools
– Utilizing vendor platforms’ verification capabilities
Stage Verification RAC deployment process is divided into stages
– Stages are steps of installation/configuration changes
– All RAC deployment stages are verified– Verification performs pre-stage and post-stage
checks
Example stages– Cluster filesystem setup– Clusterware setup– Database installation– Cluster node addition– Database configuration
Component Verification
RAC is divided into key components– Each component can be individually
verified
Examples– Node connectivity– Shared storage accessibility– Cluster integrity– System requirements– CFS integrity
Diagnostic Mode
Attempt to pinpoint root cause of a verification failure in diagnostic mode
Applicable to stage and component verification
Specific testings are performed when a verification fails
Only triggered by verification failure
How Verification is done
Command line tool: cluvfy– cluvfy stage –post HWOS (verifies
hardware and OS stage)– cluvfy comp sys –p RDBMS (verifies
system components for RDBMS installation)
APIs by other relevant tools
Summary• Diagnostics speed up problem resolutions • Cluster verification eliminates incorrect
configuration• Ultimately reducing unscheduled system
down time and improving system availability
Diagnostics tools should ONLY be used under support guidance
End Users can use cluvfy to perform verifications
Next Steps…. Recommended sessions
– 40131: Oracle Real Application Clusters 10g: The Fourth Generation
– 40132: Oracle Real Application Clusters 10g: Manage your cluster Without it Managing You
Recommended demos and hands-on labs– Visit DEMOgrounds under Real Application Clusters
See Your Business in Our Software– Visit the DEMOgrounds for more information.
Relevant web sites to visit for more information– http://otn.oracle.com/products/database/clustering/
content.html
Reminder – please complete the OracleWorld online session survey
Thank you.
AQ&Q U E S T I O N SQ U E S T I O N SA N S W E R SA N S W E R S