Post on 08-Jun-2015
transcript
Virtualization, Oracle RAC 11g and more..
Vijay RagunathanGuidewire Software Incvragunathan@guidewire.com
Guidewire Software Inc
Agenda
Introduction Set up VM and RAC Take a look at a few 11g RAC/ASM features Introduce Cache fusion and trace Cache
Fusion
Guidewire Software Inc
Virtualization
Abstraction layer between applications and hardware
Methodology to divide the resources of a computer hardware into multiple operating environments
Partition at various levels like Storage, Network, Server, Application
Guidewire Software Inc
Why Virtualize
Consolidate IT infrastructure Improve QoS Reduce cost Provide more reliable working environment for
development and testing Ease replication of environments Partition applications to get security, reliability
and flexibility
Guidewire Software Inc
Where can we Virtualize
Development and Test Virtual Desktops PoC deployments Disaster Recovery and Business continuity Technology refresh
Guidewire Software Inc
Evolution
Virtualization in place for three decades Mainframes RISC based systems And now commodity hardware
Guidewire Software Inc
Commodity Virtualization Software
VmWare Xen Oracle VM (Xen) Microsoft Virtual PC
And probably many more…
Guidewire Software Inc
Oracle Real Application Cluster
Cluster is two or more nodes sharing some common resources
They communicate with each other to check health and guarantee data integrity
For Oracle, it is two or more Oracle instances sharing a physical database
Each instance use their own SGA and background processes
Database components like Datafiles, control files and redo log files are shared
Guidewire Software Inc
Real World Oracle RAC Implementations
RISC or CISC based processers on two or more servers
A SAN or NAS for storing database Interconnect mechanisms like Infiniband,
Gigabit Ethernet, Memory channel
Guidewire Software Inc
Node1 Node2
Storage
Real Application Clusters
Guidewire Software Inc
VM and Oracle RAC
Hardware for clustering too costly Difficult to always build dev and test
environments with clustering Typically Applications built on single instance
and tested and deployed on RAC VM is Beneficial for small, medium and large
companies and cost effective
Guidewire Software Inc
Virtual Node1
VirtualNode2
Same Disks in
Host
Real Application Clusters
Physical Machine
Setup Xen
Guidewire Software Inc
Xen Architecture
Xen hypervisor has management and virtual hardware API
Dom0 hosts the management code and has control interfaces to manage other domains
All guest Virtual machines are referred as domU
Guidewire Software Inc
Packages
xen-3.0.3-41.el5 kernel-xen-2.6.18-53.el5 xen-libs-3.0.3-41.el5 kernel-xen-devel-2.6.18-53.el5
Guidewire Software Inc
Dom0 Host
Default Centos 5.1 install Virtual Machine Manager GUI Command line tool virt-install
Guidewire Software Inc
Virtual Machine Manager
The GUI VMM displays all the domains and their resource usage
Guidewire Software Inc
Create a VM
File -> New Machine
Guidewire Software Inc
Name the VM
We name the first VM as dtrac41
Guidewire Software Inc
Virtualization Method
Guidewire Software Inc
ParaVirtualization
ParaVirtualization uses a modified kernel Not very portable Not all hardware is simulated Best performing for Network and IO Unique address map for each VM Easy to implement
Guidewire Software Inc
Fully Virtualized
Complete Hardware simulation Any supported guest OS can be used Not supported by all hardware architecture VT support may need to be enabled in BIOS
for some hardware Use techniques to trap and emulate
instructions
Guidewire Software Inc
Install using PXE PXE server enables storing images in central location and install large
number of machines Using centos 5.1 for guestVM Kickstart Template file instructs what to install
Guidewire Software Inc
OS and Swap for GuestVM
Using simple files Default path /var/lib/xen/images
Guidewire Software Inc
Network for the GuestVM
Guidewire Software Inc
Memory and CPU
Do not over allocate as we need resources for the dom0 Host and other GuestVMs
Guidewire Software Inc
Summary of the Options and Resources
Guidewire Software Inc
Vm Installation operations
VMM will start allotting the disk storage Create virtual network Map memory to the Dom0 Virtual memory
map Will start installing the Guest OS
Guidewire Software Inc
PXE install looks at dhcp server
Guidewire Software Inc
Xen Utilities
xm utility - can be used to create domains
- useful for cloning virsh
- can be used to change CPU, memory- define config, dump config in xml form for cloning- start, shutdown and reboot domains
Guidewire Software Inc
Xen Utilities
xentop- similar to top but customized to list all virtual resources and domains
xentop - 21:55:32 Xen 3.1.0-53.el53 domains: 2 running, 1 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdownMem: 15723456k total, 15721284k used, 2172k free CPUs: 8 @ 2992MHz NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPUS
NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID Domain-0 -----r 9576 2.9 13498460 85.8 no limit n/a 8
4 398061 38880 0 0 0 0 0 dtrac41 -----r 2755 2.1 1048152 6.7 1048576 6.7 1
2 883749 857104 0 0 0 0 0 dtrac42 --b--- 1302 8.1 1048080 6.7 1048576 6.7 1
2 850349 813954 0 0 0 0 0
Guidewire Software Inc
XenStore
Configuration database for all domain Multiple commands to manage this database xenstore-ls
- can be used to list details of all hardware components in each domain
Guidewire Software Inc
Add Virtual Hardware Add more storage for Oracle clusterware
Guidewire Software Inc
Add Virtual Storage
Specify Simple file or disk partition (if available) Min size of 256Mb for Clusterware
Guidewire Software Inc
Add virtual network interface For Interconnect use
Guidewire Software Inc
Add virtual network interface Choose virtual network as this interconnect does not need to talk to the
external world
Guidewire Software Inc
Virtual hardware of one guestVM
Guidewire Software Inc
Oracle Install Preparation
Update sysctl.conf with the new kernel parameters rmem_default and wmem_default should be 4MB
min Update limits.conf Install any additional RPMs needed as listed in
Oracle Install documentation Edit /etc/hosts and add localhost and 6 other
addresses to indicate public, private and virtual networks for both guestVMs
Guidewire Software Inc
Create users and groups
groupadd –g 10000 oinstall groupadd –g 10001 dba groupadd –g 10002 asmdba useradd –g oinstall –Goinstall,dba,asmdba –d
/home/oracle –u 10000 oracle passwd oracle
Guidewire Software Inc
Mapping raw devices
Create something like /etc/rc3.d/S90raw and/or /etc/rc5.d/S90raw based on boot run level with contents like:raw /dev/raw/raw1 /dev/xvdbraw /dev/raw/raw2 /dev/xvdcraw /dev/raw/raw3 /dev/xvddraw /dev/raw/raw4 /dev/xvdechown root:dba /dev/raw/raw2chmod 660 /dev/raw/raw2chown oracle:dba /dev/raw/raw1chmod 660 /dev/raw/raw1chown oracle /dev/raw/raw3chown oracle /dev/raw/raw4chmod 660 /dev/raw/raw3chmod 660 /dev/raw/raw4
Guidewire Software Inc
Cloning GuestVM
In dom0, /etc/xen/dtrac41 has list of devices created for this guest VM
Copy /etc/xen/dtrac41 to /etc/xen/dtrac42 Change the mac address of the two virtual network
interfaces in dtrac42 and edit the primary disk Modify the name and uuid for the new VM Ensure the order of the network is right in the config
files Copy the virtual disk of dtrac41(dtrac41.img) to
/var/lib/xen/images/dtrac42.img
Guidewire Software Inc
Config file for guestVM
name = "dtrac41"uuid = "3656ca7c24e378bd2c63d380206a5f8d"maxmem = 1024memory = 1024vcpus = 1bootloader = "/usr/bin/pygrub"kernel = "/var/lib/xen/boot_kernel.qFE_y1"ramdisk = "/var/lib/xen/boot_ramdisk.HlhabB"extra = "ro root=/dev/vgSys/lvRoot rhgb quiet console=tty0 console=ttyS1,57600n8"on_poweroff = "destroy"on_reboot = "restart"on_crash = "restart"vfb = [ "type=vnc,vncdisplay=0,keymap=en-us" ]disk = [ "tap:aio:/var/lib/xen/images/dtrac41.img,xvda,w",
"tap:aio:/var/lib/xen/images/dtrac4x-ocr11g-shared.img,xvdb,w", "tap:aio:/var/lib/xen/images/dtrac4x-vote-shared.img,xvdc,w", "tap:aio:/var/lib/xen/images/dtrac4x-asm1-shared.img,xvdd,w", "tap:aio:/var/lib/xen/images/dtrac4x-asm2-shared.img,xvde,w" ]
vif = [ "mac=00:16:3e:0c:81:f5,bridge=xenbr0,script=vif-bridge", "mac=00:16:3e:75:41:84,bridge=virbr0,script=vif-bridge" ]
Guidewire Software Inc
Cloning GuestVM
Use xm to create the domain
xm create dtrac42 Start the new domain
xm start dtrac42 You should see both the GuestVMs in the
Virtual Machine Manager
Guidewire Software Inc
Setup GuestVMs
Reboot both the virtual machines Ensure fdisk –l is showing all the disks Edit the network configuration files to setup
the IP addresses and hostname and start the network
Ping the IP addresses of the other node and confirm that the network works fine
Guidewire Software Inc
User equivalence Setup
Use ssh-keygen and generate keys Concatenate the public keys for both
machines and store as ~/.ssh/authorized_keys
Check you can login to all combinations without prompts. For e.g.,ssh dtrac41 datessh dtrac42 datessh dtrac41p datessh dtrac42p date
Run cluster verify tool and do pre CRS inst check
Guidewire Software Inc
Vmware
Install Vmware server Create two virtual machines using Vmware
Server Console For the shared disks, edit the config files
(suffixed vmx) and add:disk.locking = "false“
scsi0:1.shared = "true"
scsi0:2.shared = "true"
scsi0:3.shared = "true"
scsi0:4.shared = "true"
Guidewire Software Inc
Oracle install media
Download the clusterware and database software
Unzip the filesunzip linux_11gR1_clusterware.zip
unzip linux_11gR1_database.zip
If using cdrom/dvd, mount the clusterware media first
Start the Installercd clusterware./runInstaller
Guidewire Software Inc
Clusterware Software Home
Guidewire Software Inc
Cluster Configuration Cluster name is dtrac4_cluster Edit details of dtrac41 Add details of dtrac42
Guidewire Software Inc
Network Interface Usage
Select the right Interface for use as Public and Private network Here eth0 is chosen as Public and eth1 as private
Guidewire Software Inc
OCR and Voting disk
Both OCR in the previous installer screen and voting disk here should have min of 256MB space
Specify mirror copies – here we use single voting disk location
Guidewire Software Inc
Clusterware Install Summary
Ensure all cluster nodes are seen
Guidewire Software Inc
Root Scripts
Run orainstRoot.sh and root.sh on all nodes (one at a time)
Guidewire Software Inc
Root.sh on node1 Configures OCR and Voting Disk and starts CRS stack on node1
Guidewire Software Inc
Root.sh on node2 Starts CRS stack on node2 and creates cluster resources
Guidewire Software Inc
Config Assistants for Clusterware
Guidewire Software Inc
Clusterware Install Completion
That completes the clusterware install OCR and Voting disk are configured Resources like ONS and VIP should be created by
now and can be seen using crs_stat Processes like evmd, crsd, ocssd and oprocd should
be running now Ready to Install RAC software
cd database
./runInstaller
Guidewire Software Inc
Installation Type
Guidewire Software Inc
Node Selection Choose all nodes in the cluster – both VMs
Guidewire Software Inc
OS Authentication
Guidewire Software Inc
RAC Install Summary Check for selection of all Cluster nodes
Guidewire Software Inc
Create RAC database
Guidewire Software Inc
Create ASM Disk Group Change disk discovery path and check permissions if disks are
not seen
Guidewire Software Inc
Starts Cluster database Database Creation is complete
Guidewire Software Inc
CRS status
crs_stat -tName Type Target State Host------------------------------------------------------------ora....SM1.asm application ONLINE ONLINE dtrac41ora....41.lsnr application ONLINE ONLINE dtrac41ora....c41.gsd application ONLINE ONLINE dtrac41ora....c41.ons application ONLINE ONLINE dtrac41ora....c41.vip application ONLINE ONLINE dtrac41ora....SM2.asm application ONLINE ONLINE dtrac42ora....42.lsnr application ONLINE ONLINE dtrac42ora....c42.gsd application ONLINE ONLINE dtrac42ora....c42.ons application ONLINE ONLINE dtrac42ora....c42.vip application ONLINE ONLINE dtrac42ora.xenrac.db application ONLINE ONLINE dtrac42ora....c1.inst application ONLINE ONLINE dtrac41ora....c2.inst application ONLINE ONLINE dtrac42
RAC and ASM
11g New Features
Guidewire Software Inc
11g New Features
Automatic Diagnostic Repository should be placed in shared disk if using shared oracle home
Kill sessions from anywhere in the cluster ADDM for RAC – dbms_addm.analyze_db
procedure and dbms_addm.get_report function Inter-node parallelism can use services to determine
the instances that it can use Grid provisioning and EM changes Clone ASM and RAC homes – clone.pl
Guidewire Software Inc
11g ASM New Features
asm_preferred_read_failure_groups parameter ASM rolling migration
– alter system start rolling migration to <pnumber>; Convert single instance ASM to cluster using
rconfig/EM ASM compatibility attributes Variable size extents Higher AU values for Coarse striping 1M – 64M
Guidewire Software Inc
11g ASM New features
Ability to drop ASM instances using dbcadbca –silent –deleteasm –nodelist dtrac41,dtrac42
ASM fast mirror resync SYSASM and OSASM ASMcmd command additions – cp,
md_backup, lsdsk, remap Mount/drop diskgroup with Force option
Cache Fusion
Guidewire Software Inc
Cache Fusion
Shared cache architecture Cache coherency – preserve integrity of data
across different caches for the same data Employs high-speed Interconnects Created in Oracle OPS 8i to reduce pings 8i handled read/write contention 9i RAC handles other contentions like write-
write
Guidewire Software Inc
Cache Fusion
Locks – mode, role, past image
E.g lock : SG1 Modes: Null, Shared, Exclusive Role: Local, Global Past Image component: 0 or 1 Maximum 3-way protocol (master, holder and
requestor)
Guidewire Software Inc
Events
Event 10432 – for diagnosing unexpected GCS events and lock requests
Event 10708 – shows block request and grants (level 7 or 103 is common)
Event 10046 – common sql trace and combines well with event 10708
Guidewire Software Inc
Trace commands
SQL> alter system set events '10432 trace name context forever, level 15:10708 trace name context forever, level 103';
System altered.
SQL> conn scott/tigerConnected.SQL> alter session set events '10046 trace name
context forever, level 8';
Session altered.
Guidewire Software Inc
First Read session
Select on node 1*** 2008-05-15 02:40:36.228kclscrs: req=0 block=5/36 *** 2008-05-15 02:40:36.228
kclscrs: req=0 typ=immediate mode=SWAIT #3: nam='db file sequential read' ela= 786 file#=5 block#=36 blocks=1 obj#=16215 tim=1210844436231630 *** 2008-05-15 02:40:36.231 KCL:
P02: cr prewarm *** 2008-
05-15 02:40:36.231kclscrs: req=0 block=5/28 *** 2008-05-15 02:40:36.231
kclscrs: req=0 typ=immediate mode=S ***
2008-05-15 02:40:36.231kclscrs: req=1 block=5/29 *** 2008-05-15 02:40:36.231
kclscrs: req=1 typ=immediate mode=S…*** 2008-05-15 02:40:36.232kclscrs: req=7 block=5/25
*** 2008-05-15 02:40:36.232kclscrs: req=7 typ=immediate mode=SWAIT #3: nam='db file scattered read' ela= 7342 file#=5 block#=25 blocks=8 obj#=16214 tim=1210844436239474
Guidewire Software Inc
Second read from same node
PARSE #9:c=0,e=1478,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,tim=1210847692139409
EXEC #9:c=0,e=37,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=1210847692139492
WAIT #9: nam='SQL*Net message to client' ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=523 tim=1210847692139514
FETCH #9:c=0,e=29,p=0,cr=2,cu=0,mis=0,r=1,dep=0,og=1,tim=1210847692139568
No communication to the other node
Guidewire Software Inc
Write Operation Tracing..
update emp set sal=sal+100 where empno = 7654END OF STMTPARSE
#3:c=20001,e=44395,p=1,cr=55,cu=0,mis=1,r=0,dep=0,og=1,tim=1210848457075954
WAIT #3: nam='enq: TM - contention' ela= 226 name|mode=1414332419 object #=16214 table/partition=0 obj#=-1 tim=1210848457076262
*** 2008-05-15 03:47:37.076kclscrs: req=0 block=5/36
*** 2008-05-15 03:47:37.076kclscrs: bid=1:3:2:0:2:c:0:0:2:0:0..
Guidewire Software Inc
Tracing Contd..
Lms trace of remote node *** 2008-05-15 03:47:37.077kcl_snd_cur: block=5/36 time=54
Back to update session trace*** 2008-05-15 03:47:37.077kclwcrs: got 1 blocks from ksxprcvimdWAIT #3: nam='gc current block 2-way' ela= 413
p1=5 p2=36 p3=1 obj#=16215 tim=1210848457077694*** 2008-05-15 03:47:37.077kclwcrs: req=0 typ=cur wtyp=2hop tm=489
Guidewire Software Inc
Tracing Contd..
*** 2008-05-15 03:47:37.140kcllkopb: req=7 block=5/25*** 2008-05-15 03:47:37.140kcllkopb: bid=1:3:2:0:2:14:0..WAIT #3: nam='gc current multi block request' ela= 27265
file#=5 block#=25 id#=33554446 obj#=16214 tim=1210848457226586
• Nothing in the LMS trace• Back to sessions trace*** 2008-05-15 03:47:37.254kcllkopb: req=0 typ=cur wtyp=2hop tm=174615WAIT #3: nam='db file scattered read' ela= 83230 file#=5
block#=25 blocks=3 obj#=16214 tim=1210848457337661
Guidewire Software Inc
Causing Write-Write Contention
update emp set sal=sal+100 where empno != 7654
PARSE #5:c=4000,e=49748,p=1,cr=9,cu=0,mis=1,r=0,dep=0,og=1,tim=1210848625816852
*** 2008-05-15 03:50:25.817
kclscrs: req=0 block=5/28
*** 2008-05-15 03:50:25.817
kclscrs: bid=1:3:1:0:2:5:0:0:4:0:..
Guidewire Software Inc
Write-Write Contention Tracing..
Remote LMS*** 2008-05-15 03:50:25.816
kclgcr: req=231 block=5/28
*** 2008-05-15 03:50:25.816
kclgcr: bid=1:3:1:0:2:5:0:0..
*** 2008-05-15 03:50:25.825
kclqsnd: req=231 status=2 fscn=0:b49c2 time=8523
*** 2008-05-15 03:50:25.826
kclpdc: le=1c/50000 rls=49
Guidewire Software Inc
Write-Write Contention Tracing..
Back to local session*** 2008-05-15 03:50:25.825kclwcrs: got 1 blocks from ksxprcvimdWAIT #5: nam='gc cr block busy' ela= 8846 p1=5 p2=28
p3=1 obj#=16214 tim=1210848625825992*** 2008-05-15 03:50:25.825kclwcrs: got 1 blocks from ksxprcvimdWAIT #5: nam='gc cr block busy' ela= 8846 p1=5 p2=28
p3=1 obj#=16214 tim=1210848625825992
*** 2008-05-15 03:50:25.826kclwcrs: req=0 typ=cr wtyp=busy tm=9178
Guidewire Software Inc
Write - Write contention tracing
*** 2008-05-15 03:50:25.826kcllkopb: req=0 block=5/28*** 2008-05-15 03:50:25.826kcllkopb: bid=1:3:1:0:2:6:0:0:
LMS trace*** 2008-05-15 03:50:25.826kcl_snd_cur: block=5/28 time=64
Local Session Trace*** 2008-05-15 03:50:25.827kcllkopb: got 1 blocks from ksxprcvimdWAIT #5: nam='gc current block 2-way' ela= 270 p1=5 p2=28 p3=33554433 obj#=16214
tim=1210848625827163*** 2008-05-15 03:50:25.827kcllkopb: req=0 typ=cur wtyp=2hop tm=417
Q& A