1. Virtualizing Mission Critical Oracle RAC with vSphere and vCOPS Steven Jones, VMware Charles Kim, Viscosity North America Kannan Mani, VMware George Trujillo, Hortonworks VAPP5834 #VAPP5834

2. iBook available: Free for session attendees ONLY Please email ckim@viscosityna.com for the URL 3. Oracle ACE Director VMware vExpert Over 22 years of Oracle Expertise, Exadata, RAC, Data Guard, ASM, RMAN, Shareplex / Golden Gate IOUG Cloud Computing SIG Executive Committee Blog Site: http://oravm.com + http://dbaexpert.com Oracle Certified RAC Expert Charles Kim @racdba 4. George Trujillo Master Principal Big Data Specialist - Hortonworks Tier One BigData, Oracle and BCA Specialist - VMware Oracle Double ACE 20+ years Oracle DBA: DW, BI, RAC, Streams, Data Guard, Perf, B/R Sun Microsystem's Ambassador for Application Middleware Oracle Fusion Council & Oracle Beta Leadership Council Two terms Independent Oracle Users Group Board of Directors Recognized as one of the Oracles of Oracle by IOUG MySQL Certified DBA VMware Certified Instructor (VCI) 5. VMware 4X and 5.X Certified Professional VCI VMware Certified Instructor CompTIA Cloud Certified Principal Instructor Oracle Corporation Master Consultant Distinction - Sun Microsystems Solaris 8 , 9, 10 Certified System Administrator Solaris 8,9 Certified Network Administrator Certified MySQL DBA Steven Jones 6. 6 Kannan Mani 15+ years Oracle experience : Oracle RAC, ASM, Clustering, CRM, ERP, Business Intelligence, Performance and Scalable Enterprise Application Architecture, Benchmark and Performance, Technical solutions marketing and management, Virtualization and Cloud solutions Oracle ACE Applications, DB Speakers @ Oracle Open World, IOUG, VMworld, VMware Partner Exchange, EMC World and Webinars Industry recognized expert in Oracle and Virtualization technologies. Blog: http://blogs.vmware.com/apps/oracle 7. 7 Agenda Business Drivers Best Practices for Virtualizing Oracle Oracle 12c Rapid RAC Deployment Creating and preparing our Golden Image Template Cloning RAC Clusters Adding and Removing node(s) to/from the cluster vCOPs Metrics and Monitoring 8. 8 Business Drivers Leveraging the advantages of virtual servers over physical servers. Virtual infrastructure advantages: Availability, Platform Consistency, Flexibility, Speed, Elasticity, Networking Standardizing on a Single Common Hardware Platform (software stack) Reduction of infrastructure costs (CapEx and OpEX) Enabling Oracle as a service in a public or private cloud Movement towards the Software Defined Data Center 9. 9 Evolution of the DBA Kind of DBA Timeline CLI DBA Early 90s DBAs GUI DBA Late 90s and Dot Com Google DBA Dot Com and 2000s iDBA Dot Com, IOUG iDBA Master Curriculum RAC DBAs 2000+ after 9.2 (but major spike with 10.2) DMA 2010+ Database Machine Administrator vDBA / vRAC DBA 2010+ Evolving role of a DBA in the virtual world Cloud DBA 2011+ Database Consolidation with Private Database Cloud Oracle Database 12c Launches June 2013 10. Agenda 10 Business Drivers Best Practices for Virtualizing Oracle Oracle 12c Rapid RAC Deployment Creating and preparing our Golden Image Template Cloning RAC Clusters Adding and Removing node(s) to/from the cluster vCOPs Metrics and Monitoring 11. 11 Use10GigE network is active/passive Should sustain the interconnect traffic Use RAC Services placement to limit internode traffic Use Jumbo frames for private interconnect network for payload efficiency Use VMXNET3 Disable NIC interrupt coalescing for RAC private interconnect Resource Sizing Network 12. 12 Use Thick Eager-zero disks as first priority Use RDMs as a last resort Set the memory reservation equal to (or slightly higher than) the size of the Oracle SGA Use Para-virtualized SCSI adapters for demanding workloads. Use VMware Virtual Hardware version 9 Do not over-commit CPU for RAC databases Do not over-commit memory VM Recommendations Disable devices that are not needed in the VM Disable processes that are not needed for a database server Use Huge pages in the VM ESXi General Recommendations VM 13. Agenda 13 Business Drivers Best Practices for Virtualizing Oracle Oracle 12c Rapid RAC Deployment Creating and preparing our Golden Image Template Cloning RAC Clusters Adding and Removing node(s) to/from the cluster vCOPs Metrics and Monitoring 14. 14 VM Template - Red Hat 6.4+ ASMLIB Support As of May 2013 ASMLib requires kmod-oracleasm (RHEL Supplementary Channel), oracleasmlib and oracleasm-support packages Convert from udev to ASMLIB http://dbaexpert.com/blog/ 15. 15 VM Template - ssh setup The Alternative Way http://www.dbaexpert.com/blog/ssh-setup-the-oracle-way/ http://www.dbaexpert.com/blog/sshsetup-on-a-7-node-rac/ $ pwd /nfs/software/12c/grid/sshsetup $ ./sshUserSetup.sh -user oracle -hosts "rac1 rac2"-noPromptPassphrase -advanced exverify The output of this script is also logged into /tmp/sshUserSetup_2013-07-09-16-31-50.log Space 16. 16 VM Template - OS Watcher The Alternative Way Install OS Watcher Service RPM: sudo rpm -ihv oswbb-service-1.1.5-1.noarch.rpm Modify /etc/oswbb.conf # Set OSW_HOME to the directory where you unpacked OSW or OSWbba OSW_HOME='/u01/app/oracle/oswbb' # Set OSW_INTERVAL to the number of seconds between collections OSW_INTERVAL='30' # Set OSW_RETENTION to the number of hours logs are to be retained OSW_RETENTION='168' # Set OSW_USER to the owner of the OSW_HOME directory OSW_USER='root' # Set OSW_COMPRESSION to the desired compression facility OSW_COMPRESSION='gzip http://www.dbaexpert.com/blog/the-better-way-to-configure-oswatcher/ How To Start OSWatcher Black Box (OSWBB) Every System Boot [ID 580513.1] Start OSWatcher: sudo /sbin/service oswbb start Setup Auto-Start: sudo /sbin/chkconfig on 17. 17 Prepare Grid and Oracle Home Unlock the Grid Home # cd $GRID_HOME/crs/install # perl rootcrs.pl unlock crshome $GRID_HOME Create TAR ball of GI Home Exclude directories, logfiles, text files # export GRID_HOME=/u01/app/grid/12.1.0 # cd ${GRID_HOME} # tar cpfX - ${EXCLUDE_LIST} * > ${NFS_DIR}/Grid.tar ${GRID_HOME}/host_name ${GRID_HOME}/rdbms/audit/* ${GRID_HOME}/log/host_name ${GRID_HOME}/gpnp/host_name ${GRID_HOME}/log/diag/asmcmd/* ${GRID_HOME}/log/diag/tnslsnr/* ${GRID_HOME}/ccr/hosts/* ${GRID_HOME}/cdata/* ${GRID_HOME}/crs/install/*.txt ${GRID_HOME}/evm/log/* ${GRID_HOME}/install/*.log ${GRID_HOME}/network/admin/*.ba k.* ${GRID_HOME}/network/admin/*.ora ${GRID_HOME}/ohasd/init/* ${GRID_HOME}/root.sh ${GRID_HOME}/*ouibak* *.ouibak *.log 18. 18 Agenda Business Drivers Best Practices for Virtualizing Oracle Oracle 12c Rapid RAC Deployment Creating and preparing our Golden Image Template Cloning RAC Clusters Adding and Removing node(s) to/from the cluster vCOPs Metrics and Monitoring 19. Execute root.sh on each node * All the Magic happens here Configure Cluster Stack * Spend all the time here performing automation Execute root.sh on each node Clone Grid Home on each node Prepare Source Environment (Exclude Misc Dirs and Logs) 19 Clone Grid Home Topology Un-TAR Grid TARUn-TAR Grid TAR 20. 20 Clone Database Home Topology Create Golden Image Database Un-TAR DB TAR Execute root.sh on each node * All the Magic happens here Configure Cluster Stack * Spend all the time here performing automation Execute root.sh on each node Clone Grid Home on each node Prepare Source Environment (Exclude Misc Dirs and Logs) Un-TAR Grid TARUn-TAR Grid TAR Clone DB Home on each node Execute root.sh on each node Create RAC DB from Golden Image 21. 21 Clone Grid Home on Each Node cd ${GI_HOME}/clone/bin perl ${GI_HOME}/clone/bin/clone.pl -silentORACLE_HOME=${GI_HOME}ORACLE_HOME_NAME=OraGridHome1ORACLE_BASE=$ORACLE_BASE-O'"CLUSTER_NODES={rac1,rac2,rac3}"'-O'"LOCAL_NODE=$(hostname -s)"'CRS=TRUE INVENTORY_LOCATION=${INVENTORY_LOCATION} 22. 22 config.sh in GUI mode 23. 23 config.sh in silent mode export DIR=$(pwd) FN=`echo $0 | sed s/.*[/]//` ${GRID_HOME}/crs/config/config.sh -silent -responseFile${DIR}/${CLUSTER_NAME}.rsp-ignorePrereq -ignoreSysPrereqs 24. 24 Clone DB Home on Each Node JAVA_HOME=${ORACLE_DB_HOME}/jdk/bin/java PATH=$PATH:${JAVA_HOME} cd ${ORACLE_DB_HOME}/clone/bin perl ${ORACLE_DB_HOME}/clone/bin/clone.pl -silentORACLE_HOME=${ORACLE_DB_HOME} ORACLE_HOME_NAME=OraDBHome1ORACLE_BASE=$ORACLE_BASE-O'"CLUSTER_NODES={rac1,rac2,rac3}"'-O'"LOCAL_NODE=$(hostname -s)"' CRS=TRUEINVENTORY_LOCATION=${INVENTORY_LOCATION} 25. 1. Create Golden Image DB Template export BKUP_DIR=/bkups/templates dbca -silent -createCloneTemplate-sourceSID RACDB-templateName db_gold_01-sysDBAUserName sys-sysDBAPassword oracle123-maintainFileLocations false-datafileJarLocation $BKUP_DIR 2. Locate the the tempate xml file in the format of .dbc $ cd $ORACLE_HOME $ find . -name '*.dbc'./assistants/dbca/templates/General_Purpose.dbc./assistants/dbca/ templates/Data_Warehouse.dbc./assistants/dbca/templates/db_gold_01.dbc 3. Copy to the target $ORACLE_HOME/assistants/dbca/templatesdb_gold_01.dbc -- to /bkups/templates (or NFS)$ ls -l /bkups/templatestotal 282784-rw-r----- 1 oracle oinstall 18497536 May 8 08:37 gb_gold_01.ctl-rw-r----- 1 oracle oinstall 270778368 May 8 08:37 gb_gold_01.dfb 25 Source DB dbca backup dbca cold backup piece Copy .dbc file to $ORACLE_HOME on Target Copy dbca backup to target Create symbolic link to NFS Mount Create Golden Image Template of Database 26. 26 Clone Database From Template dbca -silent-createDatabase-templateName db_gold_01.dbc-gdbName racdb-sid racdb-SysPassword oracle123-SystemPassword oracle123-emConfiguration NONE-redoLogFileSize 500-recoveryAreaDestination FRA-storageType ASM-asmSysPassword oracle123-diskGroupName DATA-listeners LISTENER_RAC-characterSet AL32UTF8-nationalCharacterSet AL16UTF16-databaseType MULTIPURPOSE-nodelist rac1,rac2,rac3-initparams audit_file_dest=/oraacfs/trace/racdb/adump'-initparams compatible='11.2.0.3'-initparams db_create_file_dest='+DATA'-initparams db_create_online_log_dest_1='+DATA'-initparams db_create_online_log_dest_2='+FRA'-initparams diagnostic_dest=/oraacfs'-initparams parallel_max_servers=32-initparams pga_aggregate_target=524288000-initparams processes=400-initparams sga_target=4294967296-initparams db_recovery_file_dest='+FRA'-initparams db_recovery_file_dest_size=2097152000 Source DB dbca backup Target DB restore/config from dbca dbca cold backup piece dbca cold backup piece + template .dbc file Copy .dbc file to $ORACLE_HOME on Target Copy dbca backup to target Create symbolic link to NFS Mount 27. 27 Putting it all together How do we do it: Clone GRID Homes on all RAC nodes Template script and replace Homes and Nodes Setup 2 Configuration files 1. Primary config file that has only the relevant information (Hostname, VIPs, SCAN, Private Subnet, Listener Name, DB Name) 2. Generic config file that applies to every build (OFA related directories, DB Home, Grid Home, Oracle Base, OraInventory Loc, etc) Generate Response File and tag PLACEHOLDERS within the response file Globally search and replace PLACEHOLDERS Clone DB Homes on all RAC nodes Template script and replace Homes and Nodes 28. 28 Agenda Business Drivers Best Practices for Virtualizing Oracle Oracle 12c Rapid RAC Deployment Creating and preparing our Golden Image Template Cloning RAC Clusters Adding and Removing node(s) to/from the cluster vCOPs Metrics and Monitoring 29. 29 Add Node 4 Step Process 1. From one node (First Node) export ORACLE_HOME=/u01/app/11.2.0/grid export PATH=$ORACLE_HOME/bin:$PATH export IGNORE_PREADDNODE_CHECKS=Y ${ORACLE_HOME}/oui/bin/addNode.sh -silent "CLUSTER_NEW_NODES={rac4}""CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac4-vip}""CLUSTER_NEW_PRIVATE_NODE_NAMES={rac4-priv} -noCopy 2. Target Node Only: cd /u01/app/oracle/product/11.2.0/db/clone/bin perl /u01/app/oracle/product/11.2.0/db/clone/bin/clone.pl -silentORACLE_HOME=/u01/app/oracle/product/11.2.0/db ORACLE_HOME_NAME=OraDBHome1ORACLE_BASE=/u01/app/oracle -O'"CLUSTER_NODES={rac1,rac2,rac3,rac4}"'-O'"LOCAL_NODE=$(hostname -s)"' CRS=TRUE INVENTORY_LOCATION=/u01/app/oraInventory} 3. Original Nodes: /u01/app/oracle/product/11.2.0/db/oui/bin/runInstaller -updateNodeList CLUSTER_NODES=rac1,rac2,rac3,rac4ORACLE_HOME=/u01/app/oracle/product/11.2.0/db -local 30. 30 Add Instance (Easy Way) Last Step export ORACLE_SID=racdb1 export ORACLE_HOME=$ORACLE_DB_HOME export PATH=$ORACLE_DB_HOME/bin:$PATH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH dbca -silent -addInstance -gdbName racdb -nodelist rac4 -instanceName racdb4-sysDBAUsername sys -sysDBAPassword oracle123 srvctl config database d racdb 31. 31 Delete Instance 3 Step Process (Fist Step) 1. export ORACLE_SID=racdb1 export ORACLE_HOME=$ORACLE_DB_HOME export PATH=$ORACLE_DB_HOME/bin:$PATH export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH dbca -silent -DeleteInstance -gdbName racdb -nodelist rac4-instanceName racdb4 -sysDBAUsername sys -sysDBAPassword oracle123 srvctl config database -d racdb 32. 32 Delete Node 2. From one of the nodes: export ORACLE_SID=+ASM1 export ORAENV_ASK=NO . oraenv export ORACLE_HOME=$ORACLE_CRS_HOME export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH ${ORACLE_CRS_HOME}/bin/crsctl delete node -n ${NODE_TO_BE_DELETED} 3. Remaining nodes: /u01/app/11.2.0/grid/oui/bin/runInstaller -updateNodeList CLUSTER_NODES=rac1,rac2ORACLE_HOME=/u01/app/11.2.0/grid -local /u01/app/oracle/product/11.2.0/db/oui/bin/runInstaller -updateNodeList CLUSTER_NODES=rac1,rac2ORACLE_HOME=/u01/app/oracle/product/11.2.0/db -local 33. 33 Agenda Business Drivers Best Practices for Virtualizing Oracle Oracle 12c Rapid RAC Deployment Creating and preparing our Golden Image Template Cloning RAC Clusters Adding and Removing node(s) to/from the cluster vCOPs Metrics and Monitoring 34. 34 Easy? Partial information can often lead to inaccurate conclusions and results. We Need Easy visibility into the entire stack from the application level to the guest operating system and underlying virtual layer. 35. 35 Zombie Sharks! Easy Oracle RAC day to day? Color? Number? Type? Consistent Clear Critical 36. 36 vCOPs Mgr. Architected for Ease 37. 37 Easy to Answers the Big Questions 38. 38 Pervasive or Isolated? Easy Access End to End Depth of Detail-Root Cause 39. 39 1. 2. 3. Root Analysis Gives Oracle DBAs Easy Resolutions 40. 40 Lets Review Reduced time for troubleshooting Visibility to entire environment Improved monitoring efficiency Proactive response to problems Easy Management of Oracle RAC !! 41. 41 vCenter Operations Can Easily Reach Into Oracle 42. 42 Performance Management vCenter Operations Manager(OEM Adapter) 43. 43 Monitoring: VMware vCenter Operations Enterprise Edition Benefits: Provides Operational status and the ability to drill down on a data center, cluster, VM & storage Identifies waste : Provides workload usage to application owners requesting unnecessary CPU & memory Health check with Short & Long Term Capacity Planning Oracle Enterprise Manager Adapter Single pane view for DBA and VI- Admins 44. 44 vC OPs Oracle OEM Adapter High Level Architecture OpenVPN Postgres DB vSphere WebApp Custom WebApp Admin WebApp vCenter Operations Manager vApp UI VM Rolled up capacity data Capacity Analytics FSDB Postgres DB Collector ActiveMQ Performance Analytics Analytics VM Metric Data Oracle Database Virtual Environment vSphere vC Ops Mgr vSphere UI Oracle OEM Adapter vCenter Communications over SSL vC Ops Mgr Custom Dashboard Oracle jdbc:oracle:thin:@host:port:SID 45. 45 Install and Configure Oracle Enterprise Manager Adapter https:///admin: upload .pak https:///custom : Add a Credential Add a adapter instanceValidate 1 2 34 46. 46 vCOPS Custom Interface Reached by https:///vcops-custom Enabled by the Enterprise or Enterprise Plus Key Same Data as vCOPS-vSphere Independent Access Control from VCOPS-vSphere (Configured later) Add Adapters Modify KPIs Create Relationships Create Super Metrics Create Dashboards Access Control 47. 47 Create a Custom Oracle Admin Dashboard Three Column Layout Drag the Required Widget 48. 48 Configuring Metric Graph Widget CPU metrics from OEM Adapter Widget Title Select OEM Resources Selected Resource List Select Metrics Here Selected Metrics Added Here 49. 49 Configuring Scoreboard Widget Memory metrics from OEM Adapter Widget Title Select OEM Resources Selected Resource List Select Metrics Here Selected Metrics Added Here Specify Ranges Here 50. 50 Oracle Database Final Dashboard Overview 1 2 3 4 5 6 51. 51 Oracle Database Final Dashboard Overview 1 2 3 4 5 6 Health tree shows all the Oracle DB Server related components 52. 52 Oracle Database Final Dashboard 1 2 3 4 5 6 Cache Hit Ratio Dropped CPU Utilization and Physical I/O Read Similar during Consistent Workload 53. 53 Oracle Enterprise Manager All Metrics DB Instance PRD All Metrics Download http://goo.gl/29IMHaMonitoring Business Critical Applications with VMware vCenter Operations Manager 54. 54 PLACEHOLDER - vCOPs Plug-in for OEM 12c Cloud Control 55. 55 Questions Email ckim@viscosityNA.com Charles @racdba 56. 56 More detailed information is available at viscosityna.com or by talking to a real person at 469.444.1380 @ViscosityNA 57. 57 Other VMware Activities Related to This Session HOL: HOL-SDC-1304 and HOL-SDC-1317 vSphere Performance Optimization vCloud Suite Use Cases - Business Critical Applications Group Discussions: VAPP1007-GD Oracle/Storage with Mark Achtemichuk 58. THANK YOU 59. Virtualizing Mission Critical Oracle RAC with vSphere and vCOPS Steven Jones, VMware Charles Kim, Viscosity North America Kannan Mani, VMware George Trujillo, Hortonworks VAPP5834 #VAPP5834