Doc : Dom0 Upgrade Execution Process Version : 1.5 ----------------------------------------------------------------- --------------------------------------------------------- Objective :- This Document describes the steps to be followed to perform the OVMS upgrade or Dom0 upgrade. Pre check to be performed 1 hr before the schedule of Execution 1.Ensure : DBA has taken the HC for the Affected Dom0 ping channel ovms-up-project /mig-os - script will auto mount 3.Mig-os mount source details :- mkdir /mig-os ; mount -o noacl rm02stor29-nas:/export/roh_29a_hwmig/rmdc_dom0 /mig-os mkdir /mig-os ; mount -o noacl adc08ntap17-bkp:/vol/aoh_17a_hwmig/adc_dom0 /mig-os mkdir /mig-os ; mount -o noacl sl05stor02-nas:/export/sloh_02a_hwmig/sldc_dom0 /mig-os mkdir /mig-os ; mount -o noacl tvp01stor05-nas:/export/tvpoh_05a_hwmig/tvpdc_dom0 /mig-os mkdir /mig-os ; mount -o noacl epc002oodstor02-nas:/export/epcoh_02a_hwmig/epdc_dom0 /mig-os mkdir /mig-os ; mount -o noacl syc002oodstor01-nas:/export/sycoh_01a_hwmig/sydc_dom0 /mig-os mkdir /mig-os ; mount -o noacl llg02stor02-bkp:/export/llgoh_07a_hwmig/llg_dom0 /mig-os
This Document describes the steps to be followed to perform the OVMS upgrade or Dom0 upgrade.
Pre check to be performed 1 hr before the schedule of Execution
1.Ensure : DBA has taken the HC for the Affected Dom0 ping channel ovms-up-project
/mig-os - script will auto mount
3.Mig-os mount source details :-
mkdir /mig-os ; mount -o noacl rm02stor29-nas:/export/roh_29a_hwmig/rmdc_dom0 /mig-osmkdir /mig-os ; mount -o noacl adc08ntap17-bkp:/vol/aoh_17a_hwmig/adc_dom0 /mig-os mkdir /mig-os ; mount -o noacl sl05stor02-nas:/export/sloh_02a_hwmig/sldc_dom0 /mig-osmkdir /mig-os ; mount -o noacl tvp01stor05-nas:/export/tvpoh_05a_hwmig/tvpdc_dom0 /mig-osmkdir /mig-os ; mount -o noacl epc002oodstor02-nas:/export/epcoh_02a_hwmig/epdc_dom0 /mig-osmkdir /mig-os ; mount -o noacl syc002oodstor01-nas:/export/sycoh_01a_hwmig/sydc_dom0 /mig-osmkdir /mig-os ; mount -o noacl llg02stor02-bkp:/export/llgoh_07a_hwmig/llg_dom0 /mig-os
4.Validate Prep Data and Current Environment
============== TEMPLATE ==============
SR# :
Source Dom0 :
Target Dom0 :
Source Dom0 VM list :
Nos of VMs from PREP :
Nos of VMs from Portal :
Are the VM names Listed in Prep the same as current VM List [Y/N] :
============== TEMPLATE ==============
Use This as reference through out activity
5.To Take: Source Dom0 – Pre Health checks and paste the output in the SR
Run:
a) /ptsadmin/os_migration/OEL/Dom0_Upgrade_sysinfo_pre_upg_post_checks.sh -o source -o pre -rfc 3-TESTII
Above script covers:
HW_model of the Dom0BIOS version ,ILOM versionunamecat /etc/enterprise-releasemig-os is mountedreset ilom passwordShows Total vcpus of all running VMs ,Shows Total Memory of all running VMsChecking the vm.cfg for vcpu_max parameterCheck if vms are on localdiskIdentify the container to snapEM agent status ,Ovs-agent statusLdap statusIfconfig,bond,MTU,route details
c). Ensure you are able to access the ilom console of the Dom0
d). Ensure that /etc/ovms-provision.conf exists. If it does not, perform the following step:
cat > /etc/ovms-provision.conf << EOF
DATACENTER=<Valid values found in Appendix 7.1 of OMCS_Hypervisor_Provisioning.pdf>
ZONE=<Dom0s Network ZONE>
PROVISIONING_TICKET=<SR/RFC Number>
EOF
vi /etc/ovms-provision.conf
e) Replace the values assigned to DATACENTER, ZONE, and PROVISIONING_TICKET with the relevant information for the patching ticket at hand.
6. Set BO and Disable Pinger
EM : https://omcsem.oracle.com/apex/f?p=200:35:1039424757297901::NO::: locate your EM then set BO
- This will check same Bridges from Source are available on the temp Dom0- Ensure the same container is available.
SA Has to :-
- Ensure VLANs tagged with the same VLANS as Source Dom0 –check the related SR to confirm the VLAN is already tagged.
Search for SR based on <sub> check the SR is closed and included the VLANs in use on the current Dom0 ( based on output captured in the above prep script.
***Proceed to live migrate ,if your answer is yes for the below conditions ****
- the same bridges are on Temp domo - VLAN are tagged - Source container is mounted on temp Dom0 - Source Swap container is mounted on the temp Dom0
Cases where we don’t have any dependency on other SRS for target Dom0 proceed next steps
Else if dependency go with the plan from Lead.
________________________________________________
source pre step has generated the : migVMs.lst file. --- this contains our source vm names
Ex: sample output
Creating the : migVMs.lst file for this 3-11215965617--------------------------------------------54ee4ba4a53a4430ab01b7eec5c3cc92 64bit VMf7e3fd9dd37543dca5fbc1fb6369003d 64bit VM--------------------------------------------
/mig-os/logs/OEL-DOM0_UPG/3-11215965617/migVMs.lst was generatedPlease review it, and modify if necessary
This Dom0's configurationnr_cpu : 32total_memory : 262086free_memory : 7097used memory : 254989=============================================Please login to rmc002oodhost206and run 'xm info' then check'nr_cpu' must be >= 32'free_memory' must be >= 254989'total_memory' should be >= 262086=============================================Ready to start migration [y/N]? y
Migration would take more than hour 256G RAM Dom0Do not interrupt. Be patientReally ready to start migration [y/N]? yxm migrate -l 54ee4ba4a53a4430ab01b7eec5c3cc92 rmc002oodhost206xm migrate -l f7e3fd9dd37543dca5fbc1fb6369003d rmc002oodhost206DoneLooking for any errors above
Monitor Logs on source and target : tail -f /var/log/xen/xend.log
Validate by running xm list so that no vms are there on the source dom0 now.
Access/Verify each VM after migrating to Temp dom0.
2.Ugrade : (this steps performs Disable uptrack and upgrade )
Run:
Time /ptsadmin/os_migration/OEL/Dom0_Upgrade_sysinfo_pre_upg_post_checks.sh -o source -o upgrade -rfc 3-TESTII
Log location for the upgrade: Ex /var/log/od-provision/2015-08-08-20\:40\:13.log
3. Bounce the Dom0
Check grub.conf – we should see the new kernel
Power off the Dom0
Use: /ptsadmin/os_migration/bin/system_restart.pl -s --From consoleTake current power status of host : show /SYS … if should of off Start Dom0 from Console : start /SYS--
4.Once the Dom0 is up run the below
/ptsadmin/ateam/bin/fix_firmware --- (go with defaults )
if it suggests : Firmware upgrade needed ..allow it proceed and perform below steps
b) INFORM A TEAM LEAD (this item is being tracked)
c) Shutdown EM Agent
d) Shutdown Dom0
e) Power off Dom0 host
f) Reboot ILOM by executing reset /SP in an ILOM ssh session
g) Power on Dom0 host and await boot-up
h) rerun : /ptsadmin/ateam/bin/fix_firmware ... it should not show firmware upgrade needed
Post checks on Source Dom0
To Take: Source Dom0 – Post checks
Ensure to Mount : /mig-os
1.
Run:
/ptsadmin/os_migration/OEL/Dom0_Upgrade_sysinfo_pre_upg_post_checks.sh -o source -o post -rfc 3-TESTII
Above script covers : /ptsadmin/ateam/bin/ovms-go_live_check.pl as well
2.Ensure that OVM discovery completes successfully through OVM UI
- You can identify the OVM manager for a Dom0 by running the following as root on the Dom0 (no value means that the Dom0 is not currently managed by OVMM):
-Takes a List of the VMs running on the Source Dom0
-Takes the Total Memory and VCPU used/assigned to the VMS on Source Dom0 -Identify the bridges of the VMs running on the Source Dom0 - Identify the Storage Container and swap container on the Source dom0 - Reset ILOM Password on Source Dom0- check if VM is on local disk
Same Storage and swap container is Available on Target dom0
Show if enough free memory is available for VM migration
Same bridges of source Dom0 Vms are available on the Target Dom0
- In case there are any missing bridges create them and make them up
VLAN Tagging:-
1.Check in portal for vlan tagging SR and confirm the required vlan
Confirm it is completed.
update the vlan tagging page
2.if vlan tagging is not there, then identify all the VM's in a given server pool and get the vlan details of each VM and raise network vlan tagging SR
stepsssh -l admin localhost -p 10000OVM> show serverPool name=rmc002oodpool001OVM> show serverPool name=rmc002oodpool001
update the vlan tagging page
3.IF we find more than Two Bridges for a VM – check it has –rac interface
In that case we need to take ./getnet for –rac also and get it tagged
Run for :
-rac
-vip
-Frontend
Login : hqsun1 - take the vlan id details of the VMs
4. During Prep check the status of : cit-em-agent-12-R4 and try to install
If it failed copy the : cit-em-agent-12-R4.remove from working host as : /usr/local/git/etc/software/cit-em-agent-12-R4.remove
Ex: error message
svc-quicksync Package is requested, but is Current on the hostGITPM Sending inventory data to server.GITPM Finished.==============================================================================Current package(s) 50: git-perl, git-perl-addon, git-perl-addon-yaml, git-cron, cit-chmod-r, wget, cit-powerbroker, cit-em-blackout-cmd, cit-ohs-account-locker, cit-oscc-update, git-access-tools, git-backup-common, git-transport-client, git-backup-notify, git-password-manager, git-baton, git-commander-client-v2, git-os-update, git-dell-remotecard-config, git-discovery, git-em-addon, git-em-blackout-cmd, git-java-rpm, git-openssh-config, git-lockdown, git-mktemp, git-network-setup, git-odmagent-dump-cleanup, git-vm-detect, git-package-update, git-perl-addon-rotate, git-perl-nls-libs, git-qualys-client, git-sim-syslog, git-sysinfo, ims-perl-libs, ims-cemlitools, ims-remote-logging, ops-db-recovery, ops-python, ops-denyhosts, ops-os-watcher, ops-oshc, ops-trimbox, sms-db-space-cleanup, sms-em-addon, spt-rda, svc-em-addon-ocm, svc-onrequest-ccr-collection, svc-quicksyncFailed package(s) 1: cit-em-agent-12-R4ERROR : After 3 tries, unable to install/upgrade all required packages.ERROR : Remaining to Install : 1ERROR : Remaining to Upgrade : 0You have new mail in /var/spool/mail/root[root@epc002oodhost013 ~]#
5. Run : /usr/local/git/bin/host-group -f all make sure service_code = none
If it is showing : ebso pod etc update ITAS
go to ITAS -- > Asset Details -- > change service_code to None
it will take 15 min to pushup APS
6. Check the Business_area of the host must be : commercial any other inform Lead
EX: [root@rmc002oodhost438 ~]# /usr/local/git/bin/host-group -f all | grep -i business_area
business_area: commercial
7. check ILOM DNS is resolving fine with correct format
Refer below:-
ILOM_NAME="${DATACENTER}-mgmt-${SERIAL}.${DOMAIN_NAME}" # ADC Zone 7 # ADC Zone 8 # ADC Zone 9 # ADC Zone 15 # ADC Zone 26 # ADC Zone 31 # ADC Zone 32 # ADC Zone 33 # ADC Zone 34 # ADC Zone 36 # ADC Zone 38 # ADC Zone 39 # ADC Zone 40 # RMDC Zone 2 # RMDC Zone 5 # RMDC Zone 6 # LLG Zone 1 # LLG Zone 2 # LLG Zone 6 # LLG Zone 7 # LLG Zone 3 # TVP Zone 1 # TVP Zone 2 # SLDC Zone 4 # SLDC Zone 5
ILOM_NAME="${HOSTNAME}-ilom.${DOMAIN_NAME}" # ADC Zone 42 # ADC Zone 43 # ADC Zone 44 # ADC Zone 45 # ADC Zone 46 # ADC Zone 47
# ADC Zone 48 # ADC Zone 49 # ADC Zone 50 # ADC Zone 51 # ADC Zone 52 # ADC Zone 53 # ADC Zone 54 # ADC Zone 55 # ADC Zone 56 # ADC Zone 57 # ADC Zone 58 # ADC Zone 59 # ADC Zone 60 # ADC Zone 70 # LLG Zone 10 # LLG Zone 11 # SLDC Zone 6 # SLDC Zone 7 # SLDC Zone 8 # SLDC Zone 9 # SLDC Zone 10 # SLDC Zone 11 # SLDC Zone 12 # SLDC Zone 13 # SLDC Zone 14 # SLDC Zone 40 # SYDC Zone 1 & 2 # SYDC Zone 3 # EPDC Zone 2 # TRDC Zone 1 # TRDC Zone 2
