MO007185 - T1000 EPAP a - Dual Disk Replacement From 120GB to 250GB or 500GB Nov-2011

Tekelec 5200 Paramount ParkwayMorrisville, NC 27560888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, © TEKELEC 2007.

Title: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB

Doc Number: MO007185 Revision: 1.0

TEKELECMethod of Procedure

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual DiskReplacement from 120GB to 250GB or 500GB


Doc No.: MO007185 Revision: 1.0Page 1 of 25



TEKELEC METHOD OF PROCEDURET1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB,

MO007185 General X Detailed

Address: Office(s): Date:

Start Date: Time: Completion Date: Time: General Description of Work: For dual disk replacement, run HDA and HDC replacement in any order and then execute the “PARTITION EXPANSION” procedure.

***Make note of which hard disk is failed and replace the failed hard disk first!!!!!!!

Additional Comments: Please review this MOP thoroughly, then sign this sheet and fax to 919-388-1384, ATTN: Ken Abbott, GlobalTAC Manager. This MOP must be signed before starting the maintenance activity for <Customer Name>.

***If this is the Active provisioning site, provisioning must be stopped and remain stopped until the MOP has been completed!!!

Attachments: NoneIn Attendance Title Company Telephone #

Changes Required by Customer: ( ) Yes ( ) NoThis job has been reviewed and agreement reached.Responsibility for this job has been assigned to (Please Print):

MOP Prepared by: Nitin Garg Date: June 16, 2011MOP Revisions by: Tony Fish Date: November 22, 2011MOP Tested by: Date: MOP Test Comments: N/A

No changes shall be made without the approval of the customer representative and the Tekelec representative.

***ESCALATION CALLOUT LIST***For any anomaly that results from any step in this, Method Of Procedure, halt the process immediately. Escalate to the 1st authority from both Tekelec and the customer as listed below. Both parties must agree to problem resolution, and whether to proceed with the procedureTEKELEC CONTACT / NUMBER CUSTOMER CONTACT / NUMBER Preliminary TEKELEC Review by: Title: Date:





Concurred in by TEKELEC: Title: Global TAC Manager Date:

Approved by Customer Representative: Title: Date:

Approved by Customer Representative: Title: Date:

General – For dual disk replacement, run HDA and HDC replacement in any order and then execute “PARTITION EXPANSION” procedure. If one of the hard disk is faulty, the faulty hard disk MUST be replaced first!

PREPARE THE SYSTEM

STEP#

Steps to be completed.This procedure outlines the steps for replacing both hard drives with new 500 GB drives on EPAP A running on T1000 hardware.

1 Prepare system Prior to work starting, Customer should: - If this is the Active provisioning site, the customer must stop the Provisioning system at this time. - If this is a provisioning site, perform PDBA Backups - Perform RTDB Backups - Turn off Automatic Backups

2 Log into the MPS A as the root user.

login: rootPassword: <Enter root password>

3 Verify output Should see:Last login: <date> from <hostname>

4 Block Provisioning

(Step should be executed only in case of prov Server)

Run uiEdit command on ACTIVE node. # uiEdit PDBA_REMOTE_PDBI_ALLOWED NO

5 Perform system check on MPS A. The syscheck utility should not report any problem other than a failed hard disk.

# syscheck

***Make note of which hard disk is failed and replace the failed hard disk first!!!!!!!

6 Verify mirror status is # cat /proc/mdstat





good***Make note of which hard disk is failed and replace the failed hard disk first!!!!!!!

7 Verify all meta devices are in sync

Should not see “recovery” or “resync”. Each md should show “active raid1” and “[2/2] [UU]”.

8 Stop Pdba software

(Step should be executed only in case of prov Server)

# cd /etc/init.d# ./Pdba stop

9 Verify output Should see:PDBA application stopped.

10 Stop Epap software # ./Epap stop

11 Verify output Should see:EPAP application stopped.

12 Move Pdba and Epap executables to Pdba_stopped and Epap_stopped

# mv Epap Epap_stopped# mv Pdba Pdba_stopped





HDC REPLACEMENT

1 Prepare system Please execute “PREPARE THE SYSTEM” procedure

2 Enter BIOS Upon restarting the server, enter the BIOS setup utility by pressing the DEL key when the message “Press DEL to run Setup (F4 on Remote keyboard)” is displayed.

If using a PS2 keyboard (preferred method) the DEL key must be pressed immediately after the 3 keyboard LEDs light during the boot process.

NOTE: This message is only displayed for a few seconds and the DEL or (F4 on Remote keyboard) key must be pressed before the message disappears from the screen or the boot sequence will continue as usual.

The BIOS setup utility provides navigational tips which are displayed in the lower right hand corner of the screen, but in general moving around within the utility is accomplished using the right, left, up, and down arrow keys. To select the highlighted option, simply press the Enter key, and to return to the previous screen press the ESC key.

If at anytime your output is garbled, in SecureCRT select Reset from the Edit menu.

3 Select Hard Disk Drives in the Boot Settings screen.

Once inside the BIOS setup utility verify that the disk drive that is not being replaced is set to be the boot disk. The setting of interest is on the Boot settings screen within the BIOS setup utility. On the Boot settings screen select Hard Disk Drives as shown below.

BIOS Boot Settings Screen





4 Verify that hda is listed as the 1st drive.

After selecting Hard Disk Drives, modify the drive order (the order of the entries can be reversed by pressing either the + or - key) to make the SM drive (hdc) the 1st Drive.

NOTE: The device names listed on the Hard Disk Drives screen are the model number of the server’s disk drives with one entry prepended by “PM-” (Primary Master) HDA and the other by “SM-” (Secondary Master) HDC.

BIOS Hard Disk Drives Screen





5 Save any changes If this setting requires modification make the necessary change, (the order of the entries can be reversed by pressing either the + or - key), and then save the CMOS data by pressing F10, and then selecting OK when the confirmation dialog is presented.

If the order shown is correct, simply press the ESC key or select Discard Changes and Exit on the main Exit screen and select OK when asked to confirm the exit.

6 After exiting the BIOS setup utility, the server will restart. Once the login prompt is presented, log in as the root user.

Login: rootPassword: <Enter root password>


8 Perform system checks on MPS-A

# syscheck

9 Verify no syscheck failures other than due to Pdba and Epap software being down.

The only failures should be in class proc module run due to the Pdba and Epap software being stopped. There will also be errors in connecting to the maint process and using the alarm dispatch utility.

10 Verify mirror status is good

# cat /proc/mdstat





Personalities : [raid1] read_ahead 1024 sectorsEvent: 11 md12 : active raid1 hda12[0] hdc12[1] 204242240 blocks [2/2] [UU] md11 : active raid1 hda11[0] hdc11[1] 15880128 blocks [2/2] [UU] md10 : active raid1 hda10[0] hdc10[1] 4208896 blocks [2/2] [UU] md9 : active raid1 hda9[0] hdc9[1] 538048 blocks [2/2] [UU] md8 : active raid1 hda8[0] hdc8[1] 4208896 blocks [2/2] [UU] md7 : active raid1 hda7[0] hdc7[1] 538048 blocks [2/2] [UU] md6 : active raid1 hda6[0] hdc6[1] 1060160 blocks [2/2] [UU] md5 : active raid1 hda5[0] hdc5[1] 4208896 blocks [2/2] [UU] md3 : active raid1 hda3[0] hdc3[1] 2056256 blocks [2/2] [UU] md2 : active raid1 hda2[0] hdc2[1] 538112 blocks [2/2] [UU] md1 : active raid1 hda1[0] hdc1[1] 272960 blocks [2/2] [UU] unused devices: <none>



12 Execute failDisk script # /usr/TKLC/plat/sbin/failDisk --splitOnly /dev/hdc

13 Verify output Should see:Removal of /dev/hdc completed, OK.

14 Shut down MPS-A # init 0

15 Verify output Should see:





Power down.

16 Remove power from MPS A.

If the breakers are clearly marked, power off the two corresponding breakers, otherwise remove the two power cables on the rear of MPS A.

17 Remove the /dev/hdc drive Hard drive C is on the right side of the lid. (see T1000 Hardware Manual)

18 Set jumper on new drive to match original drive.

Note that you’re trying to match (connect mode) not actual pins. Thus, comparison of the jumper diagrams will be necessary.

19 Install new drive Reconnect cables.

20 Disconnect /dev/hda Disconnect the ribbon and power cables from the good (/dev/hda) drive - hard drive A is on the left side of the lid.

The Purpose is to see that system if system is booting from HDC? If system boots from HDC drive, then we need to remove the MBR from this.

21 Close lid Close and secure lid, and if necessary slide server into rack and secure mounting brackets.

22 Apply serial number labels We should clearly mark the label on the new hard disks that we have installed. The Label should tell if the hard drive is ‘HDA’ or ‘HDC’.

Just in case there is not already a label on the removed disks, please apply label on them too.

23 Restore power to MPS A. Either re-apply power at the breaker panel or restore the power cable connections on rear of MPS A.

24 Determine if clearing of MBR is needed

If system does not boot, skip to step 30. If the system boots from the hdc drive, proceed to next step to clear out the Master Boot Record and meta device information.

25 Log on to MPS–A as root


27 Clear the Master Boot Record and meta device information from hda.

# dd if=/dev/zero of=/dev/hdc bs=512 count=1

28 Reboot MPS-A # init 6

29 Go to step 24







31 Reconnect /dev/hda Reconnect the ribbon and power cables on the good (/dev/hda) drive - hard drive a is on the left side of the lid.





36 Change runlevel to 3 in order to speed resync

# init 3

37 Verify output Should see:Messages of services stopping.

38 Verify runlevel Hit <Enter> to get prompt, and then:# runlevel

39 Verify output Should see:4 3

40 Partition new disk # /usr/TKLC/plat/sbin/cpDiskCfg hda hdc

41 Verify output Should see:WARNING: Supplied disks do not have the same model number!Disk Geometry Check, OK!Partitioning of disk ‘/dev/hdc’ completed, OK!

42 For safety, have the kernel reread the partition table

# partprobe

43 Ensure maximum speed of resync is possible

# cd /proc/sys/dev/raid/

44 Speed should be 900000. # cat speed_limit_max

If value is not 900,000 set it to 900,000:# echo 900000 > speed_limit_max

45 Re-mirror new disk # /usr/TKLC/plat/sbin/mdRepair –background

Note - Step takes several minutes.46 Verify output Should see:

Critical repairs have been made, backgrounding as requested.





47 Check progress of resync # grep ‘recovery\|resync’ /proc/mdstat

Note - Keep waiting here unless mirror status is good.48 If any output go to step 45. If there is any output, this would mean that Mirrors are not yet joined. We

need to wait for some time. If you notice that mirroring is still progressing, then wait here or else go back to the step that was issued to join to mirrors.

49 When resync completes, perform system checks on MPS-A

# syscheck



51 Move Pdba_stopped and Epap_stopped executables back to Pdba and Epap

# cd /etc/init.d# mv Epap_stopped Epap# mv Pdba_stopped Pdba

52 Change runlevel to 4 in order to bring application software up

# init 4

53 Verify output Should see:Messages of services starting.



56 Perform system checks on MPS A

# syscheck

57 Verify no syscheck failures

Should NOT get this output:One or more module in class "<class>" FAILED

58 Verify PDB, RTDB, and DSM statuses are good.

From GUI: use View RTDB Status, View PDBA Status, Run Syscheck, Capture Logs.From Eagle: use rept-stat-sccp, Rept-stat-db:display=all:db=mps, etc.

59 Have customer restart provisioning and verify DB statuses.

Run uiEdit command on ACTIVE node to allow provisioning # uiEdit PDBA_REMOTE_PDBI_ALLOWED YES

60 Done! If needed, the next disk can be replaced the same night or at the next maintenance window.





HDA REPLACEMENT

1 Prepare system Please execute “PREPARE THE SYSTEM” procedure



NOTE: This message is only displayed for a few seconds and the DEL or (F4 on Remote keyboard) key must be pressed before the message disappears from the screen or the boot sequence will continue as usual.


If at anytime your output is garbled, in SecureCRT select Reset from the Edit menu.


Once inside the BIOS setup utility verify that the disk drive that is not being replaced is set to be the boot disk. The setting of interest is on the Boot settings screen within the BIOS setup utility. On the Boot settings screen select Hard Disk Drives as shown below.






4 Verify that hdc is listed as the 1st drive.

After selecting Hard Disk Drives, modify the drive order (the order of the entries can be reversed by pressing either the + or - key) to make the SM drive (hdc) the 1st Drive.

NOTE: The device names listed on the Hard Disk Drives screen are the model number of the server’s disk drives with one entry prepended by “PM-” (Primary Master) HDA and the other by “SM-” (Secondary Master) HDC.






5 Save any changes If this setting requires modification make the necessary change, (the order of the entries can be reversed by pressing either the + or - key), and then save the CMOS data by pressing F10, and then selecting OK when the confirmation dialog is presented.

If the order shown is correct, simply press the ESC key or select Discard Changes and Exit on the main Exit screen and select OK when asked to confirm the exit.


Login: rootPassword: <Enter root password>


8 Perform system checks on MPS-A

# syscheck



10 Verify mirror status is good

# cat /proc/mdstat





Personalities : [raid1] read_ahead 1024 sectorsEvent: 11 md12 : active raid1 hda12[0] hdc12[1] 204242240 blocks [2/2] [UU] md11 : active raid1 hda11[0] hdc11[1] 15880128 blocks [2/2] [UU] md10 : active raid1 hda10[0] hdc10[1] 4208896 blocks [2/2] [UU] md9 : active raid1 hda9[0] hdc9[1] 538048 blocks [2/2] [UU] md8 : active raid1 hda8[0] hdc8[1] 4208896 blocks [2/2] [UU] md7 : active raid1 hda7[0] hdc7[1] 538048 blocks [2/2] [UU] md6 : active raid1 hda6[0] hdc6[1] 1060160 blocks [2/2] [UU] md5 : active raid1 hda5[0] hdc5[1] 4208896 blocks [2/2] [UU] md3 : active raid1 hda3[0] hdc3[1] 2056256 blocks [2/2] [UU] md2 : active raid1 hda2[0] hdc2[1] 538112 blocks [2/2] [UU] md1 : active raid1 hda1[0] hdc1[1] 272960 blocks [2/2] [UU] unused devices: <none>



12 Execute failDisk script # /usr/TKLC/plat/sbin/failDisk –splitOnly /dev/hda

13 Verify output Should see:Removal of /dev/hda completed, OK.







Power down.



17 Remove the /dev/hda drive Hard drive A is on the left side of the lid.

18 Set jumper on new drive to match original drive.

Note Note that you’re trying to match (connect mode) not actual pins. Thus, comparison of the jumper diagrams will be necessary.

19 Install new drive Reconnect Cables.

20 Disconnect /dev/hdc Disconnect the ribbon and power cables from the good (/dev/hdc) drive - hard drive C is on the right side of the lid.

The Purpose is to see that system if system is booting from HDA? If system boots from HDA drive, then we need to remove the MBR from this.


22 Apply serial number labels We should clearly mark the label on the new hard disks that we have installed. The Label should tell if the hard drive is ‘HDA’ or ‘HDC’.

Just in case there is not already a label on the removed disks, please apply label on them too.


24 Determine if clearing of MBR is needed

If system does not boot, skip to step 30. If the system boots from the hda drive, proceed to next step to clear out the Master Boot Record and meta device information.



27 Clear the Master Boot Record and meta device information from hda.

# dd if=/dev/zero of=/dev/hda bs=512 count=1

28 Reboot MPS-A # init 6

29 Go to step 24







31 Reconnect /dev/hdc Reconnect the ribbon and power cables on the good (/dev/hdc) drive - hard drive C is on the right side of the lid.





36 Change runlevel to 3 in order to speed resync

# init 3

37 Verify output Should see:Messages of services stopping.



40 Partition new disk # /usr/TKLC/plat/sbin/cpDiskCfg hdc hda

41 Verify output Should see:Model number check, OK!Disk Geometry Check, OK!Partitioning of disk ‘/dev/hda’ completed, OK!

42 For safety, have the kernel reread the partition table

# partprobe

43 Ensure maximum speed of resync is possible

# cd /proc/sys/dev/raid/

44 Speed should be 900000. # cat speed_limit_max

If value is not 900,000 set it to 900,000:# echo 900000 > speed_limit_max

45 Re-mirror new disk # /usr/TKLC/plat/sbin/mdRepair –background








Note - Keep waiting here unless mirror status is good.48 If any output go to step 45 If there is any output, this would mean that Mirrors are not yet joined. We


49 When resync completes, perform system checks on MPS-A

# syscheck



51 Reboot server to have Onsite Technician change boot device to /dev/hda.

# init 6

Note-This may have to be done several times before successful attempt to enter bios.



NOTE: This message is only displayed for a few seconds and the DEL key must be pressed before the message disappears from the screen or the boot sequence will continue as usual.


If at anytime your output is garbled, in SecureCRT select Reset from the Edit menu


Once inside the BIOS setup utility verify that the HDA is set to be the boot disk. The setting of interest is on the Boot settings screen within the BIOS setup utility. On the Boot settings screen select Hard Disk Drives as shown below.






54 Verify that hda is listed as the 1st drive.

After selecting Hard Disk Drives, modify the drive order (the order of the entries can be reversed by pressing either the + or - key) to make the PM drive (hda) the 1st Drive.

NOTE: The device names listed on the Hard Disk Drives screen are the model number of the server’s disk drives with one entry prepended by “PM-” (Primary Master) and the other by “SM-” (Secondary Master).






55 Save any changes If this setting required modification make the necessary change, and then save the CMOS data by pressing F10, and then selecting OK when the confirmation dialog is presented. If the order shown is correct, simply press the ESC key or select Discard Changes and Exit on the main Exit screen and select OK when asked to confirm the exit.


login: rootPassword: <Enter root password>




59 Start Epap software # ./Epap start

60 Verify output Should see:EPAP application started.

61 Start Pdba software # ./Pdba start






PDBA application started.

63 Perform system checks on MPS A

# syscheck

64 Verify no syscheck failures

Should NOT get this output:One or more module in class "<class>" FAILED

65 Verify PDB, RTDB, and DSM statuses are good.

From GUI: use View RTDB Status, View PDBA Status, Run Syscheck, Capture Logs.From Eagle: use rept-stat-sccp, Rept-stat-db:display=all:db=mps, etc.

66 Have customer restart provisioning and verify DB statuses.

Run uiEdit command on ACTIVE node to allow provisioning# uiEdit PDBA_REMOTE_PDBI_ALLOWED YES

67 Done! If needed, the next disk can be replaced at the same night or at the next maintenance window.

.





Backout Procedure

STEP#

Steps to be completed.This procedure outlines the steps for returning the original Hard Drives to service in a T1000 Server


2 Verify output Should see:Power down.



4 Reinstall both original drives

5 Power on MPS-A Either re-apply power at the breaker panel or restore the power cable connections on rear of MPS A.


NOTE: This message is only displayed for a few seconds and the DEL key must be pressed before the message disappears from the screen or the boot sequence will continue as usual.

If at anytime your output is garbled, in SecureCRT select Reset from the Edit menu

7 Set /dev/hda to be the 1st

drive in the Hard Disk Drives section of the Boot Settings

After selecting Hard Disk Drives, modify the drive order (the order of the entries can be reversed by pressing either the + or - key) to make the PM drive (hda) the 1st Drive.

NOTE: The device names listed on the Hard Disk Drives screen are the model number of the server’s disk drives with one entry prepended by “PM-” (Primary Master) and the other by “SM-” (Secondary Master).

8 Log on as root

9 Verify mirror status # cat /proc/mdstat

10 Repair mirrors # /usr/TKLC/plat/sbin/mdRepair –background








Note - Keep waiting here unless mirror status is good.13 If any output go to step 10. If there is any output, this would mean that Mirrors are not yet joined. We




15 Start Epap and Pdba # ./Epap start# ./Pdba start

16 Run syscheck and fix any problems

# syscheck

17 Contact TAC or ITAC Tekelec Call Center 1-888-367-8552

3. Customer Survey.





If you, our customer, have any comments that would improve the quality of our Method of Procedure, please send them along with your name, company name, site name and phone number to Tekelec at [email protected].

Thank you for contributing to the quality of this document.

Sincerely,

Tekelec





3. APPENDIX A: CHANGES IN MOP AFTER WORK HAS STARTED

When a change in the MOP or in a step within this MOP is necessary, a concurrence must be obtained between Tekelec Customer Service and Tekelec Engineering prior to its implementation.Changes to this MOP will be reviewed, agreed upon and documented in the attached MOP Change Sheet

TEKELEC METHOD OF PROCEDURECHANGE RECORD

Page __1__ of __1__ Pages

List below all changes, including item # and reason for change.Change items must also be noted on the original MOP.Note: The Tekelec Technical Service Engineer approves any changes made on a detailed MOP.

Note: Exceptions will require 3rd level review and acknowledgement before turning system back over to the customer.

Author of Changes: Date of Change:

STEP-BY-STEP PROCEDURES

RESPONSE DATE

Customer*

Tekelec 3rd Level

Reviewed and approved by: Title Date

Concurred in by TEKELEC:

Note: If applicable



Date post:	08-Nov-2014
Category:	Documents
Upload:	fdbtttklc
View:	30 times
Download:	1 times

MO007185 - T1000 EPAP a - Dual Disk Replacement From 120GB to 250GB or 500GB Nov-2011

Documents