EMC Isilon Customer Troubleshooting Guide - Problems with ... · Did the SED node add successfully?...

1 - EMC Isilon Customer Troubleshooting Guide: Problems with Adding or Joining a Node to an

Isilon Cluster

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

Abstract

This troubleshooting guide helps you fix problems that occur while you are attempting to add new nodes to a cluster. These problems include: the node fails to add or join; the node joins but alerts are generated; the node joins but shows as down; problems when the cluster is in compliance mode.

Note: This guide does not address problems with nodes that split from an existing cluster, or nodes that fail to rejoin a cluster following an upgrade. For those issues, contact Isilon Technical Support.

February 23, 2018

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

PROBLEMS WITH ADDING OR JOINING A NODE TO AN ISILON CLUSTER

OneFS 7.1 - 8.1

__________________

http://bit.ly/isi-docfeedback


https://community.emc.com/docs/DOC-49017

http://www.emc.com/contact/contact-us.htm


Isilon Cluster



___________________________

Contents and overview

Before you begin

Page 3

Appendix A

If you need further assistance

Start troubleshooting

Page 4

Upgrade

Page 9

Put the node into Compliance mode

Page 11

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these

steps.

2. Perform

troubleshooting

steps in order.

3. Appendixes

Appendix B

How to use this flowchart

Node failed to join the cluster

Page 14

Node joined the cluster, but an alert was

generated

Page 19

Node joined the cluster, but shows as

down when you run isi_status

Page 20

Problems adding SED nodes

Page 5

Routing

Page 8





Isilon Cluster



___________________________

Configure screen logging through SSH

We recommend that you configure screen logging to log all session input and output during your troubleshooting session.

This log file can be shared with EMC Isilon Technical Support, if you require assistance at any point during troubleshooting.

Note: The screen session capability does not work in OneFS 7.1.0.6 and 7.1.1.2. If you are running either of these versions,

you can configure logging by using your local SSH client's logging feature.

1. Open an SSH connection to the cluster and log in by using the root account .

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be

preceded by the sudo prefix.

2. Change the directory to /ifs/data/Isilon_Support by running the following command:

cd /ifs/data/Isilon_Support

3. Run the following command to capture all input and output from the session:

screen -L

This will create a file named screenlog.0 that will be appended to during your session.

4. Perform troubleshooting.

Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of

troubleshooting and you do not have any other way to connect to the cluster, you could

experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before

you start this troubleshooting process. The best method is to have a serial cable

available. This way, if you are unable to connect through the network, you will still be

able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the

cluster, see article 304071 on the EMC Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another

subnet or pool, or that you have physical access to the cluster.




https://support.emc.com/kb/304071


Isilon Cluster



___________________________

Start troubleshooting

Start

IntroductionStart troubleshooting here. If you need

help to understand the flowchart

conventions that are used in this guide,

see Appendix B: How to use this

flowchart.

Are you trying to add

an SED node?Yes

No

Go to Page 8

Go to Page 5

IMPORTANT NOTE FOR ADDING MULTIPLE NODES1) Adding a node can take an hour or more to complete.

2) Wait 30 to 45 minutes after a successful node join before adding another

node.

When adding a new node, numerous group changes occur on the cluster. It is

easier for the cluster to process one new node at a time. Also, adding multiple

nodes at the same time can cause the cluster to lose quorum if more than half

of the cluster nodes are unavailable over the internal network during the join

process. A loss of quorum can result in data unavailability (DU).





Isilon Cluster



___________________________

Problems adding SED nodes

Page

5

You could have arrived here from:

Page 4 - Start troubleshooting

Does the SED node

display errors similar to

those in the box at the bottom

of this page?

Error messages

Traceback (most recent call last):

File "/usr/bin/isi_evaluate_purpose_drive", line 146, in <module>

error = main(sys.argv[1:])

File "/usr/bin/isi_evaluate_purpose_drive", line 126, in main

purposedb.give_purpose(devid, drive)

File "/usr/local/lib/python2.6/site-packages/isi/fs/

drive_purposing.py", line 774, in give_purpose

isi.hw.drived.Error: [Errno 22] Invalid argument

No

Yes

Go to Page 7

Go to Page 6

If you have not done so already, log in to

the cluster and configure screen logging

through SSH, as described on Page 3.





Isilon Cluster



___________________________

Problems adding SED nodes (2)

Page

6


Page 5 - Problems adding SED nodes

Use the following article to try to resolve the

issue. Although the article is not specifically

about SED nodes, these errors are commonly

seen on SED nodes.

When joining a new node to a cluster, process

appears to hang with drive_purposing.py

errors, article 463272

Does the

SED node add

successfully now?No

Yes

Note the page number

that you are currently on.

Upload log files and

contact Isilon Technical

Support, as instructed in

Appendix A.

End troubleshooting






Isilon Cluster



___________________________

Problems adding SED nodes (3)

Page

7


Page 5 - Problems adding SED nodes

End troubleshooting

No

Yes

Have at least

four hours passed since

you started the add/join

process?

Wait until four hours have

passed since you started the

add/join process.

No

Yes

Did the SED node

add successfully?

Note It can take up to four hours for

SED nodes to join because

every drive in the node must

be encrypted. During this time,

the node will appear to be

stalled, and if you run isi

status, the node will show as

down. However, the physical

drives will be blinking,

indicating that they are being

encrypted.

Go to Page 8

Go to Page 8





Isilon Cluster



___________________________

Routing

Note Quorum means that more than

half of a cluster's nodes are

available over the internal

network. For more information,

see the "Quorum" section of the

OneFS Administration Guide.

No

Using the Isilon Supportability &

Compatibility Guide to check whether the

version of OneFS that is running on the

cluster is compatible with the new node.

Is the version

of OneFS on the cluster

compatible with the new

node?

No

Did the cluster

lose quorum?Yes

Yes

Note the page

number that you

are currently on.

Upload log files and

contact Isilon Technical

Support, as instructed

in Appendix A.

If you have not done so already, log in to

the cluster and configure screen logging

through SSH, as described on Page 3.

Go to Page 10

Go to Page 9

Page

8

Page 7 - Problems adding SED nodes (3)


Page 4 - Start troubleshooting





https://support.emc.com/docu44518


Isilon Cluster



___________________________

Upgrade

Page

9


Page 8 - Routing

Consult with your Account Manager to determine an

upgrade strategy, or review the OneFS Upgrade

Planning and Process Guide to plan for an upgrade

during your next maintenance window.

After the upgrade, check whether the problem is

resolved.

NoIs the problem

resolved?

Yes

End troubleshooting

Go to Page 10






Isilon Cluster



___________________________

Routing (2)

NoIs the cluster in SmartLock

compliance mode?

Yes

Page

10


Page 8 - Routing

Yes

No

Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.

Go to Page 11

Was the new

node put into SmartLock

Compliance mode before you

attempted to add it

to the cluster?

Go to Page 13

Page 9 - Upgrade





Isilon Cluster



___________________________

Put the node into Compliance mode

Page

11 You could have arrived here from:

Page 10 - Routing (2)

Yes

Put the node into SmartLock Compliance mode.

See the "Tips for configuring SmartLock compliance

mode" section in OneFS 7.0 and later: Smartlock

compliance mode, article 471931.

Yes

No

Try to add or join the node again.

Did the node

join successfully, with no

problems, errors,

or alerts?

No

Did you

receive the following

error when attempting to reboot the

node into SmartLock Compliance mode?

ERROR: Could not apply

Compliance settings to node!

Could not apply

Compliance settings

to sudoers.

End troubleshooting

Go to Page 12

Go to Page 13






Isilon Cluster



___________________________

Put the node into Compliance mode (2)

Page

12


Page 11 - Put the node into Compliance

mode

Follow the instructions in the following article:

OneFS 7.0 - 7.1: Error: "Could not apply

Compliance settings to node! Could not apply

Compliance settings to sudoers", article 471936

Do you

receive the same

error again?

No

Yes

Yes

No

Try to add or join the node again.

Did the node

join successfully, with no

problems, errors,

or alerts?

End troubleshooting

Note the page number that

you are currently on.

Upload log files and contact

Isilon Technical Support, as

instructed in Appendix A.

Go to Page 13






Isilon Cluster



___________________________

Routing (3)

Page

13

Did the node

join the cluster?No

Did the cluster

generate an alert after

the node joined?Yes

Does the node

show as down?Yes

Yes

No

No


are currently on.



Run the following command:

isi status

Go to Page 14

Go to Page 20

Go to Page 19




mode


mode (2)





Isilon Cluster



___________________________

Node failed to join the cluster

Page

14



Check whether the cluster is in secure join mode.

1. Enter config mode by running:

isi config

2. At the config prompt, run:

joinmode

3. Note from the output whether the add node setting is Secure

or Manual). The output looks like the following:

Current add node setting is Secure

3. Quit config mode by running:

quit

Is the setting

Secure or Manual?

Manual

When a node fails to join the cluster, errors appear

at the command prompt. Copy and save these

errors in case you need to open a Support case and

the screen log session does not capture them.

Go to Page 15Secure

Go to Page 16





Isilon Cluster



___________________________

Page

15 You could have arrived here from:

Page 14 - Node failed to join the cluster

Did the node

join successfully?No

Yes

In secure join mode, you cannot use the serial console wizard option

[2] Join an existing cluster to join a node to the cluster.

Instead, you must add the node from the cluster using the OneFS web administration interface or

command-line interface. Do one of the following now:

OneFS web administration interface

Click Cluster Management > Hardware Configuration > Add Nodes.

OneFS command-line interface

1. Run the following command for your version of OneFS:

OneFS 8.0 - 8.1

isi devices node list

OneFS 7.1

isi devices -a discover

The output contains a list of unconfigured nodes (nodes that are not joined to the cluster) along with

their serial numbers.

2. Run the following command, where <SN> is the serial number of the unconfigured node that you want

to add:

OneFS 8.0 - 8.1

isi devices node add --serial-number=<SN>

OneFS 7.1

isi devices -a add -d <SN>

End troubleshooting

Go to Page 16

Node failed to join the cluster (2)

Node is in secure join mode





Isilon Cluster



___________________________

Go to Page 17.


Page

16

!! Error: Error joining cluster myisilon: agent

denied join: Node

!! cannot join cluster: Cannot add nodes to

clusters of mixed builds:

!! found both B_6_5_5_171(RELEASE) and

B_6_5_5_124(RELEASE) on

!! cluster.

If you received this error Do this

Follow the instructions in this article:

OneFS: New node fails to join cluster and displays

"Error downloading new image

from cluster" error, article 471930


OneFS 6.5 - 7.1: "Cannot add nodes to clusters of

mixed builds" error when adding a new node to an

existing cluster,

article 471926

Take note of the the error that appears on the serial

console of the node that you are trying to add. Use

the following table to determine your next steps.

!! Error: denying join because cluster clustername

is not available:!! unjoinable -- backend cables

are cross-wired

Note the page number that you are currently on.

Upload log files and contact Isilon Technical Support,

as instructed in Appendix A.

error joining cluster - <cluster name>:

verify_join: Error


Page 14 - Node failed to join the cluster

Page 15 - Node failed to join the cluster (2)

None of these errors.

Note the page number that you are currently on.

Upload log files and contact Isilon Technical Support,

as instructed in Appendix A.

!! Error: Error joining cluster Ritpide: Node

version still

!! doesn't match cluster. Please retry the join


[OneFS 8.x] New node unable to join the cluster with

an error "Node version still doesn't match cluster.

Please retry the join",

article 485965

!! Error: Error downloading new image from cluster

my-cluster: read !! process exit with 74 - More

info in /var/log/messages on the !! cluster. Enter

'yes' or press [Enter] to re-configure the node.







Isilon Cluster



___________________________


Page

17



Run the following command to check whether any nodes have a root partition that is 95% or more full:

isi_for_array -sX 'df -i /'

The output looks like the following for each node:

cluster-1: Filesystem 512-blocks Used Avail Capacity iused ifree %iused Mounted on

cluster-1: /dev/imdd0a 1016286 875222 59762 94% 37793 89693 30% /

Both of the percentages (Capacity and %iused) should be less than 95%.

Do any nodes

show Capacity or %iused as

95% or greater? Yes

No


are currently on.



Go to Page 18





Isilon Cluster



___________________________


Page

18



Did the

node join

successfully?No

Yes

Go to the following article and follow the

instructions to bring the Capacity and

%iused below 95%. When finished, return

here to complete this flowchart.

OneFS: Event notification: Node reached

95% used capacity on the root filesystem,

article 464118

Attempt to add or join

the node again.

Note the page number that

you are currently on.

Upload log files and contact Isilon

Technical Support, as instructed

in Appendix A.

End troubleshooting






Isilon Cluster



___________________________

Node joined the cluster, but an alert was generated

Page

19


are currently on. Upload log files and contact Isilon

Technical Support, as instructed in Appendix A.

The wrong type of disk was inserted in

Bay <bay>. The drive should be a

<disk_size> <disk_tech> <disk_media>

Node contains an 812 (3/4 chassis) SKU

If you received this alert Do this

Take note of the the alert. Use the

table below for your next steps.


Isilon: "Wrong type of disk" error returned when adding

new node or upgrading drive to SSD, article 471922.

If the problem continues, note the page number in this

guide that you are currently on. Upload log files and

contact Isilon Technical Support, as instructed in

Appendix A.




are currently on. Upload log files and contact Isilon

Technical Support, as instructed in Appendix A.

Any other alert.






Isilon Cluster



___________________________

Node joined the cluster, but shows as down when you run isi_status

Page

20

Reboot the affected node

Is the issue resolved? No

Yes

End troubleshooting

Note the page number

that you are currently on.

Upload log files and contact

Isilon Technical Support, as

instructed in Appendix A.







Isilon Cluster



___________________________

Contact EMC Isilon Technical Support

If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.

This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Appendix A: If you need further assistance

Upload node log files and the screen log file to EMC Isilon Technical Support

1. When troubleshooting is complete, type exit to end your screen session.

2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your

method of uploading files. If you are not sure which method to use, use FTP.

ESRS:

isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support/screenlog.0

FTP:

isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support/screenlog.0

HTTP:

isi_gather_info --http --local-only -f /ifs/data/Isilon_Support/screenlog.0

SMTP:

isi_gather_info --email --local-only -f /ifs/data/Isilon_Support/screenlog.0

SupportIQ:

Copy and paste the following command.

Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly

as it appears on the page), but when you press Enter, the command will run as it should.

isi_gather_info --local-only -f /ifs/data/Isilon_Support/screenlog.0 --noupload \

--symlink /var/crash/SupportIQ/upload/ftp

3. If you receive a message that the upload was unsuccessful , refer to article 304567 on the EMC Online Support site for

directions on how to upload files over FTP.

____________




http://www.emc.com/contact/contact-us.htm



Isilon Cluster



___________________________

Decision diamondYes No

Process stepProcess step with command:

command xyz

Go to Page #

Page

# Note Provides context and additional

information. Sometimes a note is linked

to a process step with a colored dot.

CAUTION!Caution boxes warn that

a particular step needs

to be performed with

great care, to prevent

serious consequences.

End point Document ShapeCalls out supporting documentation

for a process step. When possible,

these shapes contain links to the

reference document.

Sometimes linked to a process step

with a colored dot.

Optional process step

Directional arrows indicate

the path through the

process flow.

IntroductionDescribes what the section helps you to

accomplish.


Page # - Page title

Appendix B: How to use this flowchart





Isilon Cluster



___________________________

Copyright © 2018 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS-IS. DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com




https://www.emc.com

Date post:	27-Oct-2019
Category:	Documents
Upload:	others
View:	76 times
Download:	3 times

EMC Isilon Customer Troubleshooting Guide - Problems with ... · Did the SED node add successfully?...

Documents