1 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
1 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Abstract
This troubleshooting guide helps you address time skews, or drifts, on your Isilon cluster.
January 18, 2019
EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE
TROUBLESHOOT A TIME DRIFT ON YOUR ISILON CLUSTER
OneFS 7.1.0 - 8.1.0
2 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
2 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Contents and overview
Before you begin
Page 3
Before you begin
Page 3
Appendix A
If you need further assistance
Appendix A
If you need further assistance
Start troubleshooting
Page 4
Start troubleshooting
Page 4
Set the time on a new node
Page 5
Set the time on a new node
Page 5
Determine which nodes are chimer nodes
Page 7
Determine which nodes are chimer nodes
Page 7
How do you keep time?
Page 8
How do you keep time?
Page 8
Note Follow all of these steps, in order, until you reach a resolution.
1. Follow these
steps.
2. Perform
troubleshooting
steps in order.
3. Appendixes
Appendix B
How to use this flowchart
Appendix B
How to use this flowchart
NTP
Page 9
NTP
Page 9
SMB Time
Page 29
SMB Time
Page 29
Appendix C
Peering table
Appendix C
Peering table
3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
3 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Configure screen logging through SSH
We recommend that you configure screen logging to log all session input and output during your troubleshooting session.
This log file can be shared with Isilon Technical Support, if you require assistance at any point during troubleshooting.
Note: The screen session capability does not work in OneFS 7.1.0.6 and 7.1.1.2. If you are running either of these versions,
you can configure logging by using your local SSH client's logging feature.
1. Open an SSH connection to the cluster and log in by using the root account.
Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be
preceded by the sudo prefix.
2. Change the directory to /ifs/data/Isilon_Support by running the following command:
cd /ifs/data/Isilon_Support
3. Run the following command to capture all input and output from the session:
screen -L
This will create a file named screenlog.0 that will be appended to during your session.
4. Perform troubleshooting.
Before you begin
CAUTION!If the node, subnet, or pool that you are working on goes down during the course of
troubleshooting and you do not have any other way to connect to the cluster, you could
experience data unavailability.
Therefore, make sure that you have more than one way to connect to the cluster before
you start this troubleshooting process. The best method is to have a serial console
connection available. This way, if you are unable to connect through the network, you
will still be able to connect to the cluster physically.
For specific requirements and instructions for making a physical connection to the
cluster, see article 304071 on the EMC Online Support site.
Before you begin troubleshooting, confirm that you can connect through either another
subnet or pool, or that you have physical access to the cluster.
CAUTION!If the node, subnet, or pool that you are working on goes down during the course of
troubleshooting and you do not have any other way to connect to the cluster, you could
experience data unavailability.
Therefore, make sure that you have more than one way to connect to the cluster before
you start this troubleshooting process. The best method is to have a serial console
connection available. This way, if you are unable to connect through the network, you
will still be able to connect to the cluster physically.
For specific requirements and instructions for making a physical connection to the
cluster, see article 304071 on the EMC Online Support site.
Before you begin troubleshooting, confirm that you can connect through either another
subnet or pool, or that you have physical access to the cluster.
4 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
4 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Start troubleshooting
Is the following error
present in the log files:
frequency error -500 PPM
exceeds tolerance
500 PPM"?
Do you have a new
node that is showing
the wrong time?
Start
IntroductionStart troubleshooting here. For an overview
of the conventions used in this flowchart, see
Appendix B: How to use this flowchart.
IntroductionStart troubleshooting here. For an overview
of the conventions used in this flowchart, see
Appendix B: How to use this flowchart.
If you have not done so already, log in to
the cluster and configure screen logging
through SSH, as described on page 3.
If you have not done so already, log in to
the cluster and configure screen logging
through SSH, as described on page 3.
Go to Page 5Go to Page 5Yes
No
Refer toFrequency error -500 PPM
exceeds tolerance 500
PPM" error, article 471912
Refer toFrequency error -500 PPM
exceeds tolerance 500
PPM" error, article 471912
Go to Page 8Go to Page 8
No
Yes
5 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
5 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Set the time on a new node
Reset the time on the recently added node by running the following command, where <LNN> is the logical node number of
the recently added node, and <timeIP> is the domain name or IP address of one of your external time sourcess:
isi_for_array -n <LNN> "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub <timeIP>"
For example:
isi_for_array -n 7 "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub 10.11.120.13"
Page
5
You could have arrived here from:
• Page 4 - Start troubleshooting
You could have arrived here from:
• Page 4 - Start troubleshooting
Wait at least five minutes.
Refresh the time on the cluster by running the following command
with the same variables as the previous step:
isi_for_array -n <LNN> "ntpdate -ub <timeIP>"
Go to Page 6Go to Page 6
Note Resetting the time on the cluster system clock helps to verify whether the
NTP internal clock can build a meaningful, stable drift value. The drift
value is how much the clock drifts per unit (parts per million) compared to
the reference clock. It takes at least 24 hours for a drift value to build, and
it usually takes about three days for a drift value to settle into a steady
state where the value changes by only very small amounts hour to hour.
6 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
6 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Set the time on a new node (2)
Page
6
You could have arrived here from:
• Page 5 - Set the time on a new node
You could have arrived here from:
• Page 5 - Set the time on a new node
Validate that the recently added node is using a chimer node as its source of time by running the following
command, where <LNN> is the logical node number of the recently added node:
isi_for_array -n <LNN> "ntpq -np"
The output will appear similar to the following, the asterisk (*) indicates the "peer", or source of time, for that node:
cluster-1: remote refid st t when poll reach delay offset jitter
cluster-1: ==============================================================================
cluster-1: +10.11.12.13 10.12.123.13 3 u 889 1024 377 0.098 -2.139 0.013
cluster-1: *192.11.12.13 10.12.123.13 3 u 901 1024 377 0.112 -2.559 0.015
cluster-1: +10.11.12.14 10.12.123.13 3 u 890 1024 377 0.089 -2.652 0.010
Go to Page 18Go to Page 18
7 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
7 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Determine which nodes are chimer nodes
Page
7
You could have arrived here from:
• Page 10 - NTP - External time sources
• Page 18 - Validate time on the non-chimer nodes (2)
• Page 19 - Validate time on the non-chimer nodes (3)
• Page 25 - Validate time on the non-chimer nodes (9)
• Page 27 - Validate time on the non-chimer nodes (11)
By default, a cluster has three chimer nodes. Chimer nodes are the three lowest logical node numbers (LNNs) in the
cluster. If a chimer node goes offline, the node with the next-lowest LNN takes its place.
In a 3-node cluster, there is only one chimer node, the one with the lowest LNN.
In a large cluster with multiple external time sources, there might be only two chimer nodes (if configured as such).
You can determine which nodes are the chimer nodes by following these steps.
Run the following command:
isi_for_array -sQX "grep -B1 server /etc/ntp.conf" | awk '/These are the cluster/,/^$/'
The output will display the IP addresses of the chimer nodes and will appear similar to the following:
# NTP servers. These are the cluster's chimer nodes:
server 10.12.13.100 iburst true
server 10.12.13.101 iburst true
server 10.12.13.102 iburst true
Run the following command to display the node IP addresses and their corresponding LNNs:
isi_nodes "%{internal} %{name}"
The output will appear similar to the following:
10.12.13.101 cluster-1
10.12.13.102 cluster-2
10.12.13.103 cluster-3
10.12.13.104 cluster-4
10.12.13.105 cluster-5
Using the output from the previous steps, determine the IP
addresses of the chimer nodes and note the LNN for each. In
the example above, the chimer nodes are LNNs 1, 2, and 4.
Go to Page 11Go to Page 11
______________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
8 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
8 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
How do you keep time?
Page
8
You could have arrived here from:
• Page 4 - Start troubleshooting
You could have arrived here from:
• Page 4 - Start troubleshooting
Which method are
you using to sync time
to the cluster: NTP or
SMB time?
Note: SMB time is not available in OneFS 7.2 and later.
Dell EMC Isilon suggests that you use NTP to sync time to
the cluster. SMB time is not a reliable method for syncing
time on your Isilon cluster.
For more information, see OneFS 6.5 and later: Time on
the cluster is not synchronized with the Active Directory
domain controller, article 463674.
Note: SMB time is not available in OneFS 7.2 and later.
Dell EMC Isilon suggests that you use NTP to sync time to
the cluster. SMB time is not a reliable method for syncing
time on your Isilon cluster.
For more information, see OneFS 6.5 and later: Time on
the cluster is not synchronized with the Active Directory
domain controller, article 463674.
SMB time
Go to Page 9Go to Page 9NTP
Are you able to
switch to NTP for
syncing your cluster
time?
Go to Page 29Go to Page 29
No
Yes Go to Page 9Go to Page 9
9 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
9 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP
Page
9
You could have arrived here from:
• Page 8 - How do you keep time?
You could have arrived here from:
• Page 8 - How do you keep time?
Configure NTP by following the
instructions in this article:
OneFS 6.5 and later: Time on the
cluster is not synchronized with the
Active Directory domain controller,
article 463674.
Configure NTP by following the
instructions in this article:
OneFS 6.5 and later: Time on the
cluster is not synchronized with the
Active Directory domain controller,
article 463674.
Note It can take 23 hours for time to be
synchronized on all nodes in the
cluster. The chimer nodes
synchronize their times to the external
time sources, and the non-chimer
nodes in the cluster synchronize their
times to the chimer nodes.
Did switching
to NTP resolve
the issue?
End troubleshootingYes
Are your external
time sources providing the
correct, current time, or an
intentionally incorrect
time?
No
Note the page number
that you are currently on.
Upload log files and contact
Isilon Technical Support, as
instructed in Appendix A.
Note the page number
that you are currently on.
Upload log files and contact
Isilon Technical Support, as
instructed in Appendix A.
Intentionally
incorrect
time
Go to Page 10Go to Page 10
Correct time
10 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
10 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - External time sources
Page
10
You could have arrived here from:
• Page 9 - NTP
• Page 16 - NTP - Chimer node time incorrect (5)
Identify the external time sources by running the following command:
isi_ntp_config list
The output will appear similar to the following, with all external sources listed:
NTP Server Setting:
time.isilon.com
Nodes excluded from contacting external NTP servers:
No nodes are excluded from contacting external NTP servers.
Chimers: 3
NTP Auth Keyfile:
No Keyfile is configured.
For each external time source, run the following command to obtain all of the
host IP addresses that the server name resolves to, where <externalTime> is
the server name obtained in the previous step:
host <externalTime>
For example: host time.isilon.com
The output will appear similar to the following:
time.isilon.com has address 10.11.12.20
time.isilon.com has address 10.11.12.21
Record this information as you will need to refer to it later in this guide.
Go to Page 7Go to Page 7
____________________________________________________________________________
______________________
11 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
11 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Compare chimer node time to known good time
Page
11
You could have arrived here from:
• Page 7 - Determine which nodes are
chimer nodes
You could have arrived here from:
• Page 7 - Determine which nodes are
chimer nodes
Open a new browser window and browse
to an authoritative, good time source, for
example: www.time.gov.
Open a new browser window and browse
to an authoritative, good time source, for
example: www.time.gov.
Check the time reported by the chimer nodes by
running the following command on the cluster, where
<LNNs> is a comma-separated list of the logical node
numbers of all of the chimer nodes:
isi_for_array -n <LNNs> date
For example:
isi_for_array - 1,2,4 date
Compare the time for each chimer node to
the authoritative, good time source, and
verify whether the chimer node time is
correct to within one second.
Is the time for each
chimer node correct to
within one second of the
authoritative, good time
source?
Go to Page 17Go to Page 17
Go to Page 12Go to Page 12
Yes
No
12 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
12 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Chimer node time is incorrect
Page
12
You could have arrived here from:
• Page 11 - NTP - Compare chimer node time to known good time
• Page 15 - Chimer node time is incorrect (4)
• Page 16 - Chimer node time is incorrect (5)
Note Resetting the time on the cluster system clock helps to verify whether the
NTP internal clock can build a meaningful, stable drift value. The drift
value is how much the clock drifts per unit (parts per million) compared to
the reference clock. It takes at least 24 hours for a drift value to build, and
it usually takes about three days for a drift value to settle into a steady
state where the value changes by only very small amounts hour to hour.
Reset the time on the recently added node by running the following command, where <LNN> is the logical node number of
the recently added node, and <timeIP> is the domain name or IP address of one of your external time sourcess:
isi_for_array -n <LNN> "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub <timeIP>"
For example:
isi_for_array -n 7 "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub 10.11.120.13"
Wait at least five minutes.
Refresh the time on the cluster by running the following command
with the same variables as the previous step:
isi_for_array -n <LNN> "ntpdate -ub <timeIP>"
Go to Page 13Go to Page 13
________________________________________________________________________________________________________
______________________________________________________________________
______________________________________________________________________
13 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
13 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Chimer node time is incorrect (2)
Page
13
You could have arrived here from:
• Page 12 - NTP - Chimer node time is incorrect
You could have arrived here from:
• Page 12 - NTP - Chimer node time is incorrect
Run the following command, where <LNNs> is a
comma-separated list of the logical node
numbers of the chimer nodes:
isi_for_array -n <LNNs> "ntpq -np"
See Appendix C for example output.
Does .STEP.
appear in the
refid column?
This should be a transient condition.
Wait five minutes and try again.
If this reoccurs more than three times,
choose a different source of time.
Yes
Go to Page 14Go to Page 14
No
____________________
14 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
14 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Chimer node time is incorrect (3)
Page
14
You could have arrived here from:
• Page 13 - NTP - Chimer node time is incorrect (2)
You could have arrived here from:
• Page 13 - NTP - Chimer node time is incorrect (2)
For all chimer nodes,
does an asterisk appear in
front of one of the time sources,
in either the remote or the
refid column?
No
The chimer nodes are
correct. Now we will check
the non-chimer nodes.Yes
Go to Page 15Go to Page 15
Go to Page 17Go to Page 17
15 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
15 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Chimer node time is incorrect (4)
Page
15
You could have arrived here from:
• Page 14 - NTP - Chimer node time is incorrect (3)
You could have arrived here from:
• Page 14 - NTP - Chimer node time is incorrect (3)
Is an asterisk in front
of .LOCL. in either the
remote or refid column
on any chimer node?
No
Yes
Go to Page 12Go to Page 12
Note the page number
that you are currently on.
Upload log files and
contact Isilon Technical
Support, as instructed in
Appendix A.
Note the page number
that you are currently on.
Upload log files and
contact Isilon Technical
Support, as instructed in
Appendix A.
Yes
No
Is this your
second time on
this page?No
A "fudge" source is a clock that
gets its time from the system
clock or some other clock that is
not providing time from a
standards-based NTP time
reference (such as GPS, CDMA,
or a network reference time
server such as time.nist.gov).
Yes
Is your external
time source a
"fudge" source?Yes
Is this your
second time on
this page?
Go to Page 16Go to Page 16
No
Note If this is your second time on this page, note the
page number that you are currently on. Upload
log files and contact Isilon Technical Support as
instructed in Appendix A.
Note If this is your second time on this page, note the
page number that you are currently on. Upload
log files and contact Isilon Technical Support as
instructed in Appendix A.
16 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
16 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Chimer node time is incorrect (5)
Page
16
You could have arrived here from:
• Page 15 - NTP - Chimer node time is incorrect (4)
You could have arrived here from:
• Page 15 - NTP - Chimer node time is incorrect (4)
How many
external time
sources do you
have?
Two or more
One
Note If this is your second time on this page, note the
page number that you are currently on. Upload
log files and contact Isilon Technical Support as
instructed in Appendix A.
Note If this is your second time on this page, note the
page number that you are currently on. Upload
log files and contact Isilon Technical Support as
instructed in Appendix A.
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
Is this a
3-node
cluster?Yes
Do you have
exactly 3
chimers?
No
No
Yes
Reduce the chimer count to two
by running:
isi_ntp_config chimers 2
Go to Page 12Go to Page 12
Note For chimer count, see the output of
isi_ntp_config list that you ran on page 10.
Note For chimer count, see the output of
isi_ntp_config list that you ran on page 10.
17 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
17 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes
Page
17
You could have arrived here from:
• Page 11 - NTP - Compare chimer node time to known good time
• Page 14 - NTP - Chimer node time is incorrect (3)
• Page 19 - NTP - Validate time on the non-chimer nodes (3)
Refresh the time on the cluster by running the following
command, where <timeSource> is the domain name or IP
address of one of the external time sources (preferably the
same one you used previously):
isi_for_array -s "ntpdate -ub <timeSource>"
Validate that the non-chimer nodes (all nodes except the chimers)
are using the chimer nodes as their source of time, by running the
following command, where <LNNs> is a comma-separated list of
the chimer node logical node numbers:
isi_for_array -sXx <LNNs> "ntpq -np && echo"
See the box on this page for example output.
Go to Page 18Go to Page 18
Example outputcluster-1# isi_for_array -sxX 1,2,4 "ntpq -np && echo"
cluster-3: remote refid st t when poll reach delay offset jitter
cluster-3: ==============================================================================
cluster-3: +10.11.12.13 10.12.111.13 3 u 889 1024 377 0.098 -2.139 0.013
cluster-3: *192.11.12.13 10.12.111.13 3 u 901 1024 377 0.112 -2.559 0.015
cluster-3: +10.11.12.14 10.12.111.13 3 u 890 1024 377 0.089 -2.652 0.010
cluster-3:
cluster-5: remote refid st t when poll reach delay offset jitter
cluster-5: ==============================================================================
cluster-5: +10.11.12.13 10.12.111.13 3 u 1 64 377 0.074 0.018 0.013
cluster-5: *192.11.12.13 10.12.111.13 3 u 41 64 377 0.087 -0.580 0.005
cluster-5: +10.11.12.14 10.12.111.13 3 u 44 64 377 0.069 -0.396 0.007
cluster-5:
cluster-6: remote refid st t when poll reach delay offset jitter
cluster-6: ==============================================================================
cluster-6: +10.11.12.13 10.12.111.13 3 u 39 64 377 0.072 -1.341 0.016
cluster-6: *192.11.12.13 10.12.111.13 3 u 1 64 377 0.082 -1.926 0.038
cluster-6: +10.11.12.14 10.12.111.13 3 u 26 64 377 0.111 -1.713 0.020
______________________________________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________________________
18 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
18 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (2)
Page
18
You could have arrived here from:
• Page 6 - Set the time on a new node (2)
• Page 17 - NTP - Validate the time on the non-chimer nodes
In the remote column for each node, check whether an
asterisk (*) appears next to one of the cluster chimer
nodes (chimer nodes were identified on page 7).
In the remote column for each node, check whether an
asterisk (*) appears next to one of the cluster chimer
nodes (chimer nodes were identified on page 7).
For each
non-chimer node,
does an asterisk
appear in front of a
chimer node?
Go to Page 20Go to Page 20
Go to Page 19Go to Page 19
No
Yes
________________________________________________________________
______________________________________________________________________________________________
19 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
19 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (3)
Page
19
You could have arrived here from:
• Page 18 - NTP - Validate time on the non-chimer
nodes (2)
• Page 23 - NTP - Validate time on the non-chimer
nodes (7)
Reset the time on each non-chimer node that lacks an asterisk, by running the following command, where <LNNs> is a
comma-separated list of the LNNs of all the non-chimer nodes that lack an asterisk, and <chimerIP> is the IP address
of the asterisked chimer node:
isi_for_array -n <LNNs> "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub <chimerIP>"
Wait 20 minutes.
Validate that the non-chimer nodes are using the chimer nodes as
their source of time, by running the following command, where
<LNNs> is a comma-separated list of the chimer node logical node
numbers (this command excludes the chimer nodes from the output):
isi_for_array -sXx <LNNs> "ntpq -np && echo"
See page 17 for example output.
In the remote column for each node, check whether an asterisk (*)
appears next to one of the cluster chimer nodes (chimer nodes are
identified on page 7).
In the remote column for each node, check whether an asterisk (*)
appears next to one of the cluster chimer nodes (chimer nodes are
identified on page 7).
For each
non-chimer node,
does an asterisk appear
in front of a chimer node
in the remote
column?
Note the page number
that you are currently on.
Upload log files and
contact Isilon Technical
Support, as instructed in
Appendix A.
Note the page number
that you are currently on.
Upload log files and
contact Isilon Technical
Support, as instructed in
Appendix A.
Go to Page 20Go to Page 20YesNo
____________
_______________________________________
________
_______________________________________
________
_______________________________________
________
_______________________________________
________
20 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
20 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (4)
Page
20
You could have arrived here from:
• Page 18 - NTP - Validate time on the non-chimer
nodes (2)
• Page 19 - NTP - Validate time on the non-chimer
nodes (3)
Refresh the time on the cluster by running the following
command, where <timeSource> is the domain name or IP
address of one of the external time sources (preferably the
same one you used previously):
isi_for_array -s "ntpdate -ub <timeSource>"
Wait 24 hours.
Compare the time on each node to the time on the external time
source by running the following command, where <timeSource>
is the domain name or IP address of one of the external time
sources (preferably the same one you used previously):
isi _for_array -s "ntpdate -qub <timeSource>"
See the note box on page 21 for example output.
Go to Page 21Go to Page 21
____________
_______________________________________
________
_______________________________________
________
_______________________________________
________
_______________________________________
________
21 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
21 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (5)
Page
21
You could have arrived here from:
• Page 20 - NTP - Validate time on the non-chimer nodes (4)
• Page 22 - NTP - Validate time on the non-chimer nodes (6)
• Page 27 - NTP - Validate time on the non-chimer nodes (11)
Your output should appear similar to the example
output in the box at the bottom of this page.
Check the offset for each node. It should be
within ± 0.500 seconds.
Is the
offset within ±
0.500 seconds for
each node?
Go to Page 22Go to Page 22
Yes
Go to Page 23Go to Page 23No
Example outputcluster-1# isi_for_array -s "ntpdate -qub 10.11.12.13"
cluster-1: server 10.11.12.13, stratum 2, offset 0.001176, delay 0.10016
cluster-1: 17 Sep 15:19:45 ntpdate[22860]: step time server 10.254.140.21 offset 0.001176 sec
cluster-2: server 10.11.12.13, stratum 2, offset 0.000484, delay 0.10120
cluster-2: 17 Sep 15:19:46 ntpdate[59483]: step time server 10.254.140.21 offset 0.000484 sec
cluster-4: server 10.11.12.13, stratum 2, offset 0.000192, delay 0.10023
cluster-4: 17 Sep 15:19:46 ntpdate[4714]: step time server 10.254.140.21 offset 0.000192 sec
cluster-5: server 10.11.12.13, stratum 2, offset -0.002581, delay 0.10712
cluster-5: 17 Sep 15:19:47 ntpdate[62312]: step time server 10.254.140.21 offset -0.002581 sec
cluster-6: server 10.11.12.13, stratum 2, offset -0.001351, delay 0.10710
cluster-6: 17 Sep 15:19:48 ntpdate[3141]: step time server 10.254.140.21 offset -0.001351 sec
Note If this is your second time on this
page, note the page number that you
are currently on. Upload log files and
contact Isilon Technical Support as
instructed in Appendix A.
Note If this is your second time on this
page, note the page number that you
are currently on. Upload log files and
contact Isilon Technical Support as
instructed in Appendix A.
______________________________________________________________________________________________
______________________________________________________________________________________________
______________________________________________________________________________________________
22 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
22 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (6)
Page
22
You could have arrived here from:
• Page 21 - NTP - Validate time on the non-chimer nodes (5)
• Page 26 - NTP - Validate time on the non-chimer nodes (10)
• Page 28 - NTP - Validate time on the non-chimer nodes (12)
Wait 48 hours.
Run the following command, where <timeSource> is the
domain name or IP address of one of the external time
sources (preferably the same one you used previously):
isi_for_array -s "ntpdate -qub <timeSource>"
See the note box on page 21 for example output.
Note If this is your second time on
this page, note the page
number that you are currently
on. Upload log files and contact
Isilon Technical Support as
instructed in Appendix A.
Note If this is your second time on
this page, note the page
number that you are currently
on. Upload log files and contact
Isilon Technical Support as
instructed in Appendix A.
Check the offset for each node. It
should be within ± 0.500 seconds.
Is the
offset within ±
0.500 seconds for
each node?
End troubleshooting
Yes
No Go to Page 23Go to Page 23
____________
____________________________________________________________________________________________________________________________________________________________________________________________
______________________________________________________________________________________________
23 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
23 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (7)
Page
23
You could have arrived here from:
• Page 21 - NTP - Validate time on the non-chimer nodes (5)
• Page 22 - NTP - Validate time on the non-chimer nodes (6)
• Page 28 - NTP - Validate time on the non-chimer nodes (12)
Check the peer status of the problem node to determine whether an asterisk appears in front of a chimer
node, by running the following command, where <LNNs> is a comma-separated list of the LNNs of all the
nodes that did not have an offset within ± 0.500 seconds:
isi_for_array -n <LNNs> "ntpq -np"
The output will appear similar to this:
cluster-3: remote refid st t when poll reach delay offset jitter
cluster-3: ==============================================================================
cluster-3: +10.11.12.13 10.12.111.13 3 u 889 1024 377 0.098 -2.139 0.013
cluster-3: *192.11.12.111 10.12.111.13 3 u 901 1024 377 0.112 -2.559 0.015
cluster-3: +10.11.12.14 10.12.111.13 3 u 890 1024 377 0.089 -2.652 0.010
For each node,
does an asterisk appear in
front of a chimer node in the
remote column?
Go to Page 24Go to Page 24
Go to Page 19Go to Page 19
Yes
No
______________________________________________________________________________________________
______________________________________________________________________________________________
________________________________________________________________________________________________
24 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
24 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (8)
Page
24
You could have arrived here from:
• Page 23 - NTP - Validate time on the non-chimer
nodes (7)
You could have arrived here from:
• Page 23 - NTP - Validate time on the non-chimer
nodes (7)
Check the frequency value (also know as "drift") for each node. Run the following command, where <LNNs> is a
comma-separated list of the LNNs of all the nodes that did not have an offset within ± 0.500 seconds:
isi_for_array -n <LNNs> "ntpq -c rv"
The output will appear similar to the following:
cluster-3: assID=0 status=0664 leap_none, sync_ntp, 6 events, event_peer/strat_chg,
cluster-3: version="ntpd 4.2.4p4-o Thu Jan 30 04:27:18 UTC 2014 (1)",
cluster-3: processor="amd64", system="Isilon OneFS/v7.1.0.1", leap=00, stratum=3,
cluster-3: precision=-22, rootdelay=74.681, rootdispersion=17.897, peer=60382,
cluster-3: refid=10.12.111.13,
cluster-3: reftime=d7b25e37.038aafea Wed, Sep 3 2014 20:42:47.013, poll=6,
cluster-3: clock=d7b25ea2.4d449a13 Wed, Sep 3 2014 20:44:34.301, state=4,
cluster-3: offset=-0.326, frequency=26.973, jitter=0.061, noise=0.052,
cluster-3: stability=0.008, tai=0
The frequency value for each node should NOT be equal to
exactly 500 or -500 (it should be between -500 and 500). Go
down one of these branches for each node:
For each node that is equal to 500 or -500, go to page 25.
For each node that is not equal to 500 or -500, go to page 27.
CAUTION!Remember to go
down the proper
branch for each
node. You might
need to go down
different branches
for different nodes.
Go to Page 27Go to Page 27 Go to Page 25Go to Page 25
Not equal to
500 or -500
Equal to
500 or -500
____________
____________
25 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
25 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (9)
Page
25
You could have arrived here from:
• Page 24 - NTP - Validate time on the non-chimer
nodes (8)
You could have arrived here from:
• Page 24 - NTP - Validate time on the non-chimer
nodes (8)
Reset the time on the nodes whose frequency is equal to -500 or 500, where <LNNs> is a comma-separated list of the
LNNs of all the nodes whose frequency is equal to -500 or 500, and <chimerIP> is the IP address of a chimer node
(see page 7 for instructions on how to determine the chimer nodes):
isi_for_array -n <LNNs> "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub <chimerIP>"
Wait at least 5 minutes.
Refresh the time on the nodes by running the following command, where
<LNNs> is a comma-separated list of the LNNs of all the same nodes as
in the previous step (the nodes with frequency equal to -500 or 500), and
<chimerIP> is the same chimer IP address used in the previous step:
isi_for_array -n <LNNs> "ntpdate -ub <chimerIP>"
Wait 24 hours.
Go to Page 26Go to Page 26
__________
26 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
26 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (10)
Page
26
You could have arrived here from:
• Page 25 - NTP - Validate time on the non-chimer
nodes (9)
You could have arrived here from:
• Page 25 - NTP - Validate time on the non-chimer
nodes (9)
Check the offset by running the following command, where <LNNs> is a comma-
separated list of the LNNs of the same nodes used in the previous two steps (nodes
with frequency equal to -500 or 500), and <timeSource> is one of the external
time sources (preferably the same one you used previously):
isi_for_array -n <LNNs> "ntpdate -qub <timeSource>"
Is the offset within
± 0.500 seconds?
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
Go to Page 22Go to Page 22
No
Yes
27 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
27 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (11)
Page
27
You could have arrived here from:
• Page 24 - NTP - Validate time on the non-chimer
nodes (8)
You could have arrived here from:
• Page 24 - NTP - Validate time on the non-chimer
nodes (8)
Refresh the time on all of the nodes whose frequency is not equal to -500
or 500 by running the following command, where <LNNs> is a comma-
separated list of the LNNs of all the nodes that did not have a frequency
of 500 or -500, and <chimerIP> is the IP address of a chimer node (see
page 7 for instructions on how to determine the chimer nodes):
isi_for_array -n <LNNs> "ntpdate -ub <chimerIP>"
Wait 24 hours.
Check the offset by running the following command, where
<timeSource> is the domain name or IP address of one of the
external time sources (preferably the same one you used previously):
isi_for_array -s "ntpdate -qub <timeSource>"
See the note box on page 21 for example output.
Is the offset
for each node
within ± 0.500
seconds?
Go to Page 28Go to Page 28
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
No
Yes
__________
____________
28 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
28 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
NTP - Validate time on the non-chimer nodes (12)
Page
28
You could have arrived here from:
• Page 27 - NTP - Validate time on the non-chimer
nodes (11)
You could have arrived here from:
• Page 27 - NTP - Validate time on the non-chimer
nodes (11)
Check the frequency value (also known as "drift") again. Run the
following command, where <LNNs> is a comma-separated list of the
LNNs of all the nodes that did not have an offset within ± 0.500 seconds:
isi_for_array -n <LNNs> "ntpq -c rv"
See the example output on page 23.
For each node, is the
frequency value equal to
exactly 500 or -500?
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
Yes
Go to Page 22Go to Page 22No
____________
29 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
29 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
SMB time
Page
29
You could have arrived here from:
• Page 8 - How do you keep time?
You could have arrived here from:
• Page 8 - How do you keep time?
IMPORTANTThis is a temporary workaround only. This
procedure synchronizes the time
temporarily, but the cluster will slide back out
of sync fairly frequently - possibly every few
days, maybe as often as every few hours.
Choose an arbitrary IP address in the 168.154.1.<xxx> range that no one is
using, where <xxx> is a number between 1 and 254. Then, ping the IP address to
confirm that no one is on the other end:
ping -c 1 169.154.1.<xxx>
The response should appear similar to the following:
ping -c 1 169.154.1.1
PING 169.154.1.1 (169.154.1.1): 56 data bytes
--- 169.154.1.1 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
Did you
receive the desired
response?
Choose a different IP
address.No
Go to Page 30Go to Page 30
Yes
30 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
30 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
SMB time (2)
Page
30
You could have arrived here from:
• Page 29 - SMB time
You could have arrived here from:
• Page 29 - SMB time
Add the IP address to the list of NTP servers, by running
the following command, where <IP> is the IP address:
isi_ntp_config add server <IP>
Wait five minutes.
Delete the IP address from the list of NTP servers by running
the following command, where <IP> is the IP address that
you added to the list of NTP servers in the previous step:
isi_ntp_config delete server <IP>
SMB time is enabled and it will attempt to find an Active
Directory (AD) server to synchronize against.
Is the time
synchronized
correctly against an
AD server?
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
Note the page number that you
are currently on.
Upload log files and contact Isilon Technical
Support, as instructed in Appendix A.
No
Repeat as needed for
all NTP servers that
need to be validated.
End troubleshooting
Yes
31 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
31 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Contact Isilon Technical Support
If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.
This information and the log file will help Isilon Technical Support staff resolve your case more quickly.
Contact Isilon Technical Support
If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.
This information and the log file will help Isilon Technical Support staff resolve your case more quickly.
Appendix A: If you need further assistance
Upload node log files and the screen log file to Isilon Technical Support
1. When troubleshooting is complete, in the command-line interface, type exit to end your screen session.
2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your
method of uploading files. If you are not sure which method to use, use FTP.
ESRS:
isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support/screenlog.0
FTP:
isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support/screenlog.0
HTTP:
isi_gather_info --http --local-only -f /ifs/data/Isilon_Support/screenlog.0
SMTP:
isi_gather_info --email --local-only -f /ifs/data/Isilon_Support/screenlog.0
SupportIQ:
Copy and paste the following command.
Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly
as it appears on the page), but when you press Enter, the command will run as it should.
isi_gather_info --local-only -f /ifs/data/Isilon_Support/screenlog.0 --noupload \
--symlink /var/crash/SupportIQ/upload/ftp
3. If you receive a message that the upload was unsuccessful, refer to article 304567 on the EMC Online Support site for
directions on how to upload files over FTP.
______________________
Upload node log files and the screen log file to Isilon Technical Support
1. When troubleshooting is complete, in the command-line interface, type exit to end your screen session.
2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your
method of uploading files. If you are not sure which method to use, use FTP.
ESRS:
isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support/screenlog.0
FTP:
isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support/screenlog.0
HTTP:
isi_gather_info --http --local-only -f /ifs/data/Isilon_Support/screenlog.0
SMTP:
isi_gather_info --email --local-only -f /ifs/data/Isilon_Support/screenlog.0
SupportIQ:
Copy and paste the following command.
Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly
as it appears on the page), but when you press Enter, the command will run as it should.
isi_gather_info --local-only -f /ifs/data/Isilon_Support/screenlog.0 --noupload \
--symlink /var/crash/SupportIQ/upload/ftp
3. If you receive a message that the upload was unsuccessful, refer to article 304567 on the EMC Online Support site for
directions on how to upload files over FTP.
___________
32 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
32 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Decision diamondYes No
Process stepProcess step with command:
command xyz
Go to Page #
Page
# Note Provides context and additional
information. Sometimes a note is linked
to a process step with a colored dot.
CAUTION!Caution boxes warn that
a particular step needs
to be performed with
great care, to prevent
serious consequences.
End point Document ShapeCalls out supporting documentation
for a process step. When possible,
these shapes contain links to the
reference document.
Sometimes linked to a process step
with a colored dot.
Optional process step
Directional arrows indicate
the path through the
process flow.
IntroductionDescribes what the section helps you to
accomplish.
You could have arrived here from:
• Page 4 - Start Troubleshooting
You could have arrived here from:
• Page 4 - Start Troubleshooting
Appendix B: How to use this flowchart
33 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
33 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
You could have arrived here from:
• Page 13 - NTP - Chimer node time is incorrect (2)
You could have arrived here from:
• Page 13 - NTP - Chimer node time is incorrect (2)
Appendix C: Peering table
Example output
The asterisk indicates the "peer", or source of time, for each node.
cluster-1# isi_for_array -n 1,2,4 "ntpq -np"
cluster-4: remote refid st t when poll reach delay offset jitter
cluster-4: ==============================================================================
cluster-4: +10.11.12.13 10.12.111.13 3 u 474 1024 277 0.059 -5.573 0.041
cluster-4: 10.11.12.13 10.12.111.13 3 u 434 1024 377 0.058 -0.180 0.019
cluster-4: 127.127.1.1 .LOCL. 13 l 2 64 377 0.000 0.000 0.000
cluster-4: *10.11.120.120 10.225.225.225 2 u 471 1024 377 74.846 -0.001 0.052
cluster-1: remote refid st t when poll reach delay offset jitter
cluster-1: ==============================================================================
cluster-1: *10.11.120.120 10.254.225.252 2 u 817 1024 277 13.966 0.267 0.024
cluster-1: +192.11.12.130 10.12.111.13 3 u 790 1024 377 0.068 0.499 0.063
cluster-1: +192.11.12.130 10.12.111.13 3 u 805 1024 377 0.060 0.663 0.049
cluster-1: 127.127.1.1 .LOCL. 9 l 40 64 377 0.000 0.000 0.000
cluster-2: remote refid st t when poll reach delay offset jitter
cluster-2: ==============================================================================
cluster-2: +10.254.140.21 10.225.225.225 2 u 955 1024 277 75.740 -5.071 0.057
cluster-2: *192.11.12.130 10.11.120.120 3 u 834 1024 377 0.083 -0.440 0.078
cluster-2: 192.11.12.130 10.12.111.13 3 u 880 1024 377 0.061 0.165 0.007
cluster-2: 127.127.1.1 .LOCL. 11 l 36 64 377 0.000 0.000 0.000
Notes on asterisks and remote and refid columns
The asterisk (*) indicates which time source is the "peer," or source of time, for each node.
The asterisk itself is located to the left of the remote column, but your chosen time source may be in either the
remote or refid column.
The remote column identifies the time source from which the node gets its time. The refid column identifies the
time source from which the source listed in the remote column gets its time.
Every chimer node should have an asterisk in front of one of your chosen time sources, in either the remote or
refid column.
The asterisk should NOT be in front of the IP address for .LOCL. in the refid column.
In the example on this page, node 2 illustrates the asterisk in front of your external time source in the refid column.
For more information about how to read the peering table, see the next page.______________________
34 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
34 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
You could have arrived here from:
• Page 33 - Appendix C: Peering table
You could have arrived here from:
• Page 33 - Appendix C: Peering table
Appendix C: Peering table (2)
The peering table appears in the output of an ntpq -p command. The tables in this appendix describe the
various features of the peering table.
Variable Description
symbol single-character code indicating current value of the select field of the peer status word
remote host name (or IP number) of peer
refid association ID or IP of the time source of the next higher stratum
st stratum
t u: unicast, b: broadcast, l: local
when sec/min/hr since last received packet
poll poll interval (log2 s)
reach reach shift register (octal)
delay roundtrip delay in milliseconds
offset offset in milliseconds
jitter phase noise value - the change of the rate of change in the source clock
The following table describes what each column in the peering table means:
Variable Description
symbol single-character code indicating current value of the select field of the peer status word
remote host name (or IP number) of peer
refid association ID or IP of the time source of the next higher stratum
st stratum
t u: unicast, b: broadcast, l: local
when sec/min/hr since last received packet
poll poll interval (log2 s)
reach reach shift register (octal)
delay roundtrip delay in milliseconds
offset offset in milliseconds
jitter phase noise value - the change of the rate of change in the source clock
The following table describes what each column in the peering table means:
Symbol Description
spaceThe peer is discarded as unreachable, synchronized to this server (synch loop), or
exhibiting significant synchronization distance.
x The peer is discarded by the intersection algorithm as a falseticker.
.The peer is discarded as not among the first ten peers sorted by synchronization
distance and so is probably a poor candidate for further consideration.
- The peer is discarded by the clustering algorithm as an outlier.
+ The peer is a survivor and a candidate for the combining algorithm.
#The peer is a survivor, but not among the first six peers sorted by synchronization
distance. If the association is ephemeral, it may be demobilized to conserve resources.
*The peer has been declared the system peer and lends its variables to the system
variables.
o
The peer has been declared the system peer and lends its variables to the system
variables. However, the actual system synchronization is derived from a pulse -per-
second (PPS) signal, either indirectly via the PPS reference clock driver or directly via
kernel interface.
Meaning
reject
falsetick
excess
outlier
candidate
selected
peer
pps.peer
The following table describes each character that might precede a peer name in the peering table:
Symbol Description
spaceThe peer is discarded as unreachable, synchronized to this server (synch loop), or
exhibiting significant synchronization distance.
x The peer is discarded by the intersection algorithm as a falseticker.
.The peer is discarded as not among the first ten peers sorted by synchronization
distance and so is probably a poor candidate for further consideration.
- The peer is discarded by the clustering algorithm as an outlier.
+ The peer is a survivor and a candidate for the combining algorithm.
#The peer is a survivor, but not among the first six peers sorted by synchronization
distance. If the association is ephemeral, it may be demobilized to conserve resources.
*The peer has been declared the system peer and lends its variables to the system
variables.
o
The peer has been declared the system peer and lends its variables to the system
variables. However, the actual system synchronization is derived from a pulse -per-
second (PPS) signal, either indirectly via the PPS reference clock driver or directly via
kernel interface.
Meaning
reject
falsetick
excess
outlier
candidate
selected
peer
pps.peer
The following table describes each character that might precede a peer name in the peering table:
35 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
35 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your
Isilon Cluster
For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.
We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.
______________________________________________________
______________________________________
Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.
Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.
Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.
EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com
Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.
Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.
Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.
EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com