© 2014 VMware Inc. All rights reserved.

Component metadata health check fails with invalid state error(2145347)

Symptoms

The Virtual SAN (VSAN) Health check plugin reports the Component metadata health test as Failed.

The component UUID reports the component state as Invalid state.

At the cluster level in your vCenter Server > Monitor > Virtual San > Health, you see an error similar to:

Why it happens?

This problem occurs due to an issue in the VSAN Local Log Structured Object Management (LSOM) component that results in components with corrupted metadata.

Note: This issue does not impact the availability of any VM objects.

ResolutionThis issue is resolved in ESXi 6.0 patch ESXi600-201608001. However, there are some considerations.

To prevent this issue from occurring, upgrade to ESXi 6.0 ESXi600-201608001, available at VMware Patch Downloads. For more information on downloading the patch, see  How to download patches in MyVMware (1021623).

Important: If you do not have any invalid state components at this time, upgrading to the patch is

sufficient. If your VSAN cluster is already reporting invalid state errors, you must evacuate and

destroy the disk or disk group that contains the corrupted component(s).

To identify and destroy the corrupted components:

Step 1 - Ensure that you are on the latest on disk version for your ESXi build.  For more information on the on disk format versions, see:Understanding Virtual SAN on-disk format versions (2145267)

How to format VSAN Disk Groups with a legacy format version (2146221)

Note: This is important as newly created disks and disk groups are created in the latest on disk format by default, if the on disk format is not on the latest version, the new disk groups are not compatible.

Step 2 -

Identify the disk or disk group where the corrupt component is located using these steps:

a) Note the component UUID listed in the Component metadata health check error.

b) Connect to RVC and navigate to /localhost/<Your Datacenter>/computers folder 

c) Run the command: vsan.cmmds_find 0 -u <component uuid>

You will see an output similar to:

Step 2 Continued

d) Note the diskUuid from the preceding output.

e) Run the command: vsan.cmmds_find 0 -t DISK -u <disk uuid>

You will see an output similar to:

Step 2 Continues

f) Note the naa id. In this

case, naa.600508b1001c5c0b1ac1fac2ff96c2b2, which hosts the

corrupt component.

g) Repeat the steps a to f for all components in Invalid state and identify

each of the disks impacted.

Step 3

After identifying the disk(s) that own the corrupt components, use the vSphere Web Client to safely destroy the disk(s).

a) Open the vSphere Web Client.

b) Navigate to vCenter Server > Datacenter > Cluster > Manage > Virtual SAN > Disk Management.c) Select the disk that owns the corrupt components.

Note: If there are multiple disks that own corrupt components in the same disk group, it is easier to destroy the entire disk group if there is adequate free capacity in the VSAN cluster.

Step 3 continues

• Destroy the disk or disk group.

Warning: Do not use the No data migration option.

VMware recommends to use the Full data migration option to ensure data integrity and availability. If there is inadequate free capacity in the VSAN cluster to accommodate full migration, use the Ensure accessibility option. 

Step 4

Add the disk(s) back to the Virtual SAN configuration. For more

information, see the Add Devices to the Disk Groupsection in the 

Administering VMware Virtual SAN guide.

Additional Information

• For information on using RVC, see:VMware vSphere Blog - Managing Virtual SAN with RVC: Part 1 – Introduction to the Ruby vSphere Console

• VMware Ruby vSphere Console Command Reference for Virtual SAN