A week ago folks twitted out about problem with NFS storage after upgrade to vSphere 5.5 U1 . I decided to blog out about it because not too many people are aware about the problem and still planning upgrade to vSphere 5.5 U1 .
RT @slysiou: ALERT: Don’t upgrade to U1 of #vSphere 5.5 if you use #NFS (any vendor). #VMware KB is coming.
— Jorge de la Cruz (@jorgedlcruz) April 17, 2014
[box type=”info”] Note: BUG discovered in vSphere 5.5 U1 affects all NFS storage vendors [/box]
Long story short, when using VMware vSphere ESXi 5.5 U1, ESXi host frequently loses paths to NFS volumes and APD (All Paths Down) are observed.
You see entries in the vobd logs similar to:
2014-04-01T14:35:08.074Z: [APDCorrelator] 9413898746us: [vob.storage.apd.start] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down state. 2014-04-01T14:35:08.075Z: [APDCorrelator] 9414268686us: [esx.problem.storage.apd.start] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down state. 2014-04-01T14:36:55.274Z: No correlator for vob.vmfs.nfs.server.disconnect 2014-04-01T14:36:55.274Z: [vmfsCorrelator] 9521467867us: [esx.problem.vmfs.nfs.server.disconnect] 192.168.1.1/NFS-DS1 12345678-abcdefg0-0000-000000000000 NFS-DS1 2014-04-01T14:37:28.081Z: [APDCorrelator] 9553899639us: [vob.storage.apd.timeout] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed. 2014-04-01T14:37:28.081Z: [APDCorrelator] 9554275221us: [esx.problem.storage.apd.timeout] Device or filesystem with identifier [12345678-abcdefg0] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.
Few days after VMware come up with kb.vmware.com/kb/2076392 and advisory to DO NOT upgrade ESXi hosts to vSphere 5.5 U1 until they will find and provide solution to customers. If you are already in vCenter vSphere 5.5 U1 there is no need to roll back change because BUG is related to ESXi hosts only. So you can keep you vCenter server in version vSphere 5.5 U1 and ESXi host in vSphere 5.5 GA with the latest security patches. See more OpenSSL Hearthbleed
UPDATE 11 June 2014 – APD issue with NFS fix and ESXi 5.5
Yesterday, 10’th of Jun 2014 VMware released fix to know issue with ESXi 5.5 and NFS storage – more info see above.
