Error 111 or 503 Service Unavailable – after ESX restart

It might happen that after restart of the ESX it’s not possible to connect it back to vCenter or connect directly with vSphere client it’s not possible. Problem usually disappears after 15 minutes or so. vSphere client gives an error:

The server could not interpret the communication from the client ( The remote server returned an error : (503) Server Unavailable)

Restart management services doesn’t help ( service mgmt-vmware restart ) , querying registered VM’s on ESX host gives an error

[root@esx01 ~]# vmware-cmd -l
vmware-cmd vim.fault.HostConnectFault: Failed to acquire session manager while querying for local ticket. (503 Service Unavailable)
or
vim.fault.HostConnectFault: Failed to acquire session manager while querying for local ticket. ((111, ‘Connection refused’))

Both errors are caused by this same ESX misconfiguration – DNS and\or wrong host name specified in configuration files. When ESX boots first it check DNS to solve its hostname, after several minutes (if DNS services still not available) checks a /etc/hosts to resolve a host name, usually check takes up to 15 minutes. After 15 minutes host become available. But definitely 15 minutes it’s too much. How to fix it and prevent from reoccurring ?

  1. Check if DNS it’s available, do nslookup- one should get FQDN of the host from DNS server.
  2. If DNS it’s not available, wipe out /etc/resolve.conf file
  3. Check /etc/hosts . Hosts file should have IP address, hostname and hostname with domain name (FQDN)
  4. Just in case update ESX advance setting with correct ESX hostname , executer esxcfg-advcfg command :
    esxcfg-advcfg -s /Misc/HostName
  5. restart management services mgmt-vmware restart
  6. After a minute try connect via vSphere client to ESX host.

 

Artur Krzywdzinski

Artur is Consulting Architect at Nutanix. He has been using, designing and deploying VMware based solutions since 2005 and Microsoft since 2012. He specialize in designing and implementing private and hybrid cloud solution based on VMware and Microsoft software stacks, datacenter migrations and transformation, disaster avoidance. Artur holds VMware Certified Design Expert certification (VCDX #077).