PowerCLI script to report and change host NTP settings

The script I would like to share today is in a way supplementary to what I wrote about in my previous post.

For those who don’t know it yet – last week I gave an example of completely turning off VMware Tools guest time synchronization via script that is able to set any of virtual machine “advanced settings”.

Now I would like to look at subject of timekeeping from vSphere host perspective, I think we all agree that you should have your time synchronized and correct among all components of virtual infrastructure – regardless if you synchronize Guest OS time with host or not.
It’s not only about “feeling nice”, as being out of sync can be a reason for many at least annoying issues – like the one described in this KB. Other issues include but are not limited to problems with snapshots (so backups!), vmotions and so on and so forth…

Of course the best practice to implement NTP on vSphere hosts is to do it via host profiles and monitoring whether the time is not drifting away should be done with vCOPs or whatever monitoring tool you are using, but putting host in maintenance mode and applying the whole profile might be too much of work if you just need to point your hosts to new time sources.
Or maybe you are consulting for some smaller customer that does not have fully matured monitoring framework in place and you just need to have a quick report of timekeeping settings in such environment.

This is where the following script comes handy.

#requires -version 2
<#

.SYNOPSIS
    Script can be used to report or setup NTP configuration on all vSphere hosts in given cluster

.DESCRIPTION
    Script takes vCenter Server name and host cluster name as mandatory parameters, NTPSources parameter is optional.
    If only mandatory parameters are provided script generates report aobut NTP settings for all vSphere hosts that are
    connected to cluster. Report include ntp service status, policy, up to 5 ntp servers configured and calculated time difference
    between host and system where the script is invoked. For consistent results script should be run from vCenter Server.
    Optional parameter NTPSources is a comma-separated list of ntp servers that will be configured in cluster.
    If NTPSources paramter is provided the script will configure ntp service to start together with host ("on" policy),
    configure ntp servers provided, set the time manually (to avoid drift problems) and restart ntpd.

.PARAMETER vCenterServer
    Mandatory parameter indicating vCenter server to connect to (FQDN or IP address)

.PARAMETER ClusterName
    Mandatory parameter indicating host cluster name where vms need to be reconfigured

.PARAMETER NTPSources
    Optional parameter indicating NTP servers that will be used

.EXAMPLE
    To configure two NTP servers provide all parameters.

    vmhost-timekeeping.ps1 -vCenterServer vcenter.seba.local -ClusterName Production-Cluster -NTPSources "time01.seba.local,time02.seba.local,10.0.0.1"

.EXAMPLE
    If you provide -NTPSources only the script will ask for mandatory parameters

    vmhost-timekeeping.ps1 -NTPSources "time01.seba.local,time02.seba.local,10.0.0.1"

.EXAMPLE
    To generate report about NTP service status provide only mandatory parameters.

    vmhost-timekeeping.ps1 -vcenter 10.0.0.1 -cluster tdq-cluster

.EXAMPLE
    Script will interactively ask for two mandatory parameters, no changes will be made, only report will be created.

    vmhost-timekeeping.ps1
#>

[CmdletBinding()]
Param(
  [Parameter(Mandatory=$True,Position=1)]
   [string]$vCenterServer,
   [Parameter(Mandatory=$True, Position=2)]
   [string]$ClusterName,
   [Parameter(Mandatory=$False, Position=3)]
   [string]$NTPSources=""
)

Function Write-And-Log {

[CmdletBinding()]
Param(
  [Parameter(Mandatory=$True,Position=1)]
   [string]$LogFile,
   [Parameter(Mandatory=$True,Position=2)]
   [string]$line,
   [Parameter(Mandatory=$False,Position=3)]
   [int]$ErrorCount=0,
   [Parameter(Mandatory=$False,Position=4)]
   [string]$type="terse"
)

$LogEntry = (Get-Date -Format ("[yyyy-MM-dd HH:mm:ss] ")) + $line
$ui = (Get-Host).UI.RawUI

if ($ErrorCount) {

   $ui.ForegroundColor = "red"
   $LogEntry = ">>> ERROR <<< " + $LogEntry
   Write-Output $LogEntry
   $LogEntry | Out-File $LogFile -Append

}
else {

   $ui.ForegroundColor = "green"
   if ($type -ne "terse"){
      Write-Output $LogEntry
      $LogEntry | Out-file $LogFile -Append
   }
   else {
      Write-Output $LogEntry
   }

}

$ui.ForegroundColor = "white"
}

#constans
$maxtimedrift = 1

#variables
$ScriptRoot = Split-Path $MyInvocation.MyCommand.Path
$StartTime = Get-Date -Format "yyyyMMddHHmmss_"
$csvoutfile = $ScriptRoot + "\" + $StartTime+ "time_config_report_for_$($ClusterName)_cluster.csv"
$logfilename = $ScriptRoot + "\" + $StartTime + "vmhost-timekeeping.log"
$transcriptfilename = $ScriptRoot + "\" + $StartTime + "vmhost-timekeeping_Transcript.log"
$all_vmhosts_timeconfig_info = @()
$total_errors = 0
$total_vmhosts = 0

#start PowerShell transcript
Start-Transcript -Path $transcriptfilename

#load PowerCLI snap-in
$vmsnapin = Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue
$Error.Clear()
if ($vmsnapin -eq $null)
    {
    Add-PSSnapin VMware.VimAutomation.Core
    if ($error.Count -eq 0)
        {
        write-and-log $logfilename "PowerCLI VimAutomation.Core Snap-in was successfully enabled." 0 "full"
        }
    else
        {
        write-and-log $logfilename "Could not enable PowerCLI VimAutomation.Core Snap-in, exiting script" 1 "full"
        Exit
        }
    }
else
    {
    write-and-log $logfilename "PowerCLI VimAutomation.Core Snap-in is already enabled" 0 "full"
    }

#check PowerCLI version
if (($vmsnapin.Version.Major -gt 5) -or (($vmsnapin.version.major -eq 5) -and ($vmsnapin.version.minor -ge 1))) {

    #assume everything is OK at this point
    $Error.Clear()

    #connect vCenter from parameter
    Connect-VIServer -Server $vCenterServer -ErrorAction SilentlyContinue | Out-Null

    #execute only if connection successful
    if ($error.Count -eq 0){

        #measuring execution time is really hip these days
        $stop_watch = [Diagnostics.Stopwatch]::StartNew()

        #use previously defined function to inform what is going on, anything else than "terse" will cause the message to be written both in logfile and to screen
        Write-And-Log $logfilename "vCenter $vCenterServer successfully connected" $error.count "full"

        #get all reachable vmhosts in cluster
        $vmhosts_in_cluster = get-vmhost -location $ClusterName | where-object { ($_.connectionstate -eq "connected") -or ($_.connectionstate -eq "maintenance") }

        #only if we've found some vmhosts
        if ($vmhosts_in_cluster){

            #if no NTP server given - create report only
            if ($NTPSources -eq ""){

                $mode = "checked"
                foreach ($vmhost in $vmhosts_in_cluster){

                        #all OK here
                        $error.Clear()
                        $total_vmhosts += 1
            
                        #display nice progress bar in PowerCLI window
                        write-progress -Activity "Gathering host NTP config report" -Status "Percent complete" -PercentComplete (($total_vmhosts / $vmhosts_in_cluster.count) * 100) -CurrentOperation "$("{0:N2}" -f (($total_vmhosts / $vmhosts_in_cluster.count) * 100))% complete"
                    
                        #retrieve NTPD information
                        $single_vmhosts_timeconfig_info = New-Object PSObject
                        $single_vmhosts_timeconfig_info | Add-Member -Name "VmHostName" -Value $vmhost.name -MemberType NoteProperty
                        $single_vmhosts_timeconfig_info | Add-Member -Name "VmHostTZ" -Value $vmhost.TimeZone -MemberType NoteProperty
                        $ntpservice = $vmhost | get-vmhostservice | Where-Object {$_.key -eq "ntpd"}
                        $single_vmhosts_timeconfig_info | Add-Member -Name "NTPDisRunning" -Value $ntpservice.running -MemberType NoteProperty
                        $single_vmhosts_timeconfig_info | Add-Member -Name "NTPDPolicy" -Value $ntpservice.policy -MemberType NoteProperty
                    
                        #retrieve NTP Servers configured, report only first 5
                        $ntpserver = @($vmhost | get-vmhostntpserver)
                        for ($index = 0; $index -lt 5; $index++){
                            if ($ntpserver[$index]){
                                $single_vmhosts_timeconfig_info | Add-Member -Name "NTPServer$($index)" -Value $ntpserver[$index] -MemberType NoteProperty
                            }
                            else{
                                $single_vmhosts_timeconfig_info | Add-Member -Name "NTPServer$($index)" -Value "none" -MemberType NoteProperty
                            }
                        }
                    
                        #calculate time difference between host and system this script is invoked from
                        $hosttimesystem = get-view $vmhost.ExtensionData.ConfigManager.DateTimeSystem
                        $timedrift = ($hosttimesystem.QueryDateTime() - [DateTime]::UtcNow).TotalSeconds
                    
                        #raise alarm if difference bigger than acceptable
                        if([math]::abs($timedrift) -gt $maxtimedrift){
                            Write-And-Log $logfilename "Time difference exceeded for host $($vmhost.name)!" 1 "full"
                            Write-And-Log $logfilename "Acceptable difference: $("{0:N2}" -f $maxtimedrift)s Current difference: $("{0:N2}" -f $timedrift)s" 1 "full"
                            $total_errors++
                        }
                        $single_vmhosts_timeconfig_info | Add-Member -Name "TimeDrift" -Value $timedrift -MemberType NoteProperty
                    
                        $all_vmhosts_timeconfig_info += $single_vmhosts_timeconfig_info
                        $total_errors += $error.Count
                        Write-And-Log $logfilename "Host $($vmhost.name) added to report" $error.Count "terse"
                }
                
                #export to CSV
                $all_vmhosts_timeconfig_info | Export-Csv -Path $csvoutfile -NoTypeInformation
                Write-And-Log $logfilename "Report created in $($csvoutfile)" $total_errors "full"
            }
            
            #if NTP servers provided - configure them
            else {
            
                #give the engineer invoking the script chance to abort
                Write-And-Log $logfilename "NTP configuration for all vSphere hosts in cluster $ClusterName will be RESET" 1 "full"
                Write-And-Log $logfilename "This is your LAST CHANCE TO ABORT" 1 "full"
                Write-And-Log $logfilename "Press Y + ENTER to continue" 0 "full"
                Write-And-Log $logfilename "Press any other key + ENTER to ABORT..." 1 "full"
                $response = read-host
                if ( $response -ne "Y" ) {
                    write-and-log $logfilename "Operation ABORTED, no changes have been made to NTP settings" 1 "full"
                    #exit
                } else {
                
                    #let's sanitize input a little and leave only NTP servers that respond to ping (from system where this script is invoked!)
                    $NTPSourcesArray = $NTPSources.Split(",") | Where-Object { Test-Connection -ComputerName $_ -Quiet -Count 1}
                    $mode = "configured"
                    
                    #make sure we've got some NTP servers left
                    if ($NTPSourcesArray){
                        
                        foreach ($vmhost in $vmhosts_in_cluster){
                    
                                #all OK here
                                $error.Clear()
                                $total_vmhosts += 1
                            
                                #display nice progress bar in PowerCLI window
                                write-progress -Activity "Configuring NTP for hosts" -Status "Percent complete" -PercentComplete (($total_vmhosts / $vmhosts_in_cluster.count) * 100) -CurrentOperation "$("{0:N2}" -f (($total_vmhosts / $vmhosts_in_cluster.count) * 100))% complete"
                            
                                #stop ntp service on host
                                $ntpservice = $vmhost | get-vmhostservice | Where-Object {$_.key -eq "ntpd"}
                                stop-vmhostservice -HostService $ntpservice -confirm:$False | out-null
                            
                                #clear current NTP servers
                                $current_NTPSources = @($vmhost | get-vmhostntpserver)
                                foreach ($current_NTPSource in $current_NTPSources){
                                        remove-vmhostntpserver -ntpserver $current_NTPSource -vmhost $vmhost -confirm:$false | Out-Null
                                }
                            
                                #and set new NTP servers
                                foreach ($NTPSource in $NTPSourcesArray) {
                                        add-vmhostntpserver -ntpserver $NTPSource -vmhost $vmhost -confirm:$False | out-null
                                }
                            
                                #set service policy to start and stop with host
                                set-vmhostservice -HostService $ntpservice -Policy "on" -confirm:$False | out-null
                            
                                #set vmhost time manually (to avoid problem with too big drift) to match time of system where script is invoked
                                $hosttimesystem = get-view $vmhost.ExtensionData.ConfigManager.DateTimeSystem
                                $hosttimesystem.UpdateDateTime([DateTime]::UtcNow)
                            
                                #finally - start NTP on vmhost
                                start-vmhostservice -HostService $ntpservice -confirm:$False | out-null
                            
                                $total_errors += $error.Count
                                Write-And-Log $logfilename "Host $($vmhost.name) NTP configuration changed" $error.Count "terse"
                        }
                   }     
                   else{
                        Write-And-Log $logfilename "None of NTP servers provided ($NTPSources) is responding, exiting" 1 "full"
                        $total_errors++
                   }      
                }
            }
        }
        else {
            $total_errors += $Error.Count
        }
        $stop_watch.Stop()
        $elapsed_seconds = ($stop_watch.elapsedmilliseconds)/1000
        
        #farewell message before disconnect
        Write-And-Log $logfilename "Total of $total_vmhosts hosts $mode in $("{0:N2}" -f $elapsed_seconds)s, $total_errors ERRORS reported, exiting" $total_errors "full"
        
        #disconnect vCenter
        Disconnect-VIServer -Confirm:$false -Force:$true
    }
    else {
        Write-And-Log $logfilename "Error connecting vCenter server $vCenterServer, exiting" $error.count "full"
    }
}
else {
    write-and-log $logfilename "This script requires PowerCLI 5.1 or greater to run properly" 1 "full"
}
Stop-Transcript

The script can be used in two “modes” – if you provide only mandatory parameters (vCenter Server and cluster name) it will query all hosts available in the selected cluster and provide a report on NTP settings.
The report will be saved in .csv file that could look like this.

"VmHostName","VmHostTZ","NTPDisRunning","NTPDPolicy","NTPServer0","NTPServer1","NTPServer2","NTPServer3","NTPServer4","TimeDrift"
"esxhost01.seba.local","UTC","False","off","10.0.0.1","time02.seba.local","none","none","none","10802.5872341"
"esxhost02.seba.local","UTC","True","on","10.0.0.1","time02.seba.local","none","none","none","0.0658459"
"esxhost03.seba.local","UTC","True","on","10.0.0.1","time02.seba.local","none","none","none","0.0676079"

Most fields in the report are self-explanatory I think, I decided to fix the number of reported NTP servers to 5, because I wanted to avoid difficulties of exporting variable length strings to .csv that I had to deal with some time ago. Most of the time I see only two NTP servers configured – which is perfectly fine, the highest number I ever saw was 4 (and that’s an overkill IMHO), so reporting 5 time sources should  be more than enough.

“NTPDPolicy” field might require some explanation, three values of “automatic”, “on” or “off” are possible for this field and they are corresponding to settings that you can see in GUI.

So “automatic” = “Start automatically if any ports are open and stop when all ports are closed”,
“on” = “Start and stop with host”
and “off” = “Start and stop manually”.

In my opinion best policy for ntp daemon is “on”, just because we always want our time synced properly. This policy also makes things easier since configuring it instructs our host to automatically update its security profile and open appropriate ports on ESXi firewall.
It is actually the other way round if we set service policy to “automatic” – service starts only when we open the ports manually, this policy might be useful for services that we use temporarily, like, say sshd, if we need to open ssh session to ESXi host we just open ports on firewall via vSphere Client and the service starts automatically (we don’t need to remember to start it), but honestly it does not make much sense with ntpd.
“Off” policy effectively disables ntpd.

You can see also “timedrift” reported for each host, this value is difference (in seconds) between current time on the host and the system you invoked the script from. I’d like to stress out that because of the way this comparison is implemented, you got to be sure time on your workstation or server where you run this script is correct – personally I was always invoking this script from vCenter Server for given infrastructure.

In line 98 of the script I defined a constant ($maxtimedrift) value of what is “acceptable” time skew for hosts. I configured it to 1 second (so maximum difference between any given pair of hosts should not be bigger than 2 seconds), but properly working ntpd is able to keep this difference within milliseconds, so feel free to adjust this value to your needs (the script will throw red lines with exclamation marks at you if it detects larger time drift during execution).

As you can see in the report one of the vsphere hosts I checked, namely esxhost01.seba.local, has some problems with being on time, ntpd is turned off there and measured time skew is more than 10 thousands seconds (or precisely 3 hours).
We definitely need to fix that and this is where 2nd “mode” of my script comes to the rescue. By simply providing the script optional -NTPDSources parameter you instruct it to change ntpd settings.

You can provide just one NTP server like this:

vmhost–timekeeping.ps1 –vCenterServer vcenter.seba.local –ClusterName Production–Cluster –NTPSources time01.seba.local

But if you want to provide multiple time sources separate them with commas and enclose the whole list in quotation marks like that:

vmhost–timekeeping.ps1 –vCenterServer vcenter.seba.local –ClusterName Production–Cluster –NTPSources “time01.seba.local,time02.seba.local,10.0.0.1”

and make sure you don’t put any spaces in this string, cause test-connection cmdlet (that I use later on to “sanitize” input) will fail if you provide it with servername that starts with blank character (sic!).

The script will give you a warning and one last chance to abort (just in case you had it invoked by mistake) but if you confirm the execution, it will first try to ping each NTP server you provided (and remove not responding ones – this is my crude way of eliminating typos from input, so be sure your time sources pong for pings from the system you are running this script or disable this check 😉 ).
Once the list of NTP servers is confirmed the script will first stop ntpd service then clear the currently configured time sources for each host in selected cluster.

Please note this is not “modifying” or “adjusting” NTP configuration, it is “wipe out everything and start from scratch” approach.

Subsequently NTP servers are added, ntpd service policy is configured for “on” and script configures time on host to match time of the system it is invoked from.
I do that because in case of big time differences (like 3 hours in our example) ntpd will not adjust time of host without a reboot, please be careful with this, since (as I explained before) the script assumes as correct the current time of system it is running on.
Make sure this assumption is valid and run this script (in “setup mode” at least) from your vCenter Server if possible (you’ve got your time correct of vCenter, right?, if not at least you don’t end-up with time difference between vCenter and hosts).

As a final step ntpd service is started with start-vmhostservice cmdlet.

I like to be informed if a script is actually doing something (don’t like scripts that can’t communicate anything but blinking cursor and you need to wonder whether it hangs or what) so I made this script rather chatty.
It throws at least one line for each host that has been added to report or changed and what I like most, there is a progress bar displayed with write-progress cmdlet at the top of PowerCLI window.

I hope you will find this script useful, as always – feel free to share and provide your feedback