This post is about one of the most challenging PowerCLI scripts I was asked to write so far.
Unfortunately I will not present the script itself, simply because it is way too long and too much “environment specific” for public domain.
Still, I would like to share my experience on how handling “asynchronous tasks” in PowerCLI caused my great confusion, so here we go.
Not so long ago I was asked to help in migration of medium-sized (~3000 VMs with total ~500TB worth of VMDK files), multi-clustered, but single-vCenter VMware Infrastructure from “legacy”, to “brand new” storage back-end.
The solution proposed by storage vendor wasn’t satisfactory for the customer and deadline was as always tight.
The only luxury I had was that new storage system was fully provisioned (up to VMFS datastore creation) and reachable by all ESXi hosts in “compute” layer of the infrastructure.
I decided to go “full retard” with this one and created a PowerCLI script, that would spawn a PowerShell scriptblock (so separate process) for every cluster in the environment, then within the scriptblock enumerated the VMs and starting with the (storage-wise) largest ones dispatched a bunch of svMotion tasks to migrate workloads from “legacy” to “brand new” storage.
The result was almost 900 lines of PowerCLI “spaghetti code”, which to my great pride managed to saturate multi-path FC links between “compute” enclosures and storage but still completed the migration in time.
After I learned that I needed to load PowerCLI modules and re-connect vCenter from inside of every scriptblock I spawn, I realized the biggest challenge will be monitoring the progress of these “per-cluster threads” and multiple asynchronous tasks inside of every “thread”.
When it comes to (per-cluster) scriptblock monitoring, I dealt with that in rather crude way: I just made each “thread” to write its progress after every N migrations to a logfile labelled with cluster name and I wrote a “control loop” in the main block, which was reading these files every N minutes.
I can see people with experience in high-level programming frown on such implementation of “inter process communication”, but hey! I was under time pressure!
At the core of “per-cluster scriptblock” there was obviously a “while loop” with Move-VM PowerCLI cmdlet in the middle.
To be efficient I needed to fire these commands with -RunAsync parameter, and this is where the real challenge of controlling asynchronous tasks emerged, especially when it turned out I had to deal with more than one class of objects returned by Move-VM.
I wanted to run as much as possible of svMotions in parallel, but I didn’t want to overload (single, remember?) vCenter with too many tasks, so I decided that for each cluster I would not start more “async” svMotion tasks, than the half of number of ESXi hosts in the cluster (this by the way, was one of the very few assumptions that proved correct in the case of this script).
The version of ESXi I used back then had a limit of maximum 2 svMotion operations per ESXi host and starting with the largest (disk-wise) VMs in the cluster I didn’t really have control of where on Earth DRS decided to run them.
For example in one of the largest clusters (32 hosts) of the environment in question, I aimed at running 16 svMotions simultaneously, but these 16 VMs in fact could have been placed on the same host and I didn’t have time to wait until those lame “Resources currently in use by other operations. Waiting” messages go away.
So I decided to detect such condition by keeping a simple hash-table with “state” (which host is doing how many svMotions) of the cluster and just vMotion the VM selected for migration to “brand new” storage out of its “native host”, before it was to break “2 svMotions per host” limit.
That “pro-active” vMotion was also Move-VM … -RunAsync cmdlet, and this is where the surprise was waiting.
Just have a look at the screenshots below.
Picture 1. svMotion async task returns “server-side” task object class.
Move-VM used for svMotion (Storage vMotion) in asynchronous mode returns object of “server-side” task class (VMware.VimAutomation.ViCore.Impl.V1.Task.TaskImpl)
You can see, that once we query vCenter for currently running tasks, our “RelocateVM_Task” is progressing, but (surprise! or not really?) the PercentComplete property of “client-side” variable $svmotion_task is not dynamically updated.
Things are somewhat different, when we use Move-VM for “regular” vMotion.
Picture 2. vMotion async task returns “client-side” task object class.
You can immediately see we are dealing with an object of different class (I call it “client-side” task): VMware.VimAutomation.ViCore.Util10.ClientSideTaskImpl.
This class has (just to confuse us? ) all the attributes and methods of “server-side” class (and a few more, so it looks like a “derived class” to my layman eyes), but what is nice about it, the properties interesting for us
(like State, Result, or PercentComplete) are dynamically updated for the variable we are using ($vmotion_task)
To make things even more confusing, both “server-side” and “client-side” tasks show up in Get-Task output and there seems to be no easy way to distinguish between them.
Picture 3. Get-Task output for “server-side” and “client-side” tasks.
These subtle differences imply different possibilities of how to monitor different types of asynchronous tasks (“server-side” vs “client-side”).
When you have a “server-side” task (like Storage vMotion), you basically have no option but to use LucD canonical method:
Save your async tasks to a hash-table (with MoRef Id to “server-side” task as a hash-key), then periodically parse the output of Get-Task to check the status of your tasks.
While dealing with “client-side” tasks (like “regular” vMotion) you can skip Get-Task part, just keep the similar hash-table of tasks at hand and enumerate it periodically (because the attributes related to progress and state of the task are dynamically updated, remember?).
This should be faster and less resource consuming on both client and vCenter end.
Quite obviously even the tasks represented by “client-side” task object in PowerCLI are executed and controlled by server(s) – in the end vMotion happens between two ESXi hosts and is controlled by vCenter.
I can only speculate that “client-side” class maintains some “underwater” connection to vCenter to update its attributes.
Apparently software architects decided to implement it only for tasks that are expected to complete relatively fast (vMotion is expected to complete much faster than svMotion), so that monitoring of them do not depend too much on connectivity between PowerCLI client and vCenter.
This solution, while a bit confusing, can be really helpful once you are aware of it (you can skip querying vCenter with Get-Task for “client-side” tasks).
I encourage you to experiment a little on your own, with different kinds of asynchronous tasks in PowerCLI, to get idea what operation returns which type of task.
Remove-Snapshot cmdlet for example (just like New-VM in Luc’s post above, you can also see how I was handling that part in one of my early posts) returns “server-side” task object, and this seems to support my hypothesis, that only “quick tasks” get to be implemented as “client-side” ones.
Picture 4. Remove-Snapshot async task.
Last but not least, there are operations that do not have dedicated PowerCLI cmdlets, but we are still able to initiate them as asynchronous tasks, by directly invoking respective object’s method.
Good example of such operation is VMDK consolidation.
Picture 5. Invoking ConsolidateVMDisks_Task() method for VM object directly.
Not so surprisingly, invoking a method in that way returns a ManagedObjectReference only (as you can see below).
Picture 6. Can’t really expect a method to return task object.
But we can easily check, that ConsolidateVMDisks_Tasks is also a “server-side” task.
Picture 7. Disk consolidation is also “server-side” task.
And this brings us to the end of my adventure with PowerCLI asynchronous tasks, I would like the “take away” from this post to be something like that:
Asynchronous tasks are great way to speed up things by executing multiple operations in parallel, just ensure you monitor the progress of each task properly (it is easy to end-up in an infinite loop of Get-Task cmdlets, if you don’t check for “failure” condition of a task for example) and make sure you know which type of asynchronous task (“server-side” vs “client-side”) you are dealing with.
The simplest way to verify that is to use Get-Member (gm) cmdlet.
I hope you find this post useful, feel free to share it and I welcome you to provide any feedback you have!
