PowerCLI Move-VM Returns failure but vm still moves

Question

I have a script that is moving VM's from one datastore to another. I have other logic to protect the Storage backend and fabric from over subscription and to wait to submit new requests at specific times (Business requirement). I am only moving datastores and not hosts. I have a try catch block around the Move-VM as I want to handle the failure. The first two or three times I ran the script only a few errors occur and were handled as I expected. The last 5 times the move returns an error but the VM move request is successful as indicated by the task I see in vCenter. This job completes successfully as well. The Move-VM is not using the RunAsync switch. What would cause Move-VM to return an error but successfully submit the move request.

$VMs = Get-VM -Name $ComputerName
$CurrentDataStores = Get-Datastore

foreach ($VM in $VMs){
   foreach ($store in $CurrentDataStores){
      if ($store.name -eq "$Datastore"){
         $rawDatastore = $store
      }
      if ($store.id -Match $VM.DatastoreIdList){
         $VMDatastore = $store.name
      } 
   }
   if ($VMDatastore -eq "$Datastore"){
      Write-Output "$VM : is already on requested Datastore"
   }
   else{
      try {
          Move-VM -VM $VM -Datastore $rawDatastore -ErrorAction Stop
      }
      catch {
         Write-OutPut "$VM : Unable to move VM to new Datastore" 
         continue
      }
   }
}

Move-VMDatastore : <VM Name> : Unable to move VM to new Datastore
At line:1 char:1
+ Move-VMDatastore -ComputerName <VM Name> -Datastore <Move TO Datastore>
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Move-VMDatastore


Move-VMDatastore : 12/24/2014 12:50:02 AM    Move-VM        Operation is not valid due to the current state of the
object.
At line:1 char:1
+ Move-VMDatastore -ComputerName <ComputerName[]> -Datastore <Datastore>
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Move-VMDatastore

Solution 1

My understanding of this issue is that in the implementation of Move-VM, PowerCLI runs an async task, gets the task and then runs Wait-Task on it.

If the operation is very fast, it will fail with this issue. This is an educated guess stemming from the fact that this is the exact exception you get when you Wait-Task for a completed task.

So basically your problem is that your VMware server is too fast...

A workaround would be to use the -RunAsync switch and implement the correct behaviour yourself. Something like:

$Task = Move-VM -VM $VM -Datastore $rawDatastore -ErrorAction Stop -RunAsync
while($true)
{
    switch ($task.State)
    {
         'Success' { $Task.Result; break }
         'Error' { throw $Task.ExtensionData.Info.Error.LocalizedMessage }
         Start-Sleep 5
    }
}

Edit: I have seen the exact same issue with Stop-VM and Remove-VM and have worked around them the same way

This issue has been submitted for improvement to the PowerCLI team here: https://powercli.ideas.aha.io/ideas/PCLI-I-208

Solution 2

Just in case anyone else comes across this issue, it would appear that the $task variable doesn't update by itself. In my case, I had to do:

 while($task.state -eq "Running")
 {
   Start-Sleep 5
   $task = Get-Task -ID $task.id
 }

Comments

[Adarsh Anurag]

It's really gonna help if you gave the exact text of the error. Also, your code.

[TechGuyTJ]

True. Edit to add code.

[Adarsh Anurag]

Change your catch block to "Write-Error $_" to see the actual error Move-VM is throwing. That will help identify the real issue.

[TechGuyTJ]

I will do that and report the new error.