Hyper V Live Migration Only Goes One Way (Error 21502)

failoverclusterlive-migrationwindows-clusterwindows-server-2008-r2

We've been running into an issue recently with one of our server stacks. Our two 2008 R2 servers are running in a cluster set up to live migrate VMs between eachother in case there is ever a detected fault.

The servers are the exact same hardware-wise; they were ordered specifically for this purpose. Live migration had been working fine up until a couple months ago when we noticed that VIR001 could not migrate to VIR002. I've looked into this issue and I know that generally it is caused by improperly-named resources, but that doesn't seem to be the case here.

VIR002 will live migrate any of its hosted VMs over to VIR001. VIR001 will not LM any VMs over to VIR002. Not sure where to start with this, I've noticed a couple Time-Server errors on VIR001, but if the issue was due to a sync problem, wouldn't both servers experience the same issue?

Right now, looking for ideas on what to check. Thanks,

(Update: I've ran the Failover Cluster Validation tool and it found no issues. I could not run the Disk validation as our cluster is still online with the cluster. Both servers in question are also set as possible owners for cluster resources)

Best Answer

Well, finally found the issue:

I noticed that some of the created cluster networks were not legitimate (ie, they only contained one NIC, or were teamed with a NIC on a different subnet). I had disabled these. I was told by my colleagues that binding on the physical servers could make a difference. I changed these. I verified the cluster, made sure all nodes had both servers listed as possible owners, and to top it off, I had found the "Network for Live Migration" tab under properties for the Virtual Machine Resource.

I had ordered the cluster networks in "Network for Live Migration" in such a way that the Live Migration cluster network was first, followed by all active networks, with the disabled networks at the bottom. No love. Today after changing the binding and seeing no change, I decided to disable the all cluster networks in the Live Migration tab beyond three internal networks (LM, host, Cluster Domain). Now it's working.

Not sure what caused this to begin with. We haven't made any physical changes to the hardware in the last year. This was working at least 4 months ago. Looks like the Cluster manager doesn't always listen to its own settings.

Thanks for the replies on this question.

Related Solutions

Unable to destroy windows 2008 r2 failover cluster after SAN rebuild

I have seen this also in the past when I deployed a failover cluster. One time I fixed it by executing the following PowerShell command: Remove-ClusterNode After that I executed the following command to clear the configuration on the node: Clear-ClusterNode You have to start the Windows Powershell Modules from the Administrative Tools otherwise these commands won't work. After that I was able to succesful join it to the cluster

If these commands don't work, I'm afraid you have to reinstall the operating system on each node of the cluster.

RPC Server Unavailable on Hyper-V cluster when moving resources after the host adapter has failed

Great question!

The most likely reason for the RPC failure is that the cluster name resource (and IP address) were likely hosted on the server whose primary network connection was flapping.

Since the interface was going up and down, access to the cluster via the cluster name would likely fail due to the network interruptions.

You should be able to execute commands against the cluster from the command line (either cluster.exe or the FailoverClusters module in PowerShell). The FailOverClusters module can be used over PowerShell remoting if the appropriate credential delegation is configured (either CredSSP or Kerberos).

In the case of a failure of the network interface hosting the cluster name, you could use PowerShell to move that cluster group to one of the nodes that is accessible, or you could just execute commands against the cluster to migrate machines, etc..

To ensure that this does not happen again, you likely will need to make the NIC highly available (NIC teaming). This is dependent on where you are managing the cluster from.. one of the servers or a remote management station. If you are managing from a cluster machine in that same cluster, you could add an IP on the cluster network to the cluster name, but you would want to make sure that was not added to DNS, otherwise you might interrupt remote management clients from being able to connect.

To add an IP Address to the cluster group via PowerShell -

$Resource = Add-ClusterResource -Name SecondaryIP -ResourceType "IP Address" -Group 'Cluster Group' 
$Resource | Set-ClusterParameter -Name 'Address' -Value 'Your-IP-Here'
$Resource | Set-ClusterParameter -Name 'SubnetMask' -Value 'Your-SubnetMask-Here'

You'll need to disable dynamic dns registration and create static entries if you don't want remote management clients trying to talk to the private network.

Best Answer

Related Solutions

Unable to destroy windows 2008 r2 failover cluster after SAN rebuild

RPC Server Unavailable on Hyper-V cluster when moving resources after the host adapter has failed

Related Topic