none
Hyper-v cluster 2012R2: VM won't migrate over with Live migrate or Quick migrate

    Question

  • I have a fail over Hyper-V cluster with several VM's on it.  The VM's all use to have their VHDX files sitting on a shared cluster storage.  I moved the VHDX file for my virtual DC to the local storage of one of the 2 Host servers in the cluster.  I have tried everything I can think of as far as permissions , etc, vut now that particular VM will only run on one side of the hyper-v cluster.  Interestingly enough.  The physical VHDX file now sits on (Lets say)  Hyper-v server # 1, but it won't run on hyper-v server # 1, it will only run on Hyper-V server # 2.  Live or quick migrations fail.  

    ------------------------

    Cluster resource 'Virtual Machine Configuration XXX' of type 'Virtual Machine Configuration' in clustered role 'XXX' failed. The error code was '0x5' ('Access is denied.').

    Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it.  Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.

    -------------------------

    User 'NT AUTHORITY\SYSTEM' failed to create external configuration store at '\\hvhost1\vm-xxx\xxx': General access denied error. (0x80070005)

    -------------------------

    The absolute path '\\hvhost1\vm-xxx\xxx\xxx.vhdx' is valid for the '' Hard Disk Image pool, but references a file that does not exist.

    Well, the file definitely does exist in that path.

    -------------------

    The latest error message I am getting is as follows : 

    ----------------------

    Live migration of 'Virtual Machine XXX' failed.

    Virtual machine migration operation for 'XXX' failed at migration destination 'HVHOST1'. (Virtual machine ID E6350A40-42A6-4E6A-BBE6-483190A4CE59)

    User 'NT AUTHORITY\SYSTEM' failed to create external configuration store at '\\hvhost1\vm-xxx\xxx': General access denied error. (0x80070005)

    Anyone help would be greatly appreciated.

    K


    • Edited by KyRou Friday, June 08, 2018 3:09 PM
    Friday, June 08, 2018 3:07 PM

Answers

  • Good idea or not, no Hyper-V host can run a virtual machine from a UNC that resolves to itself (that's loopback). It just can't be done (Storage Spaces Direct configurations aside). The computer named HYPERVBOX will never be able to run any virtual machines from \\HYPERVBOX. It's hard-coded that way and it is smart enough to see through all common trickery.

    Do not start duplicating domain controllers. That's a total waste of your time. With practice, you can kick a new domain controller into an existing domain in five minutes. With scripting, you can do it even faster. Without practice or scripting, you can still do it more quickly and with fewer headaches than duplicating.

    I would:

    1. Use Storage Migration from a non-owner node to move that DC VM back out into shared storage.
    2. Live Migrate that VM to the node you want to own it forever.
    3. Remove the VM from the cluster.
    4. Use Storage Migration to move it to local storage using a local path (D: or whatever).
    5. On another node, create a new VM on local storage using a local path. Make sure to set it to auto start and to shut down (not save) on host shutdown.
    6. Install Windows Server on the new VM and promote it to be a domain controller in your existing domain.

    When that's done, you will have two virtualized domain controllers, neither dependent on the cluster. ADDS will have greater resiliency than it would if you had a single HA domain controller.

    Demote the physical DC whenever, but do a proper demotion and get rid of it entirely.


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    • Marked as answer by KyRou Monday, June 11, 2018 12:29 PM
    Friday, June 08, 2018 5:50 PM

All replies

  • First thing, don't cluster DC VMs. It gives you exactly nothing. Move the VM to truly local storage (ex D:\VMs) and remove it from the cluster. Create other local DCs on other nodes' local storage for ADDS redundancy.

    Second thing, if you're going to cluster DC VMs against recommendations, never put them on SMB storage. You cannot universally guarantee that a Hyper-V host can authenticate to an SMB resource when the VM in question is a domain controller.

    Third, no cluster resource should ever depend on the success of any member node. In other words, don't run cluster resources from local storage.

    Your error message is because loopback configurations are not supported for anything except Storage Spaces Direct. Every node in the cluster will be able to reach that particular share except the node the node that hosts it.


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Friday, June 08, 2018 4:49 PM
  • So, on the one Hyper-v host where the actual vhdx file lives, I am using UNC path to the file with a share. (ex. \\hvhost1\vm_DC1\dc1.vhdx) That is not a good idea then?  Better to use the direct path and then duplicate on another host, having two DC's running, and not have them as a part of the failover cluster.  I think that is what you are saying here.

    My secondary DC lives on old bare metal now, so, could I just convert that to a VM and put in on another HVHost?  Incidentally, the entire point of this project was to retire the old server with the physical DC on it running old Windows Server.  We are a small enough shop that I guess going with one DC would be enough.

    -----------

    Your error message is because loopback configurations are not supported for anything except Storage Spaces Direct. Every node in the cluster will be able to reach that particular share except the node the node that hosts it.

    I am not quite sure where that comes into play here, but I will take your word for it.
    ---------------------------------------


    The key to all of this is to don't attempt to have the virtual DC a part of the Fail over cluster.  It can sit on the hosts, but run independently of the cluster.


    Friday, June 08, 2018 5:36 PM
  • Good idea or not, no Hyper-V host can run a virtual machine from a UNC that resolves to itself (that's loopback). It just can't be done (Storage Spaces Direct configurations aside). The computer named HYPERVBOX will never be able to run any virtual machines from \\HYPERVBOX. It's hard-coded that way and it is smart enough to see through all common trickery.

    Do not start duplicating domain controllers. That's a total waste of your time. With practice, you can kick a new domain controller into an existing domain in five minutes. With scripting, you can do it even faster. Without practice or scripting, you can still do it more quickly and with fewer headaches than duplicating.

    I would:

    1. Use Storage Migration from a non-owner node to move that DC VM back out into shared storage.
    2. Live Migrate that VM to the node you want to own it forever.
    3. Remove the VM from the cluster.
    4. Use Storage Migration to move it to local storage using a local path (D: or whatever).
    5. On another node, create a new VM on local storage using a local path. Make sure to set it to auto start and to shut down (not save) on host shutdown.
    6. Install Windows Server on the new VM and promote it to be a domain controller in your existing domain.

    When that's done, you will have two virtualized domain controllers, neither dependent on the cluster. ADDS will have greater resiliency than it would if you had a single HA domain controller.

    Demote the physical DC whenever, but do a proper demotion and get rid of it entirely.


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    • Marked as answer by KyRou Monday, June 11, 2018 12:29 PM
    Friday, June 08, 2018 5:50 PM
  • Thank Eric,

    I appreciate all the guidance on this.  I h ave completed the steps as you have outlined and everything is running fine.

    K

    Monday, June 11, 2018 12:30 PM
  • Eric,

    Just a point of clarification.  Why is clustering the DC VM's not recommended and gives me exactly "nothing" as you say?

    And further to that.  If I don't have another license for 2012R2 to run a Second VM of the DC on the other HVHost, would a powershell script, that exports the DC Vm daily to the second HVHost local to that HVhost so that in the event the main host goes down, the DC could be brought up on the second host.  (I understand that the DC would be out by, when the last Export was completed).  Any dangers to that? bad idea? Better suggestion?

    Thanks,

    K




    • Edited by KyRou Tuesday, June 12, 2018 3:21 PM
    Tuesday, June 12, 2018 11:39 AM
  • If you don't have a license to run the second VM, then you didn't have a license to Live Migrate the first VM, either. The licensing requirements are identical for 1 VM to Live Migrate across X hosts as it is for X non-migrating VMs to live on X hosts. That's one of the reasons that hosting domain services in a clustered VM has no value; you can't save on licensing costs.

    Also better because with 2 DC VMs on two separate hosts, your directory's availability will not go sour just because you had to reboot your one and only domain controller. If one domain controller permanently dies, it doesn't matter. Etc. ADDS has better native resiliency technologies than anything that Hyper-V can offer.

    Daily exports are possible but not a mess that I would wade into. Get a proper backup solution. If money is tight, almost every backup application vendor allows you to back up at least one VM for free. Even Windows Backup could handle this. That said, if I had to prioritize a list of datacenter services that I would be willing to spend at least a little bit of money on, backup always comes in first place.


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Wednesday, June 13, 2018 3:42 AM
  • Backups I have covered.  I use Veeam Backup and replication for Hyper-V.  I can actually launch a VM directly from the backup. 

    What are the dangers of having to restore a DC in the event of a failure. (VHDX corruption, or any other reason).  If I had to restore the DC VM, would one or 2 days old (Or more), be a problem?

    Regards,

    K

    Wednesday, June 13, 2018 1:45 PM
  • Well, at that point, you're switching topics from virtual machines to domain services restore. The risks don't really change when you go with virtual, only the process. There is no particular problem with restoring a domain controller that's a few days old. ADDS recovery and restore is a large topic, though, so I'm not sure what level of information to give you.

    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Wednesday, June 13, 2018 2:35 PM
  • well, I would consider it still part of the entire process that I am working toward.  That being, a simpler Setup for my network of 30-40 users, such that, we are running mostly Virtual on 2 HvHosts.  I would like to get down to one DC, maybe two if I elect to have two running on the two HvHosts with locally stored files.  

    But, If I can't do two, then I am back to having to restore the Virtual DC from my backup if a problem occurs.  

    Maybe this is a question for my Backup software provider.  They should have some kind of best practices documented.

    I am just trying to document and close loop a system that I know will work.

    K

    Wednesday, June 13, 2018 6:16 PM
  • If there's only one DC, then a restore is a restore. No magic or layers to it. There will be problems with objects that were created, modified, or deleted between the backup point and the failure point. Those effects will always be situational and there's not much you can do but clean up the mess. The tighter your backup frequency, the less mess.

    If there's two or more DCs, then I personally only restore in response to a catastrophe or to recover an accidentally deleted object when the AD recycle bin isn't enabled. It's easy enough to blow away and rebuild a failed DC in a multi-DC domain that going through a restore is rarely worth the effort. But, that's personal preference. Other people may feel differently. If you don't need an authoritative restore, then a restore is still just a restore.


    Eric Siron
    Altaro Hyper-V Blog
    I am an independent contributor, not an Altaro employee. I accept all responsibility for the content of my posts. You accept all responsibility for any actions that you take based on the content of my posts.

    Wednesday, June 13, 2018 6:29 PM