none
[2016 Hyper-V] unable to snapshot / backup guest cluster - Add-VMGroupMembers not working

    Question

  • We have a 2016 datacenter Hyper-V cluster. On them we have a SQL 2014 guest-cluster with Server 2016 VM's. For them to be snapshotted you would have to create VMGroups. I have lots of issues with that though. I can create a group, but not add members to it. Also I cannot get these groups removed anymore. I started out with a 2012R2 guest cluster, but when we moved from Hyper-V 2012R2 to 2016, the 'old' shared VHDX cannot be backed up anymore. So we had to move to VHDS. After that we still couldn't do it. So it turned out we had to move the guest cluster OS to 2016 as well. So we did that. And now it's still not working.


    PS C:\Users\<my-account>> new-vmgroup -name test -GroupType VMCollectionType


    Name           : test
    InstanceId     : 1c7c4301-e5ba-417a-b204-1ce5a47a7d21
    GroupType      : VMCollectionType
    VMMembers      : {}
    VMGroupMembers :
    CimSession     : CimSession: .
    ComputerName   : NLEVHYV04
    IsDeleted      : False


    PS C:\Users\<my-account>> Add-VMGroupMember -VMGroup (Get-VMGroup test) -VM (get-vm testVM1)
    PS C:\Users\<my-account>> Get-VMGroup test


    Name           : test
    InstanceId     : 1c7c4301-e5ba-417a-b204-1ce5a47a7d21
    GroupType      : VMCollectionType
    VMMembers      : {}
    VMGroupMembers :
    CimSession     : CimSession: .
    ComputerName   : NLEVHYV04
    IsDeleted      : False



    PS C:\Users\<my-account>> Add-VMGroupMember -VMGroup (Get-VMGroup test) -VM (get-vm testVM2)
    PS C:\Users\<my-account>> Get-VMGroup test


    Name           : test
    InstanceId     : 1c7c4301-e5ba-417a-b204-1ce5a47a7d21
    GroupType      : VMCollectionType
    VMMembers      : {}
    VMGroupMembers :
    CimSession     : CimSession: .
    ComputerName   : NLEVHYV04
    IsDeleted      : False



    PS C:\Users\<my-account>> Remove-VMGroup test

    Confirm
    Are you sure you want to remove group "test"?
    [Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y
    Remove-VMGroup :
    At line:1 char:1
    + Remove-VMGroup test
    + ~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [Remove-VMGroup], VirtualizationException
        + FullyQualifiedErrorId : OperationFailed,Microsoft.HyperV.PowerShell.Commands.RemoveVMGroup

    PS C:\Users\<my-account>>





    So when adding VM's as per the sparse documentation I can find on this, I get no error at all. However the VM does not get added. testVM1 is actually a clusternode with shared VHDS disk (first node), testVM2 is just a regular VM.

    I can't find any errors or even anything at all related to these groups in the eventlogs. I assume more people must have ran into this?


    Monday, January 30, 2017 11:20 AM

All replies

  • Hi Robert,

    It is difficult to troubleshoot the issue without related errors and events.

    And it seems I could not find much information about this phenomenon.

    >>I started out with a 2012R2 guest cluster, but when we moved from Hyper-V 2012R2 to 2016,

    Do you mean you upgraded the cluster to 2016? Not sure if the upgrade caused the issue, if possible, build new cluster with 2016 nodes to see if the same issue exists.

    Best Regards,

    Leo


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Tuesday, January 31, 2017 6:57 AM
    Moderator
  • Hi, yes we did a (supported!) rolling upgrade of 2012R2 Hyper-V cluster to 2016. We freshly installed each host one by one to 2016 and joined the cluster. After all were moved, the cluster level was upped to 2016, and then all VM's were upgraded to configuration level v8.

    I have tested the same on my w10 desktop with Hyper-V installed, and it works as expected there. But not on our cluster were it actually matters. I'll try to look into some logs again, however so far the only errors I see are the ones I get from the powershell commands already pasted above.

    Tuesday, January 31, 2017 9:57 AM
  • When I delete an erroneous VMGroup I get the following two events:



    Log Name: Microsoft-Windows-Hyper-V-VMMS/Admin

    Source: Hyper-V-VMMS

    EventID: 16000



    The description for Event ID 16000 from source Microsoft-Windows-Hyper-V-VMMS cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event: 

    %%2147946707
    0x800710D3

    The locale specific resource for the desired message is not present




    Immediately after that:



    Log Name: Microsoft-Windows-Hyper-V-VMMS/Admin

    Source: Hyper-V-VMMS

    EventID: 16010



    The operation failed.



    When adding hosts to a group, which gives no errors at all but also doesn't really do anything, doesn't generate any logs (I can find) at all.

    It happens on both hosts (it's just a 2 node cluster). Both nodes have been rebooted and are complete up to date as of januari 31th 2017.
    Tuesday, January 31, 2017 12:23 PM
  • Hi Robert,

    >>yes we did a (supported!) rolling upgrade of 2012R2 Hyper-V cluster to 2016.

    I have seen posts of different strange issues after upgrading like this.

    I'm not sure if the upgrade is really causing issues.

    I would post the information if I got any updates on later official documents.

    Best Regards,

    Leo


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, February 01, 2017 6:27 AM
    Moderator
  • Just to keep you informed; I've recently built up a new Hyper-V host cluster from scratch. The issues of not being able to add or remove VM's to collections has gone. So that's a win. Still, the functionality is sub-par. I can't fail over guest-cluster-vm's because of collection errors. I can't backup them because of collection errors. Documentation is seriously lacking as always with MS since Windows 95. (Remember those huge, acutally usefull ordners with documentation you got with MS-DOS 3.3? Those were the days!).

    I'll dive into it a bit more and report when I have news.

    Tuesday, February 21, 2017 10:36 AM
  • Just to update - I've raised a ticket with MS for this yesterday as this functionality is just broken beyond believe. There's another thread about this by the way, https://social.technet.microsoft.com/Forums/en-US/0d6cddde-3ff3-4a50-a627-6ad7bf7667fd/cannot-create-checkpoint-when-shared-vhdset-vhds-is-used-by-vm-not-part-of-a-checkpoint?forum=winserverClustering
    Wednesday, April 05, 2017 6:56 AM
  • Yep, just ran into this issue today as well. Compound issue for us because Veeam doesn't like the .VHDS files either, and I suspect at this stage it's trying to, but cannot, create checkpoints. At least that's Veeam it complains about.

    5/4/2017 10:14:20 AM :: Failed to create VM recovery checkpoint (mode: Crash consistent) Details: Job failed ('Failed to create checkpoint on collection 'Hyper-V Collection' (D1B15E6D-D97D-4C20-978D-3D8A1A6DB31C).

    I'm not sure if anyone has noticed the same, but for me there were a whole bunch of pre-existing VM Groups with the name 'Hyper-V collection', type 'VMCollectionType' which I presume were created when I firstly created the shared VHDS on the first VM, then attached the VHDS on the second VM. There's a total of 12 shared VHDS'es across 2-VM clusters and a total of 24 of these 'Hyper-V Collections'.

    Thursday, May 04, 2017 2:53 AM
  • Yes, Veeam creates VMGroups with name 'Hyper-V collection' but because of the errors it runs into it doesn't clean them out. I've had a very disturbing call with MS yesterday again about this issue. I've send dozens of logs, traces and whatever they asked me. But they can't find the issue and for MS that means 'there is no bug'. Which would also mean I have to pay the ticket. Now I don't mind money but I DO mind principles. So I exploded right in my telephone and thaught the poor MS support guy some new words.

    Later he claimed it would help to know what EXACT KB breaks it. Well I think that's something MS can find out rather easy, but they told me it's very hard to get a LAB setup to test for MS support. I just can't believe that. With Azure they have one of the biggest 'your lab is setup in 2 minutes' environments in the world, and it would be extremely good for customer satisfaction if they use it themselves as well. However they can't. I already had a testserver for this specific issue as MS refused to reproduce it themselves, so I reverted that to a 'no update' image and installed KB's up to the one that made it break.

    It turns out it's KB3200970, something I already reported in the case as Veeam told me it was KB3200970. I've not heard from them since that, but as said that was yesterday so they need some time.

    Please, everyone with a support contact, raise a ticket with MS for this and relate it to ticket 117040415552767. It'll only speed things up. We are too small a company for MS to take us serious, and that's almost literally what the support guy told me as well. So we need multiple tickets to get this solved. I'm utterly disappointed with MS again.

    Thursday, May 04, 2017 7:40 AM
  • Did anyone get this resolved?  I have the same issue!!!! 
    Friday, August 04, 2017 2:16 AM
  • I have just had word (minutes ago) from our Microsoft engineer that a public hotfix will be released in the next 'several days' that will hopefully address this issue. 

    As I find out more, I'll post.

    Stay tuned!

    Wednesday, August 16, 2017 3:29 AM
  • I hope so! 
    Wednesday, August 16, 2017 10:24 PM
  • Just for completeness as there are multiple threads about this issue; the mentioned fix does NOT fix the backup issue, and in fact generates huge troubles as with it installed, backing up a guest clusters locks its snapshot files, making the snapshots unable to commit, which keeps the VM's running from those snapshots. In addition, when trying to move such a VM the complete VM will finally fail and go down.

    Hyper-V Server 2016 seems to be extremely immature compared to 2012R2 which worked fine (in the end at least).

    https://social.technet.microsoft.com/Forums/windowsserver/en-US/0d6cddde-3ff3-4a50-a627-6ad7bf7667fd/cannot-create-checkpoint-when-shared-vhdset-vhds-is-used-by-vm-not-part-of-a-checkpoint?forum=winserverClustering#0771cc8f-3b9d-490d-a4cf-f0e7732b4f26

    Monday, September 11, 2017 8:27 AM