none
Cannot create checkpoint when shared vhdset (.vhds) is used by VM - 'not part of a checkpoint collection' error

    Question

  • We are trying to deploy 'guest cluster' scenario over HyperV with shared disks set over SOFS. By design .vhds format should fully support backup feature.

    All machines (HyperV, guest, SOFS) are installed with Windows Server 2016 Datacenter. Two HyperV virtual machines are configured to use shared disk in .vhds format (located on SOFS cluster formed of two nodes). SOFS cluster has a share configured for applications and HyperV uses \\sofs_server\share_name\disk.vhds path to SOFS remote storage). Guest cluster is configured with 'File server' role and 'Failover clustering' feature to form a guest cluster. There are two disks configured on each of guest cluster nodes: 1 - private system disk in .vhdx format (OS) and 2 - shared .vhds disk on SOFS.

    While trying to make a checkpoint for guest machine, I get following error:

    Cannot take checkpoint for 'guest-cluster-node0' because one or more sharable VHDX are attached and this is not part of a checkpoint collection.

    Production checkpoints are enabled for VM + 'Create standard checkpoint if it's not possible to create a production checkpoint' option is set. All integration services (including backup) are enabled for VM.

    When I delete .vhds disk of shared drive from SCSI controller of VM, checkpoints are created normally (for private OS disk).

    It is not clear what is 'checkpoint collection' and how to add shared .vhds disk to this collection. Please advise.

    Thanks.

    Friday, December 09, 2016 9:27 AM

All replies

  • hi, i'm in your situation.

    i know that we must do the new VM Group to group together virtual machines with shared VHDX.

    New-VMGroup to create the virtual machine collection group

    Add-VMGroupMember to add a vm to the group

    Checkpoint-VM -VM (Get-Group).VMMembers to create a snapshot for all vms in the group

    i try this but doesn't work (and my backup software goes in to the same error)

    Maybe someone in M$ can help us...

    Matteo

    Saturday, December 10, 2016 11:42 PM
  • Little Bump

    Anyone got a sollution for this problem. Same issue here.

    Tuesday, January 31, 2017 1:14 PM
  • Same issue in this post:

    https://social.technet.microsoft.com/Forums/windowsserver/en-US/57316c1e-6018-4ed9-aa3e-7ffbe6ad9cb2/applicationconsistent-checkpoints-for-vhd-set?forum=winserverhyperv

    And we also have this problem. Hope to find a solution soon.

    Monday, February 13, 2017 8:30 AM
  • Hi there,

    We've done request to Microsoft public help line and received the clarification for this topic:

    You are correct that an additional tool is needed to generate RSVD v2 traffic, however, it is not a 3<sup>rd</sup> party tool. When the Hyper-V role is installed on Server 2016, additional PowerShell scripts are included for working with RSVD v2 snapshots. I’ve included links below with more details.
     
    Creating a new VM group:
    New-VMGroup
     
    Adding machines to the group:
    Add-VMGroupMember
     
    Taking a snapshot collection:
    CreateSnapshot method of the Msvm_CollectionSnapshotService class
     
    Applying a snapshot:
    ApplySnapshot method of the Msvm_CollectionSnapshotService class
     
    Deleting a snapshot:
    DestroySnapshot method of the Msvm_CollectionSnapshotService class
    Monday, February 13, 2017 10:50 AM
  • Hi again,

    We have spent a lot of time for investigation of this topic and finally we have a solution. Following PowerShell commands demonstate how to get the basic flow of RSVDv2 snapshots.

    However this way of doing things (without GUI support) is not very handy. We are waiting for better feature coverage by Microsoft...

    Create VMGRoup:
    New-VMGroup -Name "RSVD" -GroupType VMCollectionType
     
    Add VM to VMGRoup:
    $VM1=Get-VM -Name "RSVDVM1"
    Add-VMGroupMember -Name "RSVD" -VM $VM1
     
    Create checkpoint:
    $collSnapSvc = Get-WmiObject -Namespace Root\Virtualization\v2 Msvm_CollectionSnapshotService
    $VmColl = Get-WmiObject -Namespace Root\Virtualization\v2 Msvm_VirtualSystemCollection
    $collSnapSvc.CreateSnapshot($vmColl, $null, 32768) 
     
    Apply checkpoint (when VM is off):
    $collSnapSvc = Get-WmiObject -Namespace Root\Virtualization\v2 Msvm_CollectionSnapshotService
    $VmColl = Get-WmiObject -Namespace Root\Virtualization\v2 Msvm_SnapshotCollection
    $collSnapSvc.ApplySnapshot($VmColl) 
     
    Delete checkpoint:
    $collSnapSvc = Get-WmiObject -Namespace Root\Virtualization\v2 Msvm_CollectionSnapshotService
    $VmColl = Get-WmiObject -Namespace Root\Virtualization\v2 Msvm_SnapshotCollection|? "CollectionID" -eq "32191801-1857-490D-B3B3-2B5F92F05487"
    $collSnapSvc.DestroySnapshot($VmColl) 


    Monday, February 13, 2017 11:00 AM
  • Hi Volodymyr,

    Just a little question for clarification: your guest cluster virtual machines are deployed on a standalone Hyper-V server, not Hyper-V cluster? I mean, they are not cluster roles?
    Monday, February 13, 2017 12:17 PM
  • Hi Volodymyr,

    thanks for sharing your experience, but I didn't fully understand. Our problem is: We like to make host-based-Backups from shared disk, performed by our backup solution Veeam. Host-based-Backups are only possible with VHDSet, so I build a test environment with a Cluster on a Windows 2016 host and a Windows 2016 VM which has a shared disk (vhdset). When trying to take a Backup with Veeam we got an error, that no checkpoint can be taken. So we tried to take a checkpoint manually (via Hyper-V) and we got this error "Cannot take checkpoint for VM001 because one or more shareable VHDX are attached and this is not part of a checkpoint collection" also.

    A performed a checkpoint by your commands above, which I didn't fully understand. There was no error but I wasn't able to see the checkpoint, neither in the Hyper-V Manager in the "Checkpoints" section, nor on the folder "checkpoints" where I thought it would store the checkpoint.

    However, the primary goal is to perform the host-based-backup and I don't understand how this commands should help to reach this. It's wired. There are plenty of statements Windows2016 and VHDset allows host-based-backups and our backup solution is able to perform host-based-backups, but it didn't work. Neither from the backup software, nor as a simple checkpoints.

    Monday, February 13, 2017 1:52 PM
  • Hi Volodymyr,

    Same problem here. Logged a case at microsoft and the official responds from microsoft support and i quote. "The only supported scenario in your case is to completely uninstall Veeam and install Windows Backup solution." I don't think they fully understood the problem.

    I had a good laugh together with veaam support and we are working on a solution. It seems that kb3213986 and kb3211320 are the problem and does something that veeam does not like. Uninstalling these updates should get the backup going again.

    Next problem is that when I try and uninstall these updates my server reboots and i'm presented with a BSOD. So again new case with microsoft. I will keep you posted on futher developments.

    Tuesday, February 14, 2017 2:39 PM
  • Hi Volodymyr,

    After a lot of trying to uninstall those updates, I just gave up and reinstalled those host without any windows updates on it. Veeam started to work and the backup is running without problems since. I contacted veeam support and they expect an update of veeam end of this month. The engineer I spoke was not completly sure if this issue was going to be resolved but as for now I disabled the updates and all is running fine.

    • Proposed as answer by ITCbe Tuesday, March 14, 2017 8:22 AM
    Tuesday, March 14, 2017 8:20 AM
  • Hi, we also have these issues. Uninstalling all patches is not a serious option for anyone I believe. We have logged a call with MS yesterday. I still have to get my initial response (it's already past its due time but let's not be pessimistic). I also have a ticket with Veeam, although we now initially concentrate on Hyper-V being able to create snapshots itself, as that's what Veeam uses anyway. Still, Veeam support will work together with MS on the ticket I created (now THAT is what I call support - eat your heart out MS!)

    I'll keep you informed of any updates.

    [edit]

    In the meanwhile I installed a new cluster - with only one node for now - without any MS updates. I can confirm Veeam works there, and also Volodymyr's PowerShell commands work to create snapshots. After updating with the lastest (as of yet) cumulative KB4016635 it stops working.

    Wednesday, April 05, 2017 6:55 AM
  • Robert, in your test with a new cluster without patches, was you also able to perform checkpoint out of the hyper-v console?
    Friday, April 07, 2017 8:30 AM
  • No, that doesn't work even without patches installed. It complains about the VM's not being in a checkpoint-collection, also when I manually add them to one. But as said, the powershell commands work.

    I did a very extensive report in my MS ticket, they tend to like that. Now they actually are willing to look into it. Initially they wouldn't, as the only support our small company is able to get is the break-fix 'professional' support which is as useful as a solar-powered torch. But with this test machine I have my break-fix scenario, and I made images of it so I can quickly move between pathed and unpatched machine.

    Will keep you informed.

    Friday, April 07, 2017 9:02 AM
  • Thank you very much for the answer and keeping up us with information.

    Yet, I still have two questions.

    1.) Where are the patches that cause the problem installed? a.) On the Hyper-V host, b.) The Hyper-V virtual machine or c.) the machine where Veeam is installed?

    2.) Performing the first to steps of the powershell commands above (New-VMGroup and Add-VMGroupMember) is even necessary when performing a backup through Veeam, or only when I intend to perform to take a checkpoint through powershell ($collSnapSvc.CreateSnapshot)? When this two steps are also necessary for veeam backups, does Veeam automatically recognize the manually created VMGroups?

    Kind regards

    Michael

    Friday, April 07, 2017 1:32 PM
  • Micael,

    this is about patching the hosts. What's inside the VMs does not matter. The powershell commands are not necessary with Veeam. Veeam detects on what nodes a VHDS is shared, and creates a VMGroup named 'Hyper-V collection' automatically. After backup it's cleaned up. But manually creating a VMGroup is indeed needed if you want to manually create a snapshot of machines with shared VHDS. I've not tested if Veeam detects an eventual manually created VMGroup, but I'm pretty sure it does. The people behind Veeam are pretty smart.


    Friday, April 07, 2017 6:52 PM
  • Thanks again. First, it also creates the VMGroup "Hyper-V Collection" if there are already selfcreated VMGroups, this only as information.

    Yeah, I had the kb3213986 (mentioned above) installed and I was in high hope this cause the failure. I uninstalled it, which caused the whole VH didn't start anymore (bluescreen). So I installed everything again (VH, virtual SAN, VM) and didn't updates this time.

    However, try to make a veeam backup fails with the error: 

    10.04.2017 14:50:53 :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed ('Failed to create checkpoint on collection 'Hyper-V Collection' (7DCA828B-7D36-47CA-8C16-E0E1AD7717CC).

    The operation failed.

    The operation failed.'). Error code: '32768'.
    Failed to create collection recovery checkpoint.  

    I don't now if this is the same error like before with the Patch kb3213986 installed or not, since I deleted the Veeam backup job meanwhile. 

    Thanks for any further help!

    Monday, April 10, 2017 1:28 PM
  • Might depend on which media you used to install. I used SW_DVD9_Win_Svr_STD_Core_and_DataCtr_Core_2016_64Bit_English_-2_MLF_X21-22843.ISO which is the original RTM ISO. However, I also have SW_DVD9_Win_Svr_STD_Core_and_DataCtr_Core_2016_64Bit_English_-3_MLF_X21-30350.ISO which is one from end of january 2017. That one has some cumulative patches already installed and hence it doesn't work 'out the box' already.

    What does Get-Hotfix report? My unpatched host only shows KB3192137. That came preinstalled, but with that everything works fine.

    I've done extensive reporting to MS and surprisingly I've not heard from them anymore since.




    Monday, April 10, 2017 1:42 PM
  • I'm almost sure I've used the older ISO. The only hotfix installed is the same as on your unpatched host (KB3192137).

    I've tested very much today and I only once was able to create a checkpoint through the powershell commands. Which is the returnvalue you get when a checkpoint has successfully taken or appied? It seems on success it will return code 4096 for both events (create and apply), can you confirm this? I only was able to create once a checkpoint through powershell (with code 4096), but trying to apply the checkpoint also gave me code 4096 and nothing happened.

    The most time I get an return value 32773 which seems to be an error, even I didn't know what the return code exactly means.

    Monday, April 10, 2017 2:45 PM
  • For the powershell approach: Ok, It didn't work to create checkpoints if you've got multiple vmgroups. If you have, either you've to delete to reduce to one, or you've to specify the right group...

    e.g. $collSnapSvc.CreateSnapshot($vmColl[4], $null, 32768) --> Index [4]

    Now I've a checkpoint (code 4096), but applying it get's a code 4096 also, but nothing happens.

    Still this are only tests. The way it have to work is through Veeam backup and there I've no clue why I get the error.

    Monday, April 10, 2017 2:59 PM
  • Yes 4096 = ok. But for VHDS backup to work there is a few other things

    - it must be a configuration version 8. (get-VM <vmname>|fl Name,Version)
    - Cluster must be running version 8 as well. Get-Cluster <clustername>|fl name,ClusterUpgradeVersion
    - You must set the ConfigStoreRootPath https://www.veeam.com/kb2194 although Veeam tells you so when you add a 2016 cluster to it's configuration.
    - So far I found it only works when both (or all) vhds VM's are on the same host. I didn't try with Veeam, but the powershells even fail when they are on different hosts (on my setup that is).

    I am running Veeam 9.5.0.823 by the way, and with that it worked.

    [edit]

    I've been testing a lot as well today, and I can make snapshots with powershell as much as I want, and revert them. I don't know why, but I never tested to apply one. Will try so tomorrow. If creating snapshots fails, do you get specific errors in the Hyper-V-VMMS or system logs?


    Monday, April 10, 2017 3:09 PM
  • Thanks. Meanwhile I've noticed that the failue in veeam only happens when I've activated "application-aware". Have you activated application awareness in your tests?
    Tuesday, April 11, 2017 8:10 AM
  • No, as my test VM's for this are just 2 empty VM's with an VHDX 'OS' disk (but empty) and one shared VHDS between them. So I did not enable application awareness as in-guest VSS is not available anyway.

    Tuesday, April 11, 2017 8:27 AM
  • Robert, could you please test with application awareness activated?
    Tuesday, April 11, 2017 8:30 AM
  • As said I don't even have an OS running in these VMs, so no, that wouldn't work.



    [edit]

    well for my MS case I don't have anything in those VM's. But as I'm curious, I'll deploy two 2016 nodes and define a cluster.

    Note that I forgot another requirement; the VM's have to be 2016 as well.


    Tuesday, April 11, 2017 8:31 AM
  • Sorry, didn't read your previous post right, that you haven't installed a OS.
    Tuesday, April 11, 2017 8:41 AM
  • Just tested with a guest cluster of 2 2016 VM's with shared disks and a CSV inside the guest-cluster. Application aware backup fails. I have to disable that to get Veeam running.

    Just got a call from the guy that replaces the one that handles my ticket, as he's on holiday leave. This one is MUCH more helpfull. I've had a good talk and he is really willing to help out. I'm collection new logs now and will keep you posted.

    Tuesday, April 11, 2017 11:07 AM
  • Hi everybody,

    I opened a Microsoft premier service request 2 or 3 weeks ago and it's a known bug, the support has reproduced the bug with the vhd-set and checkpoint. All the wmi namespace are ok but not implemented by the checkpoint process.. A patch will be released to resolve the bug, but they don't know when, maybe in the 2 next months.. #SoSick

    Tuesday, April 11, 2017 2:35 PM
  • Hi There

    We got the same Problem. Can you give me your Case-Nr. of your Microsoft-Case.

    Then I will open myself a case at Microsoft and refer to your case in order to get more pressure for this issue.

    Wednesday, April 12, 2017 10:53 AM
  • My MS case is 117040415552767. A senior support engineer that also works with Veeam is working and monitoring the case(s).
    Wednesday, April 12, 2017 11:21 AM
  • Even it is 1-2 weeks ago, thank you very much for installing an OS and testing it with and without application awareness.

    Now I'm not sure what you Robert and the others here are exactly looking for. Is it just that checkpoints aren't possible when certain patches are installed on the host, or is it that (even when patches are not installed) application aware doesn't work? Since you didn't have realized application awareness didn't work before your last test, I think you're looking for a solution it works with patches installed. But didn't you need application awareness too? What's about the others here? For me, I've to store SQL databases to the shared disk, so application awareness is needed.

    I'am afraid we are struggling with two issues and even we will have a solution for the patch issue it wouldn't help for the application awarness issue too.

    Saturday, April 22, 2017 8:05 AM
  • Initially I'd be glad to see it working without application awareness. But ofcourse, people use VHDS with guest clusters, meaning that VSS working is always a requirement in the end. In addition, in a working setup (so without any patches on the host) both VM's have to be on the same host in order for snapshots to be able to be created.

    So there's more than one thing broken. For a lot of serious Hyper-V implementations that means moving to 2016 is not a serious option yet.

    Saturday, April 22, 2017 10:41 AM
  • We also logged a case with MS; seeing exactly the same issue. Will post if we learn anything new.
    Thursday, May 04, 2017 7:35 AM
  • Our MS tech confirms that they can reproduce the issue in their labs (all but confirms a bug), and that a further update isn't too far away.
    Tuesday, May 09, 2017 5:20 AM
  • RustyJusty, if not done so already please refer your MS ticket to our 117040415552767. It's escalated now to a very dedicated person, one that's actually willing to help.
    Tuesday, May 09, 2017 6:24 AM
  • Yeah, we've done that. The guy we have (Eric) is also helping but it's pending a solution from Engineering now, I guess. Our case ID is 117050415687873.

    I assume it's still an issue for you, though?

    Our Veeam case is basically on hold until the VM checkpoint issue is resolved by Microsoft.


    • Edited by RustyJusty Tuesday, May 09, 2017 7:46 AM
    Tuesday, May 09, 2017 7:38 AM
  • Yes, still an issue. When I have a fix or any news, be sure I'll report that here :)
    Tuesday, May 09, 2017 8:01 AM
  • We are also waiting for a fix from this. Posting to remain alerted and an update from Microsoft on potential timescales would be appreciated!
    Tuesday, May 16, 2017 10:33 PM
  • We are feeling the pain from this as well. Does anyone have the internal bug number for MS so that I can include that in my case with them?  I'll add the case numbers referenced in this thread to ours as well once I get it opened.
    Monday, May 22, 2017 11:59 AM
  • Same issue with Veeam and guest cluster as well. Hope this gets sorted out soon. keep us posted guys.

    Please remember to mark the replies as answers if they help.

    Tuesday, May 30, 2017 7:06 AM
  • Same issue with Veeam and guest cluster here :-(
    Friday, June 02, 2017 11:03 AM
  • Minor update - our Microsoft case manager tech has all but confirmed it is indeed a bug, and has mentioned that the issue is fixed in Redstone 3 (currently the latest Win 10 version is RS1 (build 1607). He also mentioned that there's likely not going to be a RS2 (1703) release for the Server 2016 branch of Windows 10.

    Apparently they're internally discussing the development of a hotfix for WS 2016 RS1 (1607), and we're awaiting further advice on this.

    Tuesday, June 06, 2017 1:24 AM
  • Does anyone have the internal bug number for MS
    I asked for the internal bug ID but they're not allowed to release these.
    Tuesday, June 06, 2017 1:25 AM
  • I'm curious, I'm expecting a pingback from my MS representative one of these days as well. I think he's pretty close to a solution as well. Unfortunately there are multiple people working on the same thing individually it seems, maybe that unavaoidable in such a big company. Will keep you updated as well.
    Tuesday, June 06, 2017 9:43 AM
  • I'm expecting an update from my MS representative as well. I will keep you posted.

    • Edited by Robert Gijsen Tuesday, June 06, 2017 9:48 AM
    • Proposed as answer by buravtsov Tuesday, June 06, 2017 10:52 AM
    Tuesday, June 06, 2017 9:45 AM
  • I've had a call from my representative today, it's been identified as a bug as stated above and a hotfix is expected this month. Internal bug ID is not for public exposure.

    So relatively soon we will be able to backup our virtual clusters again!

    Wednesday, June 07, 2017 12:31 PM
  • Same problem here as all the other posters, disappointing that it is now June, this product was released in September 2016 and that this feature has never really appeared to work according to this thread.

    Spent some time setting up a VHD set guest cluster, setting up as a File Server role setting up all the Hyper-V replica cluster parts then find out I cannot back it up or use Hyper-V replica for DR once I'd hit the exact same error messages as shown above. This feels like a BETA product not a production ready system !

    Monday, June 12, 2017 12:47 PM
  • Just wondering if anyone has heard anything back yet.
    Tuesday, June 13, 2017 5:22 AM
  • Just wondering if anyone has heard anything back yet.
    No, we'll have to wait until MS releases the fix. Be sure it's posted here when they do.
    Thursday, June 15, 2017 12:13 PM
  • My MS rep has just contacted us today, and mentioned a public hotfix will be released about August time. We have been provided a private hotfix today which we are yet to test and validate, but we have been asked to not share the private hotfix so won't do so, unfortunately.

    Still, sounds like some good progress is being made! Will keep you updated as I know more.

    Cheers.


    Thursday, June 22, 2017 7:15 AM
  • Same here, the case has been closed as a bug that should be fixed soon. will update here once there is progress.

    Please remember to mark the replies as answers if they help.

    Monday, July 03, 2017 2:00 PM
  • i am also facing the same issue. No help from Veeam Support also. They keep sending me link to this thread on my every follow up. 

    Please update if anyone finds a solution.


    • Edited by komikool Sunday, July 16, 2017 10:15 AM
    • Proposed as answer by Robert Gijsen Sunday, July 16, 2017 10:33 AM
    • Unproposed as answer by Robert Gijsen Sunday, July 16, 2017 10:34 AM
    Sunday, July 16, 2017 10:15 AM
  • That's obvious as it isn't a Veeam issue but a Microsoft issue. The solution is waiting for MS to release the patch. This is why I utterly, utterly hate this cumulative patching system. In the past when issues came with a fix, we could uninstall just that fix and keep secure for the rest. Now it's all or nothing, meaning a bug like this one means we've been unable to backup our guest clusters properly for 9 months now. KB3200970 is the patch that broke this which was released in November 2016.

    Sorry, but that's how bad cumulative patching can be.

    Sunday, July 16, 2017 10:46 AM
  • Any news? Yesterday was patch thuesday.
    Wednesday, August 09, 2017 9:43 AM
  • Any news? Yesterday was patch thuesday.
    So that makes it possible for you to test those updates. Their description doesn't show any sign of fixes for this. Feel free to test yourself and report.
    Wednesday, August 09, 2017 10:15 AM
  • Hi, i've tested these update without any success.

    The current update is not adressing the issue.

    Regards

    Friday, August 11, 2017 11:16 AM
  • komikool - We're hoping that the MS issue and upcoming hotfix will resolve the Veeam issue also, since Veeam backups in WS 2016 with shared VHDS-es seem to (try to) leverage Hyper-V snapshots.
    Thursday, August 17, 2017 1:40 AM
  • I reckon the hotfix for this will be released standalone to begin with (not necessarily in the Update Rollups, or Preview Rollup). DO they still release standalone hotfixes?

    I suppose it's possible it could be in the Week 3 Preview Rollup that should be released next week, however.

    Thursday, August 17, 2017 1:45 AM
  • The hotfix that supposedly fixes our issue isn't out yet. It's not in the August Monthly Rollup, it will be a hotfix (possibly standalone if they still do that, possibly in an upcoming Preview Rollup update)
    Thursday, August 17, 2017 1:47 AM
  • The hotfix has been relased in August 16, 2017—KB4034661 (OS Build 14393.1613):

    https://support.microsoft.com/en-us/help/4034661

    Let's hope it fixes the checkpoint issue in Hyper-V, and then the Veeam issue in one go!


    • Edited by RustyJusty Wednesday, August 23, 2017 4:14 AM
    • Proposed as answer by VJurescu Thursday, August 24, 2017 10:35 AM
    Wednesday, August 23, 2017 4:14 AM
  • The hotfix has been relased in August 16, 2017—KB4034661 (OS Build 14393.1613):

    https://support.microsoft.com/en-us/help/4034661

    Let's hope it fixes the checkpoint issue in Hyper-V, and then the Veeam issue in one go!


    Hi RustyJusty, where do you read the fix? are you sure this cumulative solve the problem?

    Thanks

    Matteo

    • Proposed as answer by VJurescu Thursday, August 24, 2017 10:35 AM
    • Unproposed as answer by VJurescu Thursday, August 24, 2017 10:35 AM
    Wednesday, August 23, 2017 9:07 AM
  • i try to install the 4034661 cumulative, but it doesn't solve the problem: i still can't take a snapshot/checkpoint of vm(s) with VHDS.
    Thursday, August 24, 2017 7:48 AM
  • It definitely solved mine, with Veeam 9.5 on windows 2016, for a 2016 file server cluster with vhd set, on hyper-v 2016.

    No error or warning, it just works. Before the error was "Failed to create checkpoint on collection.."

    At last.

    Thursday, August 24, 2017 9:51 AM
  • It definitely solved mine, with Veeam 9.5 on windows 2016, for a 2016 file server cluster with vhd set, on hyper-v 2016.

    No error or warning, it just works. Before the error was "Failed to create checkpoint on collection.."

    At last.

    Hi

    if you try to take a snapshot or a checkpoint manually, does it work?

    Thursday, August 24, 2017 11:31 AM
  • I've installed the mentioned update on our hosts (it's not on WSUS, which makes it fishy to me). I still can't take manual snapshots from guest-clusters with VHDS, nor can I do it manually through PowerShell. Veeam 9.5u1 also fails with the same error number:

    24-8-2017 22:30:33 :: Error: Job failed ('Failed to get the disk information.

    Failed to open attachment 'C:\ClusterStorage\Volume3\Hyper-V\<hostname>\<diskname>.vhds'. Error: 'The process cannot access the file because it is being used by another process.'.'). Error code: '32774'.
    Failed to get information about vhd disk 'C:\ClusterStorage\Volume3\Hyper-V\<hostname>\<diskname>.vhds'.
    Failed to get shared disks extents
    Failed to enumerate VM disks.
    Failed to get information about VM '11579e1f-7679-44cc-84c7-7ce66  

    So still no for us. Also, our MS representative for the ticket I've logged for this does not know about releasing the patch. As people above here mention it actually works for them it might be that that fix is actual in KB4034661. That would once again mean MS internal communication is as bad as it is.

    Thursday, August 24, 2017 9:02 PM
  • I'd been having the same issues with Hyper-V 2016 hostinga guest failover cluster (2012 R2) with the file server role using VHDS for the storage and attempting to backup it up using Veeam. I was getting errors about not being able to create snapshots.

    I have deployed 4034661 to our Hyper-V hosts this weekend gone and it has partially improved the issue for me. I have VSS disabled in the backup job for the guest cluster nodes (have not tried with it enabled just yet) and have been able to backup one of the guest cluster nodes with Veeam, however the second node it tried to process failed with the same errors as Robert Gijsen posted above. So that update definitely changed something around snapshots of VMs with VHDS attached.

    The only thing I have observed at present that may give any indication as to why one worked and the other didn't is that the one that worked is running on the Hyper-V host that is currently the "Current Host Server" for the Hyper-V cluster and the owner of the CSV.

    Sunday, August 27, 2017 10:25 PM
  • It seems I spoke to soon. When Veeam tried to backup the guest cluster VMs again tonight it failed on both with the same errors as shown in Robert's post.

    I've also discovered that the system drive and the shared VHDS drives for the guest cluster VMs all now have checkpoints, however Hyper-V management console does not report checkpoints for either of the guest cluster VMs and neither does Get-VMCheckpoint. However under settings the system drives do state that they can't be edited because of checkpoints and the shared VHDS each have 2 avhdx files with different IDs.

    Not sure how to merge the checkpoints if they're not being reported in Hyper-V or PowerShell.

    Monday, August 28, 2017 11:51 AM
  • Had the same problem for the quorum disk(in use).

    I've opened Computer Management and in Open files, there were two open connections, with servername$. One of them was idle for about the same time it was started. I've closed the connection and the backup worked.

    Tuesday, August 29, 2017 5:14 PM
  • You have to make sure, that under Computer Management|Shared folders|Session, there is only one session open with computer name, when backup is NOT running and the following Hyper-v merge operation is completed.

    If after the backup completes the other session(with a shorter connected time) is not closed automatically, it means the MERGE operation(you can monitor it in hyper-v manager, for the host on which the vm is running), did not succed. Maybe that is way there multiple avhdx.

    The merge operation always follows the veeam backup operation, because the backup takes a snapshot and all subsequent operations on the vm are written in the avdx, and after the backup finishes they are merged. Sounds confusing and it really is, especially when it fails and the errors are not helpful.

    It shouldn't be a problem, in the future, if you run the backup again, after you manually close the other session. I thought it failed because of the large data to be backed up, after a long time, because the following backups work flawlessly.

    Wednesday, August 30, 2017 8:15 AM
  • Hi I would like to test the Hotfix-KB4034661 on my File Server Cluster

    Where do you need to install the patch

    on each of the Files servers or Just the Cluster

    Wednesday, August 30, 2017 9:31 AM
  • Just the cluster.

    Another thing. All subsequent backup fail with error Error: 'The process cannot access the file because it is being used by another process.'.'). Error code: '32774'. as long as I keep the shared qorum disk(vhd set also).

    I have configured the file server cluster with a file share for quorum and remove the quorum virtual disk and no problem since then.

    Friday, September 01, 2017 9:04 AM
  • i still can't take a checkpoint to vm group with Shared VHD Set, not event with the 4034661 update installed.
    Friday, September 01, 2017 1:31 PM
  • Out of curiosity, those of you who have been successfully backing up the guest file server cluster after the update did you do a rolling cluster upgrade from a Hyper-V 2012 R2 cluster or did you create a new Hyper-V 2016 cluster and move the VMs over?

    I've setup a test environment using Hyper-V 2016 (with the above update) with both a 2012 R2 and a 2016 guest file server cluster and have been getting some strange behaviour with backing up using Veeam. I've found that if both guest cluster nodes are running on the same Hyper-V host then the backup job always completes successfully for both guest cluster nodes. However if they are on separate Hyper-V hosts then one of the guest nodes completes successfully and the other fails with the following error. (no it's not a typo).

    4/09/2017 4:14:15 PM :: Importing snapshot on offhost proxy LAB-HV02-16  
    4/09/2017 4:14:17 PM :: Error: Not supported for windows 2015.  
    Monday, September 04, 2017 6:23 AM
  • Out of curiosity, those of you who have been successfully backing up the guest file server cluster after the update did you do a rolling cluster upgrade from a Hyper-V 2012 R2 cluster or did you create a new Hyper-V 2016 cluster and move the VMs over?

    I've setup a test environment using Hyper-V 2016 (with the above update) with both a 2012 R2 and a 2016 guest file server cluster and have been getting some strange behaviour with backing up using Veeam. I've found that if both guest cluster nodes are running on the same Hyper-V host then the backup job always completes successfully for both guest cluster nodes. However if they are on separate Hyper-V hosts then one of the guest nodes completes successfully and the other fails with the following error. (no it's not a typo).

    4/09/2017 4:14:15 PM :: Importing snapshot on offhost proxy LAB-HV02-16  
    4/09/2017 4:14:17 PM :: Error: Not supported for windows 2015.  

    Hi,

    We are having the exact same issue on a newly created Windows Server 2016 Hyper-V cluster. We have two Windows Server 2012 R2 VMs running as a guest cluster, with a shared VHD-set. When both VMs are on the same host, Veeam backup is running fine. When both VMs are on different hosts, we get the exact same error message (Not supported for Windows 2015) on one of the VMs. When the backup job is being retried, it runs fine, but then the shared VHD is being backed up twice. When both VMs are on the same host, Veeam knows that they share the same VHD, so it will back it up just once.

    Another issue that we have, is that snapshots aren't being cleaned up properly. In Hyper-V manager, I don't see any snapshots for the VMs, but the VMs are still running on differencing disks (AVHDX-files).

    Monday, September 04, 2017 8:15 AM
  • Out of curiosity, those of you who have been successfully backing up the guest file server cluster after the update did you do a rolling cluster upgrade from a Hyper-V 2012 R2 cluster or did you create a new Hyper-V 2016 cluster and move the VMs over?

    I've setup a test environment using Hyper-V 2016 (with the above update) with both a 2012 R2 and a 2016 guest file server cluster and have been getting some strange behaviour with backing up using Veeam. I've found that if both guest cluster nodes are running on the same Hyper-V host then the backup job always completes successfully for both guest cluster nodes. However if they are on separate Hyper-V hosts then one of the guest nodes completes successfully and the other fails with the following error. (no it's not a typo).

    4/09/2017 4:14:15 PM :: Importing snapshot on offhost proxy LAB-HV02-16  
    4/09/2017 4:14:17 PM :: Error: Not supported for windows 2015.  

    Hi,

    We are having the exact same issue on a newly created Windows Server 2016 Hyper-V cluster. We have two Windows Server 2012 R2 VMs running as a guest cluster, with a shared VHD-set. When both VMs are on the same host, Veeam backup is running fine. When both VMs are on different hosts, we get the exact same error message (Not supported for Windows 2015) on one of the VMs. When the backup job is being retried, it runs fine, but then the shared VHD is being backed up twice. When both VMs are on the same host, Veeam knows that they share the same VHD, so it will back it up just once.

    Another issue that we have, is that snapshots aren't being cleaned up properly. In Hyper-V manager, I don't see any snapshots for the VMs, but the VMs are still running on differencing disks (AVHDX-files).

    2012R2 guest clusters arent supported anyway on 2016 hosts with regards to backup. The only supported guest clusters are the ones with VHDS disks and guest os 2016 as well.

    Basically MS forces us to migrate to 2016 guest clusters which then turn out to be bugged like hell. Again this is why I hate the cumulative patches. Some patch killed it, and you MUST install it or your security is compromised. Then it takes over a year to fix the issue. Way to go.

    If you want to backup guest 2012R2 clusters, host them on 2012R2 hosts with shared vhdx.


    Monday, September 04, 2017 8:55 AM
  • I have just created a test environment. On a single Windows Server 2016 Hyper-V host, I installed the failover cluster role and connected a Synology NAS as iSCSI target for shared storage. I have created a single node Hyper-V cluster with a cluster shared volume and installed two Windows Server 2016 VMs with shared VHDS disks as a guest cluster.

    I have backed up both VMs succesfully, but also on this environment the snapshots aren't being cleaned up properly. There is no checkpoint in Hyper-V manager, but the VMs keep running from differencing disk. In the eventlog on the Hyper-V host, I see the following events for both VMs:

    • Error 4-9-2017 11:17:08 Hyper-V-VMMS 19100
      'VM1' background disk merge failed to complete: General access denied error (0x80070005). (Virtual machine ID 646F79C2-368B-499B-8414-43405F755F9F)
    • Error 4-9-2017 11:17:09 Hyper-V-VMMS 19100
      'VM1' background disk merge failed to complete: The process cannot access the file because it is being used by another process. (0x80070020). (Virtual machine ID 646F79C2-368B-499B-8414-43405F755F9F)

    When shutting down one of the VMs, to let the disks merge, I first get the error 

    • 'VM1' background disk merge failed to complete: The process cannot access the file because it is being used by another process. (0x80070020). (Virtual machine ID 646F79C2-368B-499B-8414-43405F755F9F).

    After shutting down the other VM, I get the following event:

    • Error 4-9-2017 11:32:54 Hyper-V-VMMS 19100
      'VM2' background disk merge failed to complete: The request is not supported. (0x80070032). (Virtual machine ID 7A7C05E6-1DF8-471F-A87E-CED91CACE9F2)

    • Edited by Leo Schroer Monday, September 04, 2017 9:40 AM
    Monday, September 04, 2017 9:37 AM
  • I have just created a test environment. On a single Windows Server 2016 Hyper-V host, I installed the failover cluster role and connected a Synology NAS as iSCSI target for shared storage. I have created a single node Hyper-V cluster with a cluster shared volume and installed two Windows Server 2016 VMs with shared VHDS disks as a guest cluster.

    I have backed up both VMs succesfully, but also on this environment the snapshots aren't being cleaned up properly. There is no checkpoint in Hyper-V manager, but the VMs keep running from differencing disk. In the eventlog on the Hyper-V host, I see the following events for both VMs:

    • Error 4-9-2017 11:17:08 Hyper-V-VMMS 19100
      'VM1' background disk merge failed to complete: General access denied error (0x80070005). (Virtual machine ID 646F79C2-368B-499B-8414-43405F755F9F)
    • Error 4-9-2017 11:17:09 Hyper-V-VMMS 19100
      'VM1' background disk merge failed to complete: The process cannot access the file because it is being used by another process. (0x80070020). (Virtual machine ID 646F79C2-368B-499B-8414-43405F755F9F)

    When shutting down one of the VMs, to let the disks merge, I first get the error 

    • 'VM1' background disk merge failed to complete: The process cannot access the file because it is being used by another process. (0x80070020). (Virtual machine ID 646F79C2-368B-499B-8414-43405F755F9F).

    After shutting down the other VM, I get the following event:

    • Error 4-9-2017 11:32:54 Hyper-V-VMMS 19100
      'VM2' background disk merge failed to complete: The request is not supported. (0x80070032). (Virtual machine ID 7A7C05E6-1DF8-471F-A87E-CED91CACE9F2)

    That's the exact same thing as I am seeing. Same errors. Files being locked and VM's running from snapshots. In addition because of this when you try to move an affected VM to another host, it fails miserably and the VM will crash.

    No MS, this is certainly not fixed yet. I can't believe how they seriously want us to use this in production, moving from 2012R2 were everything works to 2016 which is bugged beyond believe.

    Also my MS representative doesn't respond anymore.

    Monday, September 04, 2017 9:48 AM
  • Seems I spoke to soon. I have the exact same issue. Disk merge does not complete after backup and the VM crashes if you try to move the virtual machine. I have a lot of leftover avhdx files and I haven't got any way to inspect them or list their chain.

    I will try to take the csv offline to see if it releases the lock. Hope it won't fail with corrupt differencing disk chain error.

    2016 is a turning point in MS history. Top buggy and unstable software products and windows services. At least with hyper-v. Had major issues with hyper-v services also.

    Monday, September 04, 2017 1:11 PM
  • Hey guys,

    in my case the update which fixes the issue for some people, brokes up our 2 Node S2D Cluster.

    See https://social.technet.microsoft.com/Forums/en-US/8df0407e-aa1c-4faa-9b04-decccaddb154/storage-spaces-direct-2node-cluster-break-by-kb4034661?forum=winserverClustering&prof=required

    I was fully able to recreate the issue serveral times. Currently i'm working on reverting to the RTM Version without any dataloss on our productive system.

    Has anyone else same kinds of issues?

    Regards

    Thursday, September 07, 2017 12:30 PM
  • To all,

    I also opened a case with MS, several months ago.

    They said the problem would be fixed at the end of august 2017, but it appeared not to be the case.

    Therefore, I contacted them again, this time they said:

    "The public fix has not yet released and expected to release on OCT 17 along with RS3 update"

    So let's wait another month, and keep our fingers crossed

    Sunday, September 10, 2017 6:48 AM
  • Our MS tech confirms they're still looking into the issue, and that customers are still experiencing issues post-hotfix.
    Tuesday, September 26, 2017 5:27 AM
  • My MS rep does not respond anymore to anything. I'm quite mad at MS for this by now.

    After KB4034661 we have had the beforementioned issues; file being locked after which in the end the VM would crash when it's moved to another node. After that had been cleared up again, it now turns out I can't move the guest-cluster VM's at all anymore, meaning I am unable to drain my nodes for maintenance. I now get messages like:

    The collection 'Hyper-V Collection' (Collection ID 46482503-DF2E-4701-8B00-831698947195) was not found for virtual machine '<vm name>' (Virtual machine ID 11579E1F-7679-44CC-84C7-7CE66584D7E6).

    Indeed, that VM-Group does not exist, but also I am unable to get it recreated. I don't now when it gets created in the first place, probably when assigning VHDS disks to it. But I removed all VHDS disks from both guest-cluster nodes, re-added them but still the same issue. Manually creating a VM-group with both VM's in it does not work either.

    This whole Hyper-V 2016 adventure left us mainly with very, very sore stumachs with regards to Server 2016. There is so many bugs. The start-menu not working properly on RDS hosts. Hyper-V not having working features that 2012R2 did have. Our 2016 AppV farm completely unusable after yet another weird move in some cumulative patch. Cumulative meaning either you get new issues, or your security is at risk. Very bad situation in our opinion.

    If I could without too much hassle I'd revert to Hyper-V 2012R2 and advice everyone to stay away from 2016. The cumulative patch system alone would justify that for us.

    I'm thinking about raising a new ticket for this VHDS issue, so I at least get some response from MS again.

    Tuesday, September 26, 2017 7:02 AM
  • Haha!

    That is a bit bad!

    In any case, it shouldn't be that complex - we shouldn't need to mess about with VM Groups by CLI (even though I'm all for CLI). It should be easily visible and configurable through FCM / HV Manager.

    Maybe we need to mention we're considering moving to Nutanix + Acropolis to get some traction :)

    Tuesday, September 26, 2017 7:33 AM
  • Yeah that's one of my main complains with MS these days. In the good ol' days of VMS and the likes, you got 10 kilograms of manuals with an OS, and even MS-DOS and the early Windows versions had proper manuals. Now with the likes of Server 2016 it's released, and if you are lucky there are some technical blogs with tech info about new features, but MS own documentation is merely on functional level on a lot of subjets. How on earth was I to find out the manual commands to create snapshots (sorry checkpoints) for VMGroups? Apart from that, the logging... oh my. In the aforementioned VMS, if an error occured, the log would tell you what you needed. Now, it's so often that we see an 'unknown error', -38474 or some other random negative number.

    Of course, all is in perspective. A lot of things are well done in 2016. And of course OS'es these days are much more complex than 25 years ago, but still I think there is a lot, and actually more and more lacking in Windows OS'es since 2008R2 with every release. The amount of serious bugs that had us revert to our 2008R2 environment (especially in the RDS environment), and if it were easy to 2012R2 (Hyper-V especially) are astonishing.

    Ah well.... we'll see how this goes. I don't believe any ETA MS gives out anymore until I have a working patch running. They often give you a candy to keep you quiet but no real fix.

    Tuesday, September 26, 2017 9:54 AM
  • Any updates on this? Kind of insane this has been an issue this long. Are failover guest clusters and Veeam just not used very often out there?
    Friday, October 06, 2017 2:25 PM
  • I have exactly the same problem. 

    One of the servers is in an off state and it can't be started anymore. The rest is running, but with a status "incomplete VM configuration" (scvmm). Is there a way to get the servers back to a healthy state?

    Thanks!

    Monday, October 16, 2017 8:46 AM
  • Try from the Hyper-V manager, or the Failover Cluster manager (I suspect you are using VMM now to start the VM). With all this unreliable crap since Hyper-V 2016 I've had VMM screw up multiple times. Sometimes I even have one VM twice in VMM, on seperate hosts while of ocurse it only runs on one. Try to 'repair' the VM in VMM, and choose 'ignore' to refresh it. If that doesn't work, refresh the hosts themselved, or do a 'Refresh Virtual Machines' on host-level and give that some time.

    But I've had to delete VM's from VMM more than once, in order for VMM to 'find it back'.

    Monday, October 16, 2017 9:04 AM
  • I've taken these steps already, but without success. Deleting the vm's is not an option I'm afraid...

    Anyway thank you for the quick answer.

    Monday, October 16, 2017 10:07 AM
  • When the VM is running, deleting it from VMM does not actually delete the VM. It only deletes its record from VMM and it will 'find' the VM again after a refresh. Anyhow that's a bit tricky I reckon.

    But still assuming you are using VMM, what's the state of the VM in Failover Cluster manager? do you actually have missing resources there? What does the eventlog say for the VM?

    Monday, October 16, 2017 10:10 AM
  • In failover cluster manager everything seems healthy. Except for the vm that was migrated and is now down (cluster service errors).

    But the servers that were not migrated are running healthy in hyper-v manager.

    edit: Cluster Shared Volume 'DSL-CSV51' ('Cluster Virtual Disk (DSL-CSV51)') has entered a paused state because of 'STATUS_USER_SESSION_DELETED(c0000203)'. All I/O will temporarily be queued until a path to the volume is reestablished.

    Now there is a warning concerning the csv's. Will solve this and hope this will resolve the issue.

    • Edited by Matiman1993 Monday, October 16, 2017 11:10 AM
    Monday, October 16, 2017 10:21 AM
  • Any news regarding this issue?

    Is this fixed in https://support.microsoft.com/da-dk/help/4041691/windows-10-update-kb4041691 - or?

    Monday, October 23, 2017 1:06 PM
  • As far as my testing goes it doesn't fix it. I've read somewhere (maybe even in this thread, I'm not sure) that Server build 1709 contains a fix. But it turns out that's CORE-server only, not GUI. For our internal systems we use GUI, only DMZ is core, so I've not tried with 1709 as it's of nu use to us anyway.
    Monday, October 23, 2017 1:24 PM
  • Does anyone else have problem inspecting the VHD-Set disks from the GUI as well?

    They are fully functional from within the VMs and works well when we move them from one node to the other.


    Friday, November 03, 2017 7:23 PM
  • Hi, 

    after implementing the latest patches in our test environment, i can say it is not fixed with the gui based version.

    A snapshot is possible, but just 1 time, after that you need to reboot the machine, because the VMs Disks are in use.

    


    Tuesday, November 14, 2017 9:04 AM
  • i have tested Veeam backup of guest cluster with VHD Set, with Hyper-V 1709, same situation: one backup works, the next raise an error and the vhds stay in use until the virtual machine is rebooted.
    Tuesday, November 14, 2017 10:34 AM
  • I've updated with 2017-11 patch and it made the VHDS disks corrupt or something, I can still inspect them but they won't let me delete and add them again or add them to a new VM..

    Also get Event ID 128 in Eventviewer

    # Application and Services logs -> Microsoft-> Windows -> Hyper-V-Shared-VHDX #

    Error opening handle for initiator. Initiator: {00000000-0000-0000-0000-000000000000}. Hostname: . File: \HYPER-V\VHDSETS\CLUSTER-LOG.VHDS:SHAREDVIRTUALDISK. Error: The layered file system driver for this IO tag did not handle it when needed..



    Thursday, November 16, 2017 12:44 PM
  • It's rediculous by now. Ever since kb3213986 these issues are breaking backup. That's almost a full year now. My MS rep doesn't respond at all anymore.

    By now I'd advice anyone to stay the hell away from Hyper-V 2016 with its broken cumulative update system and all its flaws. It's beyond believe MS is not fixing this. They just won't. We've had so many promises. First July, then August, then October... Then it was in build 1709, but still that's broken.

    I don't have any trust in this ever getting fixed honestly.

    Thursday, November 16, 2017 12:50 PM
  • A little bit of news...

    I run a 3 host HV Cluster, with 2016 with Veeam as the backup software.  After upgrading to KB4048953 it appears to have successfully created the backup with the VM's off last night.  This morning, I started the incremental and it appears to be succeeding (11%, snapshot created).  I am going to do some more testing (a full with it running and a incremental with it running), however either this KB or the previous one (KB4048955) may have fixed it.


    ----------------------------------------- Dan Sheppard

    Wednesday, November 22, 2017 3:15 PM
  • A little bit of news...

    I run a 3 host HV Cluster, with 2016 with Veeam as the backup software.  After upgrading to KB4048953 it appears to have successfully created the backup with the VM's off last night.  This morning, I started the incremental and it appears to be succeeding (11%, snapshot created).  I am going to do some more testing (a full with it running and a incremental with it running), however either this KB or the previous one (KB4048955) may have fixed it.


    ----------------------------------------- Dan Sheppard

    Check https://blog.workinghardinit.work/author/workinghardinit/. For him it seems checkpoints themselves are working with the November patch, but merging them doesn't.

    Are your checkpoints actually merging?

    Wednesday, November 22, 2017 4:30 PM
  • I have tested on my test-Hyper-V cluster with KB4048953 installed, but it does not function for me. I still can't create snapshots of guest-cluster-VM's like I used to before KB3200970. Veeam also still fails backing up those VM's.

    So no fix for us.

    Wednesday, November 22, 2017 6:15 PM
  • Merged fine.

    The VM's were on the same host however.  I am having to wait for a different job to finish that is about 1TB.


    ----------------------------------------- Dan Sheppard

    Wednesday, November 22, 2017 6:34 PM
  • I am getting a issue with Veeam when they are not on the same host now (this appears to be a Veeam specific issue), however they still merge the Checkpoint without any issues.

    ----------------------------------------- Dan Sheppard

    Thursday, November 23, 2017 3:25 PM
  • For some reason it seems my vhds decided it wanted to be a "parent / child" disk. Or maybe this is always the case?

    Anyway it seems veeam has tried to make .mrt and .rct files. And that there is also a parent / child relation on the disks. Veeam fails on 2nd backup. And I'm unable to "merge" the parent / child disk. Anyone stuck in this state as well?

    Thursday, November 30, 2017 6:22 AM
  • I am getting a issue with Veeam when they are not on the same host now (this appears to be a Veeam specific issue), however they still merge the Checkpoint without any issues.

    ----------------------------------------- Dan Sheppard

    No, this is NOT Veeam specific. The issue is, and has always been (for us at least) creating checkpoints of VM groups when VM's reside on different hosts. Now since KB3200970 creating snapshots wouldn't work anyway. But that has been fixed in KB.... (forgot the exact number). However now we have merging issues and stuff like that. The original issue still exists though, in our Hyper-V cluster it only works when the guest-cluster-vm's reside on the same hosts. That's totally useless once again, as you want to device all guest-cluster-vm's over multiple hardware boxes to lower the risk of failure.

    With CSV's on a SAN and VHDS on that, we've got severe checkpoint merge issues now, resulting in VM's finally crashing and rebooting. I really would like some insights from MS why it takes such a rediculous long time to fix this.

    Thursday, November 30, 2017 8:45 AM
  • No, this is NOT Veeam specific. The issue is, and has always been (for us at least) creating checkpoints of VM groups when VM's reside on different hosts. Now since KB3200970 creating snapshots wouldn't work anyway. But that has been fixed in KB.... (forgot the exact number). However now we have merging issues and stuff like that. The original issue still exists though, in our Hyper-V cluster it only works when the guest-cluster-vm's reside on the same hosts. That's totally useless once again, as you want to device all guest-cluster-vm's over multiple hardware boxes to lower the risk of failure.

    With CSV's on a SAN and VHDS on that, we've got severe checkpoint merge issues now, resulting in VM's finally crashing and rebooting. I really would like some insights from MS why it takes such a rediculous long time to fix this.

    This error is veeam specific, as it is related to importing the snapshots into Veeam.

    My checkpoints are successful, they also merge successfully.  Don't presume to tell me what is and is not going on in my own Hyper-V & SOFS cluster as there is a good chance I know way more about it then you.  I think I have tracked it down to the offhost/onhost proxy.  I have a feeling a offhost proxy is required when doing this, but I am not 100% sure, but I will be standing up a offhost proxy shortly.


    ----------------------------------------- Dan Sheppard

    Thursday, November 30, 2017 3:50 PM
  • Success!

    Here is what I had to do in Veeam:

    • KB4048953 minimum
    • CBT off
    • Use a offhost proxy that had full access to the SOFS share.

    ----------------------------------------- Dan Sheppard

    Friday, December 01, 2017 10:35 PM
  • Hi,

    I followed Volodymyr posted information and created a checkpoint, however i'm unable to delete it (CollectionID is eq mine VMgroup):

    Delete checkpoint:
    $collSnapSvc = Get-WmiObject -Namespace Root\Virtualization\v2 Msvm_CollectionSnapshotService
    $VmColl = Get-WmiObject -Namespace Root\Virtualization\v2 Msvm_SnapshotCollection|? "CollectionID" -eq "32191801-1857-490D-B3B3-2B5F92F05487"
    $collSnapSvc.DestroySnapshot($VmColl) 

    here are more details:

    https://social.technet.microsoft.com/Forums/windows/en-US/bd66c764-4b49-4e87-b8e5-75f905732599/remove-checkpoint-from-vmgroup?forum=winserverClustering

    could someone advise anything? 

    thanks

    Monday, December 04, 2017 3:54 PM
  • Great, how would I do off-host proxy with Dell MD1420 and Cluster shared volumes not on SOFS?

    How would it work with CSVs? Do I need another server with direct attachment to the JBOD/storage to install proxy software on?

    Wednesday, December 06, 2017 11:27 PM