none
Server 2016 - Unable to apply checkpoints RRS feed

  • Question

  • We have a Server 2016 Hyper-V cluster (fully patched) with which we are experiencing issues applying checkpoints. If we manually create a production or standard checkpoint in Hyper-V manager on 2012R2/Server 2016 VMs we are unable to revert back to it. The error message we get is "An error occured while attempting to apply the checkpoint". 

    To simplify troubleshooting we have moved a Server 2016 VM out of the cluster and onto local storage, but with the same results. There doesn't appear to be much else to go on in the event viewer. Can anyone suggest a resolution please?


    Monday, September 11, 2017 1:38 PM

All replies

  • Hi CaptainCTrick,

    1. Please inspect the virtual hard drive of the VM you failed to apply the checkpoint, check if the chain between the avhdx and the parent vhdx is well:

    2. Please check if you delete the checkpoint, it can merge the avhdx files to the parent disk correctly;

    3. Also test if you could merge the checkpoints to the parent disk manually via "Edit Disk">"Merge";

    Best Regards,

    Anne


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Tuesday, September 12, 2017 9:13 AM
    Moderator
  • Thanks for the reply. I have followed your advice and:

    1. The chain between the avhdx and vhdx is fine.

    2. and 3. I can successfully delete the checkpoint and the merge works. 

    It appears I'm having similar issues to EvgenGematogen looking at his thread. Maybe the latest update broke something?

    Tuesday, September 12, 2017 3:08 PM
  • Hi CaptainCTrick,

    >Maybe the latest update broke something?

    In my lab, I did a test, I have no issue to apply checkpoints for each OS VM with the August update installed.

    As a test, you may check if uninstalling the update could resolve the issue. If the issue is actually related with the updates in your scenario, welcome to feedback.

    Best Regards,

    Anne


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, September 13, 2017 1:42 AM
    Moderator
  • Hi,

    Please go through the logs in eventvwr for further details

    Eventvwr --> Applications and services logs --> Microsoft --> windows --> Hyper-V VMMS, Admin, Operational, etc., also there are many other Hyper-V logs.

    please post the same here for further assistance

    Regards,

    Bala

    Wednesday, September 13, 2017 4:57 AM
  • I have checked all the Hyper-V event logs, unfortunately the error isn't very informative:

    Event 15060, Hyper-V-VMMS

    'SERVER1 failed to apply checkpoint. (Virtual machine ID A3EE80F1-24D8-4919-9D68-2DC9909DC1F2)


    Friday, September 15, 2017 2:28 PM
  • Further update on this bizarre issue. If I restore the server from a Veeam backup (from the previous day when checkpoints didn't work) then I can create and apply a checkpoint successfully. It doesn't appear to be a permissions issue. Would there be a 'lock' file or anything similar that could stop a checkpoint applying?  
    Tuesday, September 19, 2017 10:29 AM
  • Hi CaptainCTrick,

    What is the different configuration between the orignial problematic point and  the "backup" which have no issues applying checkpoint? Does the "backup" without the KB?

    Best Regards,

    Anne


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Tuesday, September 19, 2017 2:43 PM
    Moderator
  • I have the exactly same issue. It worked perfectly on Server 2012, but since we reinstalled our Hyper-V cluster with Server 2016, I can't apply checkpoints anymore.

    I am pretty sure that our setup is ok, and with Procmon I could not find permission issues or something like that.

    I noticed that when I create a test VM and a checkpoint and try to apply that, it always works. But when I create a checkpoint of an actual server, it does not.

    Could it be related to Veeam? Say, when Veeam backs up a VM (with CBT), it somehow prevents applying of checkpoints?

    Tuesday, October 3, 2017 6:36 AM
  • Hello,

    I have exactly the same issue on my hv16 cluster.

    Tried also the same tips. We alse use Veeam Backup.


    Regards, 

    Tuesday, October 10, 2017 2:39 PM

  • I have exactly the same issue on my hv16 cluster.

    Tried also the same tips. We alse use Veeam Backup.



    Interesting. Seems like we all are using Veeam Backup.

    I already opened a case with Veeam because I thought it might be related, but they claim that it has nothing to do with them. Actually I suspected their CBT driver, but at least when evacuate a Hyper-V node and stop their driver, I still cannot apply checkpoints to a local VM.

    Also opened a case with MS now, but there is not real progress yet, except that they acknowledged that there are quite many issues with Checkpoints in Hyper-V 2016.

    Wednesday, October 11, 2017 4:56 AM
  • According to Microsoft this is a known issue and will be fixed with a future update.

    As a workaround they suggested to create new VMs and attach the existing VHDs to those, which works but causes other issues (like resetting NICs). I found another potential workaround that seems to do the trick:

    - copy VM's configuration file to another location
    - delete VM
    - copy configuration file back (which was deleted with the VM)
    - import VM again

    • Proposed as answer by svhelden Thursday, October 19, 2017 5:38 PM
    Thursday, October 19, 2017 5:38 PM
  • This issue still seems to exist. Not with 2K16 VMs but with a VM running Windows Server 2012 R2 (Azure AD Connect VM)/ Did you receive any feedback from Microsoft on this? Thank you.
    Friday, February 9, 2018 9:29 AM
  • yes, feedback from MS would be really nice.

    Same topic in this Thread:

    https://social.technet.microsoft.com/Forums/sharepoint/en-US/229ca1e2-befa-430e-9c96-c58ac23a1e56/apply-the-checkpoint-error-event-id-1506015160?forum=winserverhyperv&prof=required

    Monday, February 19, 2018 2:46 PM
  • This issue still seems to exist. 

    Yes, can confirm that. Issue is still there, even with 2018-02 updates.

    Did you receive any feedback from Microsoft on this?
    No.

    Tuesday, February 20, 2018 6:26 PM
  • We have had a case open with Microsoft on issues involving VMs with checkpoints running on Hyper-V 2016 since August 2016.  Unfortunately 7 months later, we are no better off.  We're actually working on rebuilding VMs and hosts back to 2012 R2 Hyper-V with the assumption we'll get a reliable service back.   I'd really like to hear from others if anyone has found any more information troubleshooting checkpoint issues with Hyper-V 2016 VMs.  Perhaps if more people opened cases with Microsoft, they might take it more seriously? 
    Tuesday, March 6, 2018 9:50 PM
  • How are the VMs configured? Do you use shared VHDX/guest clustering? Are there VMs (apart from freshly created test VMs) where checkpoints work for you?

    Please collect a set of event logs while trying a repro (take prod snapshop / failing apply prod snapshot) using the HyperVLogs PowerShell module:

    https://github.com/MicrosoftDocs/Virtualization-Documentation/tree/live/hyperv-tools/HyperVLogs

    Unfortunately, I can't promise that I'll be able to take a look right away...


    Thursday, March 22, 2018 11:06 PM
  • Hi Lars, since you replied, please take a look at case 117073116120191.   The checkpoint state/tree seems to get corrupt somehow in the process of checkpoints being created or deleted. 

    It doesn't involved shared VHDX's.  We don't use guest or host clustering. Stand alone Hyper-V hosts, VMs running off a SOFS cluster.  The case should have all the debug and diagnostics you want. 

    Thursday, March 22, 2018 11:13 PM
  • Sorry, I do not have access to the support case data/customer data. Which is good from a privacy/data segregation perspective, but sometimes makes things more complicated.

    Is this case still open for you? I can see that it is frustrating if things are not moving forward. Your Technical Account Manager should be able to help escalate this appropriately.

    If you want to follow up directly, you can reach me via [firstname].[lastname]@microsoft.com

    Thursday, March 22, 2018 11:21 PM
  • May 17, 2018—KB4103720 (OS Build 14393.2273)

    Seems to have a fix:

    Addresses an issue that makes it impossible to revert to a virtual machine checkpoint. Reapplying the checkpoint fails with an error. 

    • Proposed as answer by CR64NM98BF67 Monday, May 21, 2018 8:30 PM
    Monday, May 21, 2018 4:06 AM
  • I applied this update to my host...after applying KB 4072650 to my VM (its a recommended update, so I had to go in and look for it and apply it).

    So far I have watched 3 backups start, checkpoints created and then successfully merge with out issue.

    Thanks very much, Robert!!!!!

    Monday, May 21, 2018 8:30 PM
  • and I come in this morning and see orphaned AVHDX...thought I could move onto something else.....guess not : (
    Tuesday, May 22, 2018 1:59 PM
  • So far I've had no issues anymore with KB4103720 installed, after excessive testing. Except for guest-clusters, which is a different story all together...
    Tuesday, May 22, 2018 2:17 PM
  • and I come in this morning and see orphaned AVHDX...thought I could move onto something else.....guess not : (

    "Orphaned AVHDX" are not really related to this issue. This discussion is about applying checkpoints.

    Creating / Deleting / Merging a checkpoint has always worked. Applying a checkpoint (= reverting to it) has not.

    If you cannot delete checkpoints (or orphaned AVHDX files remain after seemingly successful deletion), you have a totally different problem and should create a new thread.

    Wednesday, May 23, 2018 4:15 AM
  • Sorry...I believe (and perhaps in error) the reason checkpoints can't be created is because there are orphaned Avhd(x)'s out there that are not merged and they are being "locked"....but....my mistake. we have rebuilt the server, so, not really looking for solution at this point, was just going to provide an update on what we've seen here over the last week...not worth a new thread to me.
    Friday, June 1, 2018 3:58 PM
  • I just spent 3 days recovering from "orphaned" avhdx files. Last night when I was getting ready to power up the VM's on my HyperV 2016 host, I checked for updates. Lo and behold, the 2018-07 Cumulative Update was available. I ran it.

    I then made a copy of my smallest 2016 Server and configured it as a test vm with no network (so as to not confuse the entire network of course!).

    I ran backups every 5 minutes until I had 20 successful backups.

    I then disabled that backup and ran tests of the production machine. I ran those all night and have 13 good copies and no .avhdx files hanging out.

    I have since enabled the other 2 production backups for the VMs on this 2016 host. They will run today at 1230, 1530, and 1830 local. I will report results here.

    This is the furthest I've gotten with this issue, so I am encouraged yet skeptical. I'm waiting for the other shoe to drop and to have more failures.

    Tick tock.

    Yes, I use Veeam like everyone else in this thread.

    JK
    Tuesday, July 24, 2018 3:24 PM
  • So the backups of the production servers worked normally on the 2 smaller VM's, but the 2.2TB SQL server failed to remove the .avhdx files, but the checkpoints are not shown in HyperV Manager.  Same problem.  Discussion with Veeam and a look at the event log shows the following:

    'SERVERNAME' background disk merge failed to complete: Account restrictions are preventing this user from signing in. For example: blank passwords aren't allowed, sign-in times are limited, or a policy restriction has been enforced. (0x8007052F). (Virtual machine ID AE580C5A-3DCC-4D4B-A395-E9146DF8DF53)

    Veeam suggested I edit the properties of the hypervisor and use the local administrator account, then fix the disks and back up the machine again to see if it works this time.  

    Tuesday, July 24, 2018 9:06 PM
  • Can you please provide more detail on the environment you're using (apart from Veeam)? Are you using a cluster? CSV/SMB shares? If CSV: Are the VMs running on the node that owns the CSV?

    Anything else that should be mentioned?

    Tuesday, July 24, 2018 9:40 PM
  • UPDATE - Luckily when I shutdown the SQL server to edit the disks, the checkpoints merged automatically, so I didn't have that nightmare to haunt me.

    What I changed: I edited the hypervisor in Veeam and added the .\administrator credentials. I also set the SQL job to run every two hours, round the clock and increased my retention points to 360 so I have a month locally. I'm hoping doing so will alleviate the timeout issue and the credential issue all in one fell swoop. My backups will also be much faster as the amount of change should be less for each job run.
    Wednesday, July 25, 2018 1:57 PM
  • I'm sorry for not mentioning it.  There are 3 hypervisors, all are standalone with local SSD RAID on this particular hypervisor.  Each VM has its configuration and disk files on this RAID Disk, the F: drive on the hypervisor.
    Wednesday, July 25, 2018 4:01 PM
  • These backups ran for 19 hours, then failed again and left .avhdx files in the folders.  Same error as above regarding account policies.  I'm still working with Veeam to figure this out.  #Annoyed

    Thursday, July 26, 2018 12:35 PM
  • That seems to be frustrating!

    @Jeff: Did you get this fixed? If so, how?

    Appreciate if you could share how the issue got resolved.

    Same issue with someone else here, they're facing this issue intermittently(ie. after one or two successful backups, the checkpoint fails to merge leading to backup failure).

    Issue is 100% similar.


    Ramkumar

    Monday, April 29, 2019 3:17 PM
  • I also had a similar issue on Windows 10 as my host, with Windows 10 as my guest. 'Windows10' failed to apply checkpoint. (Virtual machine ID 2DAA7A90-4DC7-4BC1-9FDD-AD1545B04FCD).

    No checkpoints could be applied, and unhelpful messages.

    Lucky for me, restarting my laptop fixed it!

    Tuesday, October 22, 2019 10:33 AM