none
2012 R2 - Checkpoint cleanup from failed VM backup RRS feed

  • Question

  • We recently had a network traffic / virtual switch issue crop up, due to VMQ being enabled and improperly configured on Broadcom 10GbE adapters.  While that failure occurred, our backup appliances were taking backups of VMs.  Because of lost network connectivity, the Hyper-V checkpoints used for those backups remain on some of our VMs.

    These checkpoints do not have the usual "Delete Checkpoint" / "Delete Checkpoint Subtree" options available to them.  How do I remove them so the virtual disks are merged and back to a single VHDX per virtual disk?


    Monday, March 17, 2014 3:23 PM

Answers

  • I tried using remove-vmsnapshot, with no success.  The snapshot/checkpoint type is "recovery".  I can go in and manually merge the disks together with no problems, but obviously that does nothing for the checkpoint that still exists for the VM.

    The PowerShell cmdlet executes error free, but nothing is done to the checkpoint.

    As crude as it is, at this point the only way to clean it up that I can see is to create another VM, attach the merged disks, then clean the configuration as needed.

    Wednesday, April 9, 2014 7:53 PM

All replies

  • You should be able to delete this through PowerShell with the "remove-vmsnapshot" cmdlet.

    This posting is provided AS IS with no warranties, and confers no rights. You assume all risk for your use.

    Tuesday, March 18, 2014 12:20 AM
  • Hi lassja,

    It seems that snapshot type is not standard .

    I tried to delete a snapshot which is "replica type" using " remove-vmsnapshot " ,but I can not delete it .

    you may try to check the snapshot type via powershell command " get-vmsnapshot " .

    If it is "replica type" , please refer to following thread to merge snapshot manually :

    http://social.technet.microsoft.com/Forums/en-US/68cc78dd-0e71-4bec-ab29-ffa095f01b21/unable-to-delete-initial-replica-snapshots-in-hyperv-2012?forum=winserverhyperv

    Best Regards

    Elton Ji


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
    Thanks for helping make community forums a great place.



    Wednesday, March 26, 2014 1:00 AM
    Moderator
  • I tried using remove-vmsnapshot, with no success.  The snapshot/checkpoint type is "recovery".  I can go in and manually merge the disks together with no problems, but obviously that does nothing for the checkpoint that still exists for the VM.

    The PowerShell cmdlet executes error free, but nothing is done to the checkpoint.

    As crude as it is, at this point the only way to clean it up that I can see is to create another VM, attach the merged disks, then clean the configuration as needed.

    Wednesday, April 9, 2014 7:53 PM
  • Hello, I'm also facing the same thing as lassja did.  2012R2 Hyper-V with VMs that cannot be deleted, only we had issue with lsass.exe crashing our systems and maintenance on the Equalogic SAN.  So we have about 220 VMs with about 53 of them being affected with Snapshots that show as the name of the server and the Snapshot Type as Recovery, like below.   Not able to be deleted and not able to go through the steps as above.  We are in a 4-node Cluster so the concern of going in and making changes to the cluster storage is an issue.  I attempted to do the remove-vmsnapshot.  No luck.  Shutting down. No Luck.  Running a new backup. No luck.  Anyone have anything else that might be possible at the same time not blowing up our Cluster volumes.

    wsus1     wsus1- Backup - (5/10/2014 - 4:58:21 PM)     Recovery     5/10/20...

    Thanks ahead of time.


    Dave


    Dave Kreitel

    Sunday, May 11, 2014 8:17 AM
  • We have the same issue with a number of VMs on a cluster. Opened a support case.

    A team effort has resulted in manually merging disks and editing the configuration XML files.

    Not an ideal solution, but works...

    Is there an easier solution?

    Monday, January 12, 2015 10:40 PM
  • We have now got an improved way to do this without ANY VM downtime. Edit the snapshot configuration file, in the snapshot location, <snapshot_guid>.xml  file as follows:

    At the bottom of the <snapshots> section change:

     <type type="integer">32768</type>

    Which I guess represents a recovery snapshot to:

     <type type="integer">2</type>

    Which I guess is a standard snapshot.

    Need to edit when elevated and stop and disable the VMMS service prior to editing and start and change back to automatic after.

    The snapshot type changes and can now be deleted!! I take no responsibility if this goes wrong but worked for us.

    • Proposed as answer by Dave-Hall Friday, January 16, 2015 11:17 AM
    Thursday, January 15, 2015 8:48 PM
  • Marcus, could you provide more details regarding the 'manually merging disks and editing the configuration XML files'?  I have a similar situation except there are a couple of actual checkpoints in the beginning of the chain and then 47 - Backup- checkpoints in the chain.  This makes it impossible to follow other suggested procedures like creating an additional checkpoint an then using the PowerShell with the "remove-vmsnapshot".

    Changing the XML file to make the checkpoint appear as a standard snapshot still results in a 'catastrophic faillure'.

    I was able to manually merge all the backups and snapshots, leaving only the original 3 disks + 3 files without a parent.  I now have no option but to recreate a new VM and attach these disks, resulting in having to reregister in domain etc. Unless I would know how to change the configuration XML files.

    Any other suggestions or help is appreciated.

    Wednesday, March 25, 2015 6:16 PM
  • Could you elaborate on this, i tried and it is alwas telling me that the file is locked

    Need to edit when elevated and stop and disable the VMMS service prior to editing and start and change back to automatic after.

    TKS Marcus

    Thursday, April 9, 2015 3:24 PM
  • You need to set the VMMS service to disabled and then stop it, before editing. If the service is running you cannot edit the file.

    Must remind you that this isnt supported but we have found it works!

    Marcus

    Thursday, April 9, 2015 3:27 PM
  • Just noticed this problem with my entire cluster, most of my vm's now have a unremovable checkpoint due to failed snapshotting during backup.

    Tried Marcus' method of stopping the vmm service, editing the snapshot xml and changing the integer from 32768 to 2, restarting the service and then deleting the checkpoint, which appears to merge the differencing disk back into the parent disk and appears to retain recent changes made.

    Can I just double/triple check that this method hasn't caused you any unforeseen problems since you started using this workaround, as I am planning to fix all my vm's with this method and don't want anything cropping up down the line that can't be fixed!

    Thanks,

    Craig

    Monday, February 22, 2016 10:04 AM
  • Hi Craig,

    We used the method multiple times and it didn't have any adverse effects.

    However, I'm now an MS employee (wasn't when I answered the post), and note it shows that on all the posts above I want to reiterate it isn't supported etc, etc...

    Hope that helps,

    Marcus
    Monday, February 22, 2016 10:13 AM
  • Hi Craig,

    We used the method multiple times and it didn't have any adverse effects.

    However, I'm now an MS employee (wasn't when I answered the post), and note it shows that on all the posts above I want to reiterate it isn't supported etc, etc...

    Hope that helps,

    Marcus
    Are you aware of a supported method? Is there any official word from MS on this?
    Monday, February 22, 2016 10:19 AM
  • Hi- I'm having the same issue. 

    I followed the instructions to change a checkpoint from a recovery checkpoint to a regular one. I got the option to delete the checkpoint and it failed with a Catastrophic error 0x8000FFFF.

    Looking at the snapshot xml files I think I figured out the problem but I’m a little leary to change it without discussing it first. I believe that after the first few snapshots the snapshot list order changes.

    The first snapshot is: 098AA005-FD8C-4194-9EEE-C52142CEAA6F

    The snapshot section lists it as “0” in the list:

    -<snapshots>

    <backed_up type="bool">True</backed_up>

    -<list>

    <size type="integer">0</size>

    </list>

    <type type="integer">2</type>

    </snapshots>

    The next snapshot is: 0284F644-7AC0-4D0B-86F1-0D73BEB6AA0A

    This is correctly listed as “1” in the chain with node 0 listed correctly

    -<snapshots>

    <backed_up type="bool">True</backed_up>

    -<list>

    -<node0>

    <guid type="string">098AA005-FD8C-4194-9EEE-C52142CEAA6F</guid>

    </node0>

    <size type="integer">1</size>

    </list>

    <type type="integer">2</type>

    </snapshots>

    The next snapshot is: EC10F0D5-2C43-4DB5-975B-28873291CF1E

    Here it lists itself correctly as “2” in the chain BUT it seems to have swapped the previous 2 nodes listing 0284F644-7AC0-4D0B-86F1-0D73BEB6AA0A as node 0, and 098AA005-FD8C-4194-9EEE-C52142CEAA6F as node 1.

    -<snapshots>

    <backed_up type="bool">True</backed_up>

    -<list>

    -<node0>

    <guid type="string">0284F644-7AC0-4D0B-86F1-0D73BEB6AA0A</guid>

    </node0>

    -<node1>

    <guid type="string">098AA005-FD8C-4194-9EEE-C52142CEAA6F</guid>

    </node1>

    <size type="integer">2</size>

    </list>

    <type type="integer">2</type>

    </snapshots>

    The next snapshot is: 37FACF2C-74E9-42F6-B9C7-5FABC7ECF70C

    Again listed itself correctly as number “3”in the list BUT the first two are swapped.

    -<snapshots>

    <backed_up type="bool">True</backed_up>

    -<list>

    -<node0>

    <guid type="string">0284F644-7AC0-4D0B-86F1-0D73BEB6AA0A</guid>

    </node0>

    -<node1>

    <guid type="string">098AA005-FD8C-4194-9EEE-C52142CEAA6F</guid>

    </node1>

    -<node2>

    <guid type="string">EC10F0D5-2C43-4DB5-975B-28873291CF1E</guid>

    </node2>

    <size type="integer">3</size>

    </list>

    <type type="integer">32768</type>

    </snapshots>

    This continues throughout all 50 files. The question is should I modify all the files to make them correct?

    Sunday, February 28, 2016 2:21 AM
  • Did anyone get over this? Im having the exact same issue I have manually merged the avhd(x) files but the checkpoints still exist within the GUI of the VM Manager. I have restarted the service and no joy :(
    Monday, March 13, 2017 11:00 AM
  • When a failed backup of the VM left the hindering checkpoint. The only way to deal with it is by using Powershell.

    This command will help you to determine that a checkpoint is currently open on the VM:

    Get-VMSnapshot -VMName VMNAME-WITHCHECKPOINT.domain.com -ComputerName HOST-HYPER-V.DOMAIN.COM| fl – this returns the VM state

    and his command will remove the checkpoint and merge the AVHDX file into it’s parent VHDX file.

    Get-VMSnapshot -VMName VMNAME-WITHCHECKPOINT.DOMAIN.COM -ComputerName HOST-HYPER-V.DOMAIN.COM| Remove-VMSnapshot


    Monday, April 24, 2017 11:51 AM
  • Same thing happens to us - running a Server 2016 Hyper-V Cluster with Veeam B&R 9.5.0.1536 - and I also experience that Veeam/Windows doesn't always remember to clean up correctly.

    Anyone else?

    Monday, October 8, 2018 11:55 PM
  • Exactly the same on Server 2016 Hyper-V backing up also Server 2016 VM (cluster with vhd sets), Veeeam B&R 9.5 U3

    Snapshots do NOT show in GUI, not in PS.

    All data is written to now snapshot/differential disk.

    In event log I have entries:

    Cannot delete checkpoint: One or more arguments are invalid (0x80070057).

    

    Tuesday, March 12, 2019 10:17 PM
  • Exactly the same on Server 2016 Hyper-V backing up also Server 2016 VM (cluster with vhd sets), Veeeam B&R 9.5 U3

    Snapshots do NOT show in GUI, not in PS.

    All data is written to now snapshot/differential disk.

    In event log I have entries:

    Cannot delete checkpoint: One or more arguments are invalid (0x80070057).

    


    Same. Does anyone has a Suggestion?
    Monday, August 12, 2019 3:41 PM
  • It requires downtime. If you can afford it then rest is easy:

    - if using cluster VM, power off BOTH

    - in Hyper-V Manager console chose Edit disk, select the current differencing disk / merge / to parent

    - rename the original mydisk.vhds to mydisk.vhds_

    - in FCM edit settings of the VM, select disk in question, New (create new mydisk.vhds, no matter what settings, defaults are fine), do NOT yet click apply, leave FCM & carry on below:

    process created .vhds control file & corresponding .avhdx (copy the name of that file & delete it), now rename your original merged .avhdx to what you have on clipboard

    - in FCM click Inspect (it will show your original disk details), now you can click apply & now delete mydisk.vhds_

    At this point didk is back to just a single .vhds & .avhdx

    I have done it for my VM cluster with 9 disks, not a problem.

    But if I can do it by hand, sure MS could manage to write they software to do it automatically?

    Seb

    Friday, August 16, 2019 9:22 PM