locked
Mystery AVHD file and no snapshots lists in Hyper-v RRS feed

  • Question

  • Need some serious help here.

    I was the victim of the runaway shapshot eating all the availble disk space last year - that was a fun 30 hours to fix.  I still hold the record of having a 400GB snapshot.  Snapshot are evil in production - never to be used again.

    Anyway...

    I have one virtualized exchange server and in Hyper-V mangare, there is no snapshots listed (this is good, since it is an Exchange server and it is in production).

    But, we are running out of diskpace again.  I check the drive and there is a 151GB snapshot (which is active and being used by the Exchange server).  A AVHD file - your basic snapshot differential file. 

    So, it looks like this blight of snapshots is not completely gone.  How do I get rid of this snapshot?  

    I would like some suggestions before I start working on a production system.



    Sunday, September 13, 2009 7:46 PM

Answers

  • Please note the below was with AVHDX not AVHD files and may behave differently in your environment.

    I just ran into this issue - as an alternative resolution with no down time when you find an "orphaned" but live AVHDX file

    • Edit the settings of the affected VM
    • Change the Snapshot File Location to a new location ( I made sure there was space for a new snapshot in this location, but no data was created there in this case)
    • As soon as you apply the snapshot begins merging
    • In a few minutes the AVHDX file will be deleted and merged with the VHDX
    • Reset your Snapshot location to the original location

    • Edited by tommcgeown Thursday, August 7, 2014 2:46 AM
    • Proposed as answer by Kristian Eklund Monday, December 15, 2014 8:15 PM
    • Marked as answer by Denis Dyagilev Saturday, August 25, 2018 7:45 AM
    Thursday, August 7, 2014 2:45 AM
  • Thanks for all of your help.

    Yes, the .xml file does refer to the snapshot in question.  There is nothing in hyper-v manager that says there are any snapshots for that VM.

    There is no AV running on the hyper-v.

    We had in the past had to merge in VHD files (several of them) during Snapshothell and those worked just fine.

    I checked the start date and it was created two months after Snapshothell even happened. 

    We do have a backup (wbadmin) that runs every night - and we are down to 17GB's of free disk space.  Could that be a possible problem?  One of the correlations is that this only happens at night and in the morning, we have to restart the servers.
    • Proposed as answer by Alpha11111 Wednesday, October 14, 2009 10:08 PM
    • Marked as answer by Vincent Hu Monday, November 23, 2009 8:00 AM
    Wednesday, September 16, 2009 4:34 PM

All replies

  •  Even though you deleted it in the GUI the snapshot is not removed until you power down the guest VM in this case your Exchange server.  It will take some time as it looks like a larger snapshot.  I have seen snapshots that are 145GB take about 6-8 hours to merge properly.

    As for suggestions, make sure you have a good backup and give yourself plenty of down time.


    Joseph Noga MCITP, MCSE, MCSA, MCTS CCNA,CCDA,CCVP Managing Consultant VSD Technologies
    Monday, September 14, 2009 1:44 AM
  • Hi Joseph,

    Thanks for the response.

    I'm wondering if any additional hard drive space is required to perform the merge since the hard drive in question currently has ~10GBs free.

    I believe we have 50GBs we CAN free up, but otherwise need to make room...I suppose running the pre-compactor at this point would take quite a while as well?

    Ben
    Monday, September 14, 2009 2:49 AM
  • It shouldn’t take any more space but each environment is different.  The more space you have the better.


    Joseph Noga MCITP, MCSE, MCSA, MCTS CCNA,CCDA,CCVP Managing Consultant VSD Technologies
    Monday, September 14, 2009 2:53 AM
  • Hi,

     

    Generally speaking, we don’t recommend that you use snapshot in a production environment.

     

    Please perform Joseph’s suggestions to power off the problematic VM and then let it merge the snapshot to the parent VHD. More free disk space will be better.

     

    Hyper-V Virtual Machine Snapshots: FAQ

    http://technet.microsoft.com/en-us/library/dd560637(WS.10).aspx

     

    Managing Snapshots with Hyper-V

    http://blogs.msdn.com/virtual_pc_guy/archive/2008/01/16/managing-snapshots-with-hyper-v.aspx

     

    Virtual Machine Snapshotting under Hyper-V

    http://blogs.msdn.com/virtual_pc_guy/archive/2008/03/11/virtual-machine-snapshotting-under-hyper-v.aspx

     

     

    Best Regards,

    Vincent Hu

     

    Monday, September 14, 2009 9:12 AM
  • Thank you very much for the responses.

    So, all I do is pick a down time for the client and turn off the Virtual Machine (Exchange).

    And wait..............

    Are there any indicators?  Or do we press the refresh key every hour and see if that AVHD file disapears?  

    Normally, when you merge in a file, it requires a tonne of disk space.  If we have a 200GB VHD file, and a 100GB AVHD file, we need 300GB's free for the merge to be completed. From my understanding, hyper-v creates another completely new VHD  file and merges the results together.  Once it is done, it deletes the AVHD and old VHD file and make the new merged VHD file the master file.

    Is this true?
    Monday, September 14, 2009 4:01 PM
  • Not a problem.  Yes there are indicators that it’s working as you will see the percentage in the Hyper-V Manager window. 

    As for Hyper-V making a new VHD when deleteing snapshots I don’t think that is the case with Hyper-V.  I just deleted a snapshot on my test Hyper-V server and I didn’t see any new VHD’s created. 

    Let me verify this again with a larger shapshot tree.  I have some MP’3 that I can use to fill up a disk rather quickly.


    Joseph Noga MCITP, MCSE, MCSA, MCTS CCNA,CCDA,CCVP Managing Consultant VSD Technologies
    Monday, September 14, 2009 7:50 PM
  • So I had a VM that was aprox 2GB in size took a snapshot and then added some MP3's.  So this gave me an AVHD file of 2.3GB in addition to the 2GB VHD..  I then deleted the snapshot tree and watched the disk spce on my data drive.  It didn't change more than 1GB during the merge process.  Once the merge was done it modified the original VHD file it didn't make a new one. 08



    Joseph Noga MCITP, MCSE, MCSA, MCTS CCNA,CCDA,CCVP Managing Consultant VSD Technologies
    Monday, September 14, 2009 9:50 PM
  • We have had the server off for 6 hours now.

    I will update later on this morning with the results.

    Thanks again for your assistance.

    Regarding the progress bar, there are no visible snapshots in the Hyper-V Manager, it is just the .avhd file in the snapshots directory with a recent modification date (and the name of the server in question).
    Tuesday, September 15, 2009 6:05 AM
  • Good morning,

    We gave the server just short of 13 hours with no change. I definitely did not see the "Merge in Progress" in Hyper-V Manager.

    It seems to not know that this .avhd file exists. Any ideas?

    Ben
    Tuesday, September 15, 2009 12:56 PM
  • Take a look in the VM config file if the snapshot is still there.  That will indicate if the VM is using the snapshot file.  Also do you have any antivirus on the host server?  If so are the snapshot and the vhd's excluded?
    Joe

     

     

     


    Joseph Noga MCITP, MCSE, MCSA, MCTS CCNA,CCDA,CCVP Managing Consultant VSD Technologies
    Tuesday, September 15, 2009 2:30 PM
  • Thanks for all of your help.

    Yes, the .xml file does refer to the snapshot in question.  There is nothing in hyper-v manager that says there are any snapshots for that VM.

    There is no AV running on the hyper-v.

    We had in the past had to merge in VHD files (several of them) during Snapshothell and those worked just fine.

    I checked the start date and it was created two months after Snapshothell even happened. 

    We do have a backup (wbadmin) that runs every night - and we are down to 17GB's of free disk space.  Could that be a possible problem?  One of the correlations is that this only happens at night and in the morning, we have to restart the servers.
    • Proposed as answer by Alpha11111 Wednesday, October 14, 2009 10:08 PM
    • Marked as answer by Vincent Hu Monday, November 23, 2009 8:00 AM
    Wednesday, September 16, 2009 4:34 PM
  • I actually had to contact MS tech support.

    There are such things as snapshots that do not appear in hyper-v.  Here is what I had to do:

    1) Backup my guest and snapshot
    2) Use Ghost to capture a complete image of the server to a Ghost Cast server
    3) Blow away my guest and snapshot (hopefully my backups will work if this fails)
    4) Create a new Virtual Machine, mount the Ghost Boot CD as an ISO
    5) Use Ghost Cast server to push all the data back
    6) Run the Integration Utility again get get the NICs to work
    a) this does cause another network adapter to be loaded since the origional is hidden and virutual and can't be removed easily.

    According to MS: This is a known problem with Hyper-V sometimes a snapshot just gets created and they don't know why.
    Wednesday, October 14, 2009 10:12 PM
  • We have had the same problem when we did physical to vitrual transfer of one of our servers by SCVMM 2008 R2 to Windows Server 2008 R2 host with Hyper-V role.
    I think that the reason was that when transfer was processing it was somewhat like "live" transfer with taking snapshot of the disk state before transferring and storing all data changes on this disk while transferring P2V to somewhat like "temp datastore'. Thus when all data from server disk snapshot were transferred to new VM .VHD disk, after that SCVMM 2008 R2 transferred to VM also all data changes made to Physical server while being transferring to Virtual Machine from "temp datastore" but SCVMM 2008 R2 transferred them not in created before .VHD but in new .AVHD disk. And if you do not shut down new VM after transferring P2V you'll not start your host Hyper-V to merge this .AVHD disk into .VHD disk and Hyper-V will continue to use that .AVHD file to store all data changes in our new started and working VM instead of using .VHD file. But if you start and after that shut down you newly created VM Hyper-V will immediately start to merge .AVHD into .VHD. By the way - you do not need much space for this to accure successfully - we had 65Gb .VHD, 35 Gb .AVHD and only 15 Gb of free space on all host machine disks and everything went just fine.

    Monday, November 23, 2009 4:56 AM
  • It would have worked with using Disk2Vhd from sysinternals or any other p2v tools at hand also
    http://technet.microsoft.com/nb-no/sysinternals/ee656415(en-us).aspx
    • Proposed as answer by z_lives Thursday, October 20, 2011 3:51 PM
    • Unproposed as answer by Denis Dyagilev Saturday, August 25, 2018 7:43 AM
    Thursday, March 4, 2010 6:06 PM
  • Hi OlavT,

    Will that work with a VM that has snapshots? Should the machine be down or can it be done while in operation? It would be an interesting solution.

    Wednesday, April 17, 2013 10:16 AM
  • I am having the exact same problem, my server has 2 VHD files one for the OS and one for exchange DB's, i stupidly created 3 snapshots and over time these ".avhd" files have chewed up all the remaining space on the server.

    I deleted the snapshosts from hyper-vI and tried as some people suggested powering off the Guest VM and monitoring the byte's read/written for any processes related and nothing much happening,  and booted the guest OS again still no change in the files, below is the AVHD sizes from the second hdd.

    I carried out the shutdown from the Guest OS should i have shut this down differently?

    The server is hosting  146 users and running in production, we have a big database failure just before the end of last year and the last thing i want to do is have to go through another rebuild......

    Wednesday, April 17, 2013 9:39 PM
  • I found a solution to the phantom AVHD files that worked for me.

    I have a 2008r2 Hyper-V host with a 2008r2 VM.  I have never created a snapshot of the problem VM, but needed to expand the VHD recently.  When I inspected closer I discovered that there was an AVHD file.  Shutting the VM down, rebooting the host, etc., etc. didn't ever get rid of the AVHD file (plenty of free space too).

    I have recently set up a new 2012 Hyper-V host.  Thought I would restore the server from backup to the new host, which in theory should create a new VHD file.  I didn't restore the VHD files.  I restored the server to a new VM on the new server.  The restore worked, but I again ended up with another phantom snapshot file!

    After poking around, I found an option to edit the disk that had the option to remove the differencing disk.  Performing this on the snapshot file finally worked (server booted up and no snapshot file!).

    Going back and looking at the old 2008r2 host, I see that there is the same option.  I'm not going to try it as my VM is already fixed and running.  But, on 2008r2 here are the steps that look like they will work:

    I assume the VM should be turned off (mine was).

    In Hyper-V Manager select Edit Disk...

    Browse to and select the AVHD file

    Select Merge

    Select either To the parent virtual hard disk, or to a new virtual hard disk, depending on what makes sense for you.

    After the merge is done the AVHD file should be gone.  You might need to edit the VM settings to point the disk to the proper VHD file (mine was pointing to the AVHD file).

    Tuesday, May 14, 2013 11:58 PM
  • Hi I have the same problem on a Hyper V 2012 system, it started merging the disk after I deleted the snap shot but I cancel this process (not my best idea I know) , can anyone tell me how I restart this process?

    I tried shutting down the VM but it did not auto restart!


    Regards Dagz

    Friday, September 20, 2013 9:18 AM
  • To me what worked was this (I had two drives, one vhd and one avhd for each drive. No snapshots showing in the history). W2008R2

    1) stop the VM

    2) create a snapshot (you will see another avhd file)

    3) delete the snapshot (the merge process will trigger)

    4) Merge completed and only the vhd were left

    Tuesday, January 28, 2014 2:05 AM
  • Great reply Pablo, worked like a charm

    Monday, June 30, 2014 8:19 PM
  • Please note the below was with AVHDX not AVHD files and may behave differently in your environment.

    I just ran into this issue - as an alternative resolution with no down time when you find an "orphaned" but live AVHDX file

    • Edit the settings of the affected VM
    • Change the Snapshot File Location to a new location ( I made sure there was space for a new snapshot in this location, but no data was created there in this case)
    • As soon as you apply the snapshot begins merging
    • In a few minutes the AVHDX file will be deleted and merged with the VHDX
    • Reset your Snapshot location to the original location

    • Edited by tommcgeown Thursday, August 7, 2014 2:46 AM
    • Proposed as answer by Kristian Eklund Monday, December 15, 2014 8:15 PM
    • Marked as answer by Denis Dyagilev Saturday, August 25, 2018 7:45 AM
    Thursday, August 7, 2014 2:45 AM
  • Dude, this was so useful!

    I had a large disk-to-disk backup vault on a 2012 r2 hyper-v host with  several terabyte disks that I filled trying to recover a VMware machine that got corrupted during a SAN disk failure.  Yea and all complicated with snapshots and checkpoints.

    The initial merge "completed" with a 19080 event... but the files were still there.

    So... thanks to Pablo I created and deleted a checkpoint and voila!  files are disappearing. 

    Yea so never create a checkpoint on a production server.  And don't even think about it on a backup vault.
    • Edited by swirch Wednesday, October 8, 2014 7:13 PM added clause about backup server.
    Wednesday, October 8, 2014 7:11 PM
  • Thank you very much tommcgeown!

    We had the exact same issue. However, us was due to a merging failed due to disk running full during the process. The merge failed and we had one vhdx and one larger avhdx. I just changed the Snapshot location to another drive with plenty of space and it continued to merge (started around 60-70% or so) and was done in a couple of minutes. No files created in the new snapshot location, just one large avhdx-file gone!

    Stand-alone Windows Server 2012 with Hyper-V role.

    /Kristian

    Monday, December 15, 2014 8:18 PM
  • Curious about Snapshots that seem to appear out of nowhere. We don't use snapshots at all but suddenly we have 6 Hyper V machines that all have snapshots. Like I said we do not use them so this is annoying at best, is there any known situation that will trigger the creation of a snapshot or anything that I can do to disable this feature?

    Thanks!

    Dave Moses

    Monday, September 25, 2017 10:17 PM
  • Curious about Snapshots that seem to appear out of nowhere. We don't use snapshots at all but suddenly we have 6 Hyper V machines that all have snapshots. Like I said we do not use them so this is annoying at best, is there any known situation that will trigger the creation of a snapshot or anything that I can do to disable this feature?

    Thanks!

    Dave Moses

    Yes backup products will create special snapshots called storage snapshots to do their job. Veeam has more than once left me with tons of these.
    Friday, April 13, 2018 7:38 PM