none
Best practice on free disk space for Hyper-V VHDX?

    Question

  • I have a large Hyper-V 2012 R2 host server with many FC RAID storage unit (SAN) that are physically connected to it. All storage unit are formatter in NTFS filesystem as well. They are all large capacity RAID, from 10 TB (the smaller) to 35 TB (the larger), total capacity is around 65 TB overall. Now on my Hyper-V host I have a few virtual servers (all 2012 R2 STD) and one is my main business file virtual server, on the Hyper-V HOST I did create a few VHDX volumes that are physically located on my RAID storages and that are attached to the virtual server (guest OS), according to Microsoft best practice for Hyper-V (no physical attach for a Virtual guest OS). So everything is working very fine and quick no problem. I did observe Microsoft recommendation on VHDX, as for my main "production" volume, I did a fixed size VHDX (15 TB) and for my archives volumes (low access), I leave them variable VHDX.

    Now I am getting a bit confused regarding Microsoft best practice for NTFS filesystem free space management... because in one hand I have de physical RAID disks that is NTFS and I have virtual VHDX that are NTFS too.

    During my initial setup (on a "test lab"), I created test VHDXs that fit 100% (well 99% technically) of the physical disk because I don't want to loose uses of my disk capacity... but when I wanted to use VEEAM backup to backup my VHDX, I hit a problem where Veeam can't use shadow copy because the lack of free "physical" space on the VHDX up-level. So I did resized my volume according to MS recommendation, which is 15% free space. So for example I have a 5 TB physical drive, hosting a 4.25 TB VHDX file. Now Veeam is working fine... but I now have a slight problem with a software that is running on the virtual file server and the manufacturer told me "hey, your volume is getting low and you are not following MS recommandation of 15% free space on NTFS volume".

    So now I am confused, I don't want to loose 30% of my physical storage on each of my volume, that doesn't make sense??? I have a 5 TB physical drive, hosting a 4.25 TB VHDX volume where I can exceed 3.60 TB of datas... so it goes from a 5 TB physical drive, to a 3.60 TB total capacity? Did I miss something here?

    Could you imagine on my 35 TB storage? A 35 TB physical storage, hosting a 29.75 TB VHDX where I can't exceed 25,30 TB capacity? Really?!

    Please help me clarifying the point of the free space management... thanks!

    Thursday, March 16, 2017 8:32 PM

All replies

  • Hi Sir,

    >>but when I wanted to use VEEAM backup to backup my VHDX, 

    This backup is working at hyper-v host level . 

    >>a software that is running on the virtual file server and the manufacturer told me "hey, your volume is getting low and you are not following MS recommandation of 15% free space on NTFS volume".

    This software is working at VM level .

    I can understand that both of them may need 15% free space .

    In your case , I'd suggest you to ignore/disable "software warning" in VM , only backup VM on Host Level .

    Or only backup VM inside the virtual machine .

     

    Best Regards,

    Elton


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Monday, March 20, 2017 2:31 AM
    Moderator
  • Hi Elton,

    Thank you very much for your answer! To be honest I was expecting way more reply for this basic question, did I published it on the right TechNet forum or not?

    I would like to discuss further about your answer, because here's what I was told by my software provider running on my virtual server on the VHDX virtual drive :

    Microsoft recommended a minimum of 15% free space and due to issues tracking free space on NTFS volumes, this drive may have actually run out of free space and corrupted the file system. Left unaddressed, this can result in the loss of the volume and all the data on it. Low disk space is known to cause filesystem corruption and the only way to make sure all issues are found is to run chkdsk in read-write mode (chkdsk /f), chkdsk in read-only mode does not detect corruption on NTFS Volume.

    (Please consider this answer is specific to my actual VHDX where it left around 5% free space)

    So since both on my physical RAID storage and virtual VHDX disk are using NTFS file system, they can get both corrupted because of tracking free space issues on NTFS...

    Because I repeat that both Veeam (on the physical host) and software provider (on virtual guess) told be that I need to have 15% free space on my NTFS volume, on both layer finally...

    However in Veeam I have an option where I can build shadow copy buffer on a different drive if the main drive is running out of free space, but this doesn't prevent any corruption on the NTFS filesystem itself, if information given by the provide are true.

    I work as IT manager for 16 years ago on very large infrastructure, but they were all "physical" infrastructures yet and I only begin to work with virtualization for 2 years now and this is still a bit confusing at some point between physical and virtual. Like the free space management.

    Thanks again!


    • Edited by SimmZ Monday, March 20, 2017 1:18 PM
    Monday, March 20, 2017 1:17 PM
  • Hi Sir,

    Yes , that is a recommended value of free disk space for backup .

    We may avoid "zero free space" happen (it is a potential issue causes file system crash).

     

    >>However in Veeam I have an option where I can build shadow copy buffer on a different drive if the main drive is running out of free space, but this doesn't prevent any corruption on the NTFS filesystem itself, if information given by the provide are true.

    That is a feature provided by VSS .

    AFAIK , if shadow copy location setting was changed to other drive , these shadows resided in original disk will be removed .

    Regarding that behavior , I'd suggest you contact with Veeam .

    In addition , I haven't seen "file system corruption" while lacking free disk space .

    Hope it is helpful to you .

    Best Regards,

    Elton


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Tuesday, March 21, 2017 10:08 AM
    Moderator
  • Hi SimmZ

    I work for BackupChain and the fact you are trying out Veeam prompted me to respond so you can consider other solutions (^_^).

    Now the 15% free disk space is just a recommendation.

    VSS is a skunk so it's not easy to deal with.

    Basically when you start a backup, VSS provides a 'snashot' kind of view of the disk. The changes that occur to the volume during the backup need to be buffered somewhere.

    The more blocks are being updated, the more space you need. The longer the backup takes, hence, it's more likely to use more blocks.

    The 15% is hence only a rule of thumb. Theoretically the VM could alter all of its blocks! So you could in theory use up 2x the disk space during backups!

    Now if there is no free disk space, simply configure VSS to use another disk.

    Use the command

    vssadmin resize shadowstorage /for=X: /on=Y: /maxsize=unbounded

    The above command will instruct VSS to use the Y drive to buffer altered blocks of X: for the duration of the backup.

    Problem solved....well almost. Ditch Veeam ;-)

    Wednesday, March 22, 2017 1:45 PM
  • Thank you both Elton and RunnerManHere! In conclusion what would you recommend in my situation? Would you recommend me to load-up my physical storages up to 99-100% with VHDX virtual volumes (files), then deal with VSS for backup on another drive with free space. Then deal with usual free space recommendations from inside the virtual volumes on guest OS, just like if the virtual OS were physical machines?

    If this is your recommendation, is there an concern of performance loss, stability loss or any data corruption risk by running physical disk to 99-100% full with 1 or 2 VHDX volumes inside?

    The point is we are a large business with critical operations (just like any business! LOL) and the point is we don't want to skimp on any of the "best practice" for security and performance, but when you manage ultra large volumes such as 35 Terabytes each, dealing with a loss of 15% (physical) + 15% (virtual) is a lot of space when you consider these high performance/capacity SAN storages worth more than 60K$ each... and where I was confused is when I receive recommandation from Veeam from a physical perspective, then after from my software provided from inside the virtual guest server perspective... both requesting for 15% free space from it own perspective...

    Thanks again folks!


    • Edited by SimmZ Wednesday, March 22, 2017 2:17 PM
    Wednesday, March 22, 2017 2:17 PM
  • Our recommendations that we give to all of our customers doing Hyper-V backup is that the 15% rule of thumb is a 'wise' one ;-)

    If money is tight you can do this:

    Simply add to each server a 'cheap' drive to do the VSS buffering. If each volume on the server is 35 TB, you need about 3-5TB for VSS for each of those giant volumes. Use the vssadmin command to direct all VSS activity to the cheap drive. If the drive ever fails, all that will happen is the backup will fail, that's all.

    It's not too much of a cost or a risk. Simply replace that drive when it wears out and that's all. No info will be lost because data is always written to the destination disk. The cheap drive would only hold the previous version of a block, which is only needed for the purposes of completing the backup cycle at hand. Once the backup finishes, the space on the cheap drive is cleared again.

    Inside the VMs you also need 15% free because the VM internally will also do buffering while being backed up.

    Generally speaking when buying and assigning storage you will need to take all of this into consideration since you have to have live backup these days. If servers were backed up offline you wouldn't any extra temp space

    Hope this helps!

    Wednesday, March 22, 2017 3:22 PM
  • Our recommendations that we give to all of our customers doing Hyper-V backup is that the 15% rule of thumb is a 'wise' one ;-)

    If money is tight you can do this:

    Simply add to each server a 'cheap' drive to do the VSS buffering. If each volume on the server is 35 TB, you need about 3-5TB for VSS for each of those giant volumes. Use the vssadmin command to direct all VSS activity to the cheap drive. If the drive ever fails, all that will happen is the backup will fail, that's all.

    It's not too much of a cost or a risk. Simply replace that drive when it wears out and that's all. No info will be lost because data is always written to the destination disk. The cheap drive would only hold the previous version of a block, which is only needed for the purposes of completing the backup cycle at hand. Once the backup finishes, the space on the cheap drive is cleared again.

    Inside the VMs you also need 15% free because the VM internally will also do buffering while being backed up.

    Generally speaking when buying and assigning storage you will need to take all of this into consideration since you have to have live backup these days. If servers were backed up offline you wouldn't any extra temp space

    Hope this helps!

    I definitely understand your point as I usually purchase any IT stuff in a perspective of expandability, but the thing that happen is when the storage as been originally purchased it was supposed to be physical storages connected to physical servers. But when we decided to go virtual infrastructure with Hyper-V because our actuals HP Proliant Server were WAY TOO huge and powerful to host a single server, what I understand is we loose that extra (second) 15% storage capacity on the physical layer by going virtual, which we would not care about with a physical server only. That's the point...

    That being said, in addition to my main very high performance storage (15K RPM) SAN storage, I have few other lower performance but large storage (7.2K RPM SAS) that are 15-20 TB each that I can use for VSS "buffer" if needed, but what I understand is I am still better to stick with the 15% (physical) + 15% (virtual) free space in the best of all worlds...

    Wednesday, March 22, 2017 3:50 PM
  • Hi Sir,

    >>but what I understand is I am still better to stick with the 15% (physical) + 15% (virtual) free space in the best of all worlds...

    That's the best option .

    If it is needed to consider the budget/cost of storage space you may consider only backup VM at host level and ignore the "warning" in guest OS.

    Best Regards,

    Elton


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Tuesday, March 28, 2017 9:29 AM
    Moderator