none
Backup Fails on Hyper-V RRS feed

  • Question

  • Hi friends,

    We've a 2012 R2 Cluster where so many VMs are running from. We're facing backup failure issues on random VMs. The VMs are from 2012 and 2016 operating system versions. We've tried verifying the shadow copies, services, WMI services, rebuilding WMI repository and moving the running WMI files from a running VM to the problematic VMs. Also, we've succeeded in taking SystemState backup using WBADMIN. The only problem is that the full backup fails only when it is trigerred from Veritas. 
    When we take file/folder/drive level backups using veritas, it gets succeeded too. Not sure, how to proceed further. 
    We've already done below stuffs.
    * Failing over the VMs to different hosts.
    * Migrating the VMs to other CSVs where the backup gets succeeded.
    * The CSVs have enough space.
    * ICS is up to date.
    * Firmware is up to date.
    * The VMs have no additional or unallocated space.
    Monday, October 8, 2018 2:40 PM

All replies

  • Hi,

    Thanks for your question.

    May I ask more information about this issue so that we can narrow down it.

    1 When failover to a healthy node, the fullbackup of the VMs can proceed properly?

    2 ICS is up to date.>>>What is ICS?

    3 The VMs have no additional or unallocated space.>> DO you mean that there’s no unallocated space or offline disks on the disk management of the VMs?

    4 Any error in the cluster event or event viewer?

    5 And what’s status of the CSV? Right click the CSV resource, can see any critical event?

    6 Due to the question is also related to third party, it is recommended to consult the Veritas for advisement about their products' compatibility with windows cluster and Hyper-V.  

    If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, October 9, 2018 2:33 AM
    Moderator
  • Hi,

    How are things going on?

    Please feel free to let us know if you need further assistance.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com


    Wednesday, October 10, 2018 9:59 AM
    Moderator
  • Hi Michael,

    * I'm referring to Hyper-V Integration Services.
    * I'm referring to the CSVs that have enough space where this VM runs from. Also, the logical drives of the VMs have enough free space in them. 
    * The cluster events say sometimes "The virtual machine 'VM001' is using processor-specific features not supported on physical computer 'Hyper-V Host 01' or "Not enough memory in the system to start the virtual machine VM001 with ram size 8192 megabytes. (Virtual machine ID 123456789)". But in actual, the Hyper-V hosts have enough space and the CPU compatibility is enabled. 
    * When we take SystemState backup from Windows Backup Feature or WBADMIN, it gets succeeded.
    * Veritas has already confirmed that the issue isn't from their side as the issue happens with most of VMs in the environment. But, we need a concrete justification to confirm that the issue isn't with the operating system/cluster as the SystemState backups get succeeded. 
    * CSVs critical events say "Cluster Shared Volume '' ('CSV001') has entered a paused state because of '(c00000be)'. All I/O will temporarily be queued until a path to the volume is reestablished." This issue occurs very frequently. But, the backup gets succeeded for other VMs even when this error message occurs. 
    Friday, October 12, 2018 6:23 AM
  • Hi Michael,

    Any update on this.. Very curious to know :)

    Monday, October 15, 2018 10:55 AM
  • Hi,

    I've got your message, and I'm trying to involve someone together to further look at this issue. I'll be back to update here in one day. 

    Appreciate your kind patience.

    Please also feel free to let me know if you have any update, I'm standing by with you.

    Thanks for your understanding and support.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, October 16, 2018 12:28 PM
    Moderator
  • Thanks Michael.

    Usually, the Backup application sends a request before and after snapshot creation to VSS provider. Once the snapshot is generated, the Backup application should start backing up the file from the read only disk and merge the .AVHDX file with it once the backup is succeeded. Here, I would like to know when exactly the backup fails. 

    * Is it during snapshot creation?
    * Is it during file backing up?
    * Is it during files merge?
    * Is it during requests transmission to VSS?
    * Also, would like to confirm if VSS service is responsible for VHD and AVHD files merger in a Backup application environment. 

    Please let me know if there is a way to find this out other than checking the Event Viewer. I've not found any events related to it. Please assist. 


    • Edited by Mushthaq Wednesday, October 17, 2018 9:17 AM
    Wednesday, October 17, 2018 9:10 AM
  • Hi guys,

    Any update on this. 

    Monday, October 29, 2018 7:45 AM
  • I think I'm close to the issue now.

    Any one has any idea how does the Checkpoints merge happen? Need core level algorithm. 

    VSS Provider is responsible for Snapshot creation. Like this, would like to know which components/service or other stuff is responsible for merging/deleting/applying Checkpoints.

    Please help. 

    Tuesday, October 30, 2018 12:12 PM
  • Hi,

    Sorry for my delay.

    Based on this research (https://www.altaro.com/hyper-v/hyper-v-manager-2/ ),

    Hyper-V Manager is the most common tool used to interact with Hyper-V. It has these primary features:

    • Configure Hyper-V feature settings, such as Live Migration
    • Create, destroy, and configure Hyper-V virtual machines
    • Create, destroy, and manage virtual hard disks
    • Direct control of virtual machine functions; for example: stop, start, and save
    • Create, delete, apply, and revert checkpoints
    • Configure and control Hyper-V Replication

    Additional, Virtual Machine Management Service, also known as VMMS or vmms.exe, is the main module in Microsoft Windows operating system that controls all aspects of Hyper-V server virtualization.

    Virtual Machine Management Service uses the submodules for virtual machine deployment and management. Virtual Machine Management Service uses several submodules, including Windows Management Instrumentation Provider; Hyper-V Volume Shadow Copy Service Writer; Virtual Machine, Worker Process and Snapshot Managers; Single Port Listener for Remote Desktop Protocol and Cluster Resource Control.

    Furthermore, Hyper-V Volume Shadow Copy Service Writer & Snapshot Managers submodules are responsible for Hyper-V checkpoints operation.

    For this details, we can refer to this article,

    https://searchservervirtualization.techtarget.com/definition/Virtual-Machine-Management-Service

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, October 30, 2018 2:22 PM
    Moderator
  • Hi Michael,

    The issue persists in the environment. 

    The VMs are still failing for backup. 

    * We're able to take file level backup using Veritas.
    * We're able to take SystemState backup using Windows Backup Feature (Have tried throgh WBAdmin as well)
    * Whenever the backup is initiated, the VSS writers are going to failed or error state. When we try restarting the services, they're going back normal.
    * On both Hyper-V host and VM, the writers are going unstable and thus, the backups fail. 
    * Have tried rebuilding WMI repository, restarting WMI, VSS, Cryptographic services. 

    Still the issue is existing. Please help. 

    Tuesday, December 18, 2018 1:50 PM
  • Any update on this?
    Thursday, December 20, 2018 9:48 AM
  • Hi,

    Let's concentrate on the backup failure first. For now, you use Windows backup server to backup host and VMs, but VMs backup fails, whereas the system state backup can work, is that right?

    Please check if there's any error regarding backup in the event viewer. 

    Location to Applications and services logs> Microsoft> Windows> Backup

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Thursday, December 20, 2018 12:28 PM
    Moderator
  • Hi Michael,

    We're already taking file level backups of the servers. 

    As I  mentioned, there are no specific logs for this backup failure. The VSS writers go to an error state when the backups are initiated by Veritas.

    Tuesday, December 25, 2018 11:42 AM