locked
Production Checkpoint Failure RRS feed

  • Question

  • Hyper-V Host is Server 2016, VM is Server 2016, when attempting a production checkpoint I get the following:

    'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

    Production Checkpoint for VM 'VM' failed in the guest VSS Integration Component with error Element not found. (0x80070490). If you continue to see this problem please use regular checkpoints instead (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

    I have two Server 2016 Hyper-V hosts, and each one Server 2016 VMs, on one host my backup software which requires production checkpoints for application aware processing works just fine, on the other I get this error.

    The difference is that prior to installing my backup software, on the host with the error I created a production checkpoint, then deleted the checkpoint subtree.  Even after removing my backup software I still cannot create a production checkpoint, I get the same error, on the host that works just fine, I never did the production checkpoint / subtree delete. 

    The production checkpoint / subtree delete seems to be what causes the issue, after testing on test VMs.

    All VSS writers on host and VM are stable and not in error, reboot of the VM and/or host do not correct the issue.

    Tuesday, July 30, 2019 4:40 PM

All replies

  • Hi,

    Good Day!

    1.Please try to uncheck the “Backup” in VM integration service.

    2.Please make sure that the integration service has already been update to the latest version.

    3.Can you create standard checkpoint?

    4.Please check if there is any error message in Event Viewer.

    Please let me know if you have any question.

    Cheers,

    Daniel


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, July 31, 2019 2:11 AM
  • Unchecking Backup in the integration services doesn't change anything (and wouldn't allow for application aware processing which I'll need).

    Both the VM and the Host have been updated fully with Windows Update. Since they are both Server 2016 that is the only way to update integration services that I'm aware of.

    I can create a standard checkpoint, I can also create a production checkpoint if the VM is powered off.  Unfortunately I'm going to require the ability to create production checkpoints while the VM is running for application aware processing for backups.

    There are no errors in the Event View on the VM, and the only relevant entries I can find on the host are in the Hyper-V Worker Admin log and they are:

    'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

    and

    Production Checkpoint for VM 'VM' failed in the guest VSS Integration Component with error Element not found. (0x80070490). If you continue to see this problem please use regular checkpoints instead (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

    All VSS writers on Host and VM are stable and not in an error state.

    Wednesday, July 31, 2019 6:19 AM
  • update:

    Please run this PowerShell command on your host and upload your screenshot :

    Get-VMIntegrationService -VMName {VMNAME} | where Name -eq VSS

    please run cmd command to check your integration service:

    REG QUERY "HKLM\Software\Microsoft\Virtual Machine\Auto" /v IntegrationServicesVersion

    https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-integration-services

    Regards,

    Daniel


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.




    Wednesday, July 31, 2019 7:21 AM
  • I can post screenshots when my account gets verified and I'm allowed, but just to get the answer for you:

    From the host:

    PS C:\Windows\system32> Get-VMIntegrationService -VMName CITG2K16 | where Name -eq VSS

    VMName   Name Enabled PrimaryStatusDescription SecondaryStatusDescription
    ------   ---- ------- ------------------------ --------------------------
    CITG2K16 VSS  True    OK

    From the VM:

    Windows PowerShell
    Copyright (C) 2016 Microsoft Corporation. All rights reserved.

    PS C:\Windows\system32> REG Query "HKLM\Software\Microsoft\Virtual Machine\Auto" /v IntegrationServicesVersion

    HKEY_LOCAL_MACHINE\Software\Microsoft\Virtual Machine\Auto
        IntegrationServicesVersion    REG_SZ    10.0.14393

    PS C:\Windows\system32>

    Wednesday, July 31, 2019 4:44 PM
  • It seems that integration services are the latest and VSS of VM is no problem, which is really strange.

    Can we use the current VHD to rebuild a VM to see if there are still problems?

    Regards,

    Daniel


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Thursday, August 1, 2019 1:40 AM
  • I shut down my original VM, create a new VM and attached my VHD to it and booted the VM.

    Unfortunately attempting a production checkpoint still produces the same error:

    Production Checkpoint for VM 'VM' failed in the guest VSS Integration Component with error Element not found. (0x80070490). If you continue to see this problem please use regular checkpoints instead (Virtual machine ID F5B0472F-0BF3-4C04-A1FE-C4109C226496)

    and

    'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID F5B0472F-0BF3-4C04-A1FE-C4109C226496)

    Thursday, August 1, 2019 5:56 AM
  • Something in here might be helpful:

    http://backupchain.com/i/hyper-v-backup-error-could-not-initiate-a-checkpoint-operation-element-not-found-0x80070490

    Regards,

    Daniel


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Friday, August 2, 2019 1:21 AM
  • I had found that page before I made my thread here, but, I repeated the steps there anyway.

    I redid the security settings, unchecked the Backup integration, powered down my VM, restarted my host, brought everything back up and then did a Windows backup,  after I did the Windows backup, I rechecked Backup integration and did another Windows backup.  The windows backup completed with warnings (which was the backup VSS integration), I also attempted another production checkpoint which failed.  The errors I got are:

    For the snapshot failure:
    'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

    Production Checkpoint for VM 'VM' failed in the guest VSS Integration Component with error Element not found. (0x80070490). If you continue to see this problem please use regular checkpoints instead (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)

    However, for the Windows backup with the backup component checked I got:

     Log Name:      Microsoft-Windows-Backup
    Source:        Microsoft-Windows-Backup
    Date:          8/2/2019 1:42:41 AM
    Event ID:      7
    Task Category: None
    Level:         Warning
    Keywords:      
    User:          SYSTEM
    Computer:      HOST.FQDN.FQDN.net
    Description:
    The backup operation that started at '‎2019‎-‎08‎-‎02T05:03:03.432289600Z' has completed with errors. Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Backup" Guid="{1DB28F2E-8F80-4027-8C5A-A11F7F10F62D}" />
        <EventID>7</EventID>
        <Version>3</Version>
        <Level>3</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x4000000000000000</Keywords>
        <TimeCreated SystemTime="2019-08-02T05:42:41.855150200Z" />
        <EventRecordID>9</EventRecordID>
        <Correlation />
        <Execution ProcessID="3128" ThreadID="2920" />
        <Channel>Microsoft-Windows-Backup</Channel>
        <Computer>Host.FQDN.FQDN.net</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
        <Data Name="BackupTemplateID">{E94040C2-0625-4561-8D68-E2935797283C}</Data>
        <Data Name="HRESULT">0x0</Data>
        <Data Name="DetailedHRESULT">0x0</Data>
        <Data Name="ErrorMessage">%%0</Data>
        <Data Name="BackupState">14</Data>
        <Data Name="BackupTime">2019-08-02T05:03:03.432289600Z</Data>
        <Data Name="BackupTarget">\\192.168.16.75\temp</Data>
        <Data Name="NumOfVolumes">5</Data>
        <Data Name="VolumesInfo">&lt;VolumeInfo&gt;&lt;VolumeInfoItem Name="Esp" OriginalAccessPath="" State="14" HResult="0" DetailedHResult="0" PreviousState="17" IsCritical="1" IsIncremental="0" BlockLevel="1" HasFiles="0" HasSystemState="0" IsCompacted="1" IsPruned="0" IsRecreateVhd="2" FullBackupReason="3" DataTransferred="100663296" NumUnreadableBytes="0" TotalSize="100663296" TotalNoOfFiles="0" Flags="523" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /&gt;&lt;VolumeInfoItem Name="C:" OriginalAccessPath="C:" State="14" HResult="0" DetailedHResult="0" PreviousState="9" IsCritical="1" IsIncremental="0" BlockLevel="1" HasFiles="0" HasSystemState="1" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="10" DataTransferred="20100939776" NumUnreadableBytes="0" TotalSize="20100939776" TotalNoOfFiles="0" Flags="5672" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /&gt;&lt;VolumeInfoItem Name="Recovery" OriginalAccessPath="" State="14" HResult="0" DetailedHResult="0" PreviousState="9" IsCritical="1" IsIncremental="0" BlockLevel="1" HasFiles="0" HasSystemState="1" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="10" DataTransferred="341049344" NumUnreadableBytes="0" TotalSize="341049344" TotalNoOfFiles="0" Flags="552" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /&gt;&lt;VolumeInfoItem Name="E:" OriginalAccessPath="E:" State="14" HResult="0" DetailedHResult="0" PreviousState="9" IsCritical="0" IsIncremental="0" BlockLevel="1" HasFiles="0" HasSystemState="0" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="10" DataTransferred="212068335616" NumUnreadableBytes="0" TotalSize="212068335616" TotalNoOfFiles="0" Flags="8" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /&gt;&lt;VolumeInfoItem Name="F:" OriginalAccessPath="F:" State="14" HResult="0" DetailedHResult="0" PreviousState="9" IsCritical="0" IsIncremental="1" BlockLevel="0" HasFiles="1" HasSystemState="0" IsCompacted="0" IsPruned="0" IsRecreateVhd="0" FullBackupReason="0" DataTransferred="4624396" NumUnreadableBytes="0" TotalSize="4628492" TotalNoOfFiles="15" Flags="6228" BackupTypeDetermined="1" SSBTotalNoOfFiles="0" SSBTotalSizeOnDisk="0" /&gt;&lt;/VolumeInfo&gt;</Data>
        <Data Name="SourceSnapStartTime">2019-08-02T05:03:02.103845600Z</Data>
        <Data Name="SourceSnapEndTime">2019-08-02T05:03:41.423493700Z</Data>
        <Data Name="PrepareBackupStartTime">&lt;TimesList&gt;&lt;Time Time="2019-08-02T05:05:12.154Z" /&gt;&lt;Time Time="2019-08-02T05:05:27.219Z" /&gt;&lt;Time Time="2019-08-02T05:08:38.071Z" /&gt;&lt;Time Time="2019-08-02T05:08:42.164Z" /&gt;&lt;Time Time="2019-08-02T05:42:21.429Z" /&gt;&lt;/TimesList&gt;</Data>
        <Data Name="PrepareBackupEndTime">&lt;TimesList&gt;&lt;Time Time="2019-08-02T05:05:12.154Z" /&gt;&lt;Time Time="2019-08-02T05:05:27.422Z" /&gt;&lt;Time Time="2019-08-02T05:08:38.071Z" /&gt;&lt;Time Time="2019-08-02T05:08:42.195Z" /&gt;&lt;Time Time="2019-08-02T05:42:22.273Z" /&gt;&lt;/TimesList&gt;</Data>
        <Data Name="BackupWriteStartTime">&lt;TimesList&gt;&lt;Time Time="2019-08-02T05:05:12.154Z" /&gt;&lt;Time Time="2019-08-02T05:05:27.422Z" /&gt;&lt;Time Time="2019-08-02T05:08:38.071Z" /&gt;&lt;Time Time="2019-08-02T05:08:42.195Z" /&gt;&lt;Time Time="2019-08-02T05:42:22.273Z" /&gt;&lt;/TimesList&gt;</Data>
        <Data Name="BackupWriteEndTime">&lt;TimesList&gt;&lt;Time Time="2019-08-02T05:05:18.960Z" /&gt;&lt;Time Time="2019-08-02T05:08:37.400Z" /&gt;&lt;Time Time="2019-08-02T05:08:41.742Z" /&gt;&lt;Time Time="2019-08-02T05:42:00.701Z" /&gt;&lt;Time Time="2019-08-02T05:42:22.617Z" /&gt;&lt;/TimesList&gt;</Data>
        <Data Name="TargetSnapStartTime">1601-01-01T00:00:00.000000000Z</Data>
        <Data Name="TargetSnapEndTime">1601-01-01T00:00:00.000000000Z</Data>
        <Data Name="DVDFormatStartTime">&lt;TimesList&gt;&lt;/TimesList&gt;</Data>
        <Data Name="DVDFormatEndTime">&lt;TimesList&gt;&lt;/TimesList&gt;</Data>
        <Data Name="MediaVerifyStartTime">&lt;TimesList&gt;&lt;/TimesList&gt;</Data>
        <Data Name="MediaVerifyEndTime">&lt;TimesList&gt;&lt;/TimesList&gt;</Data>
        <Data Name="BackupPreviousState">19</Data>
        <Data Name="ComponentStatus">&lt;ComponentStatus&gt;&lt;ComponentStatusItem Name="E397EC81-9B8C-435F-95C9-F9950638BF28" LogicalPath="" Caption="Online\VM" AppId="HyperV" WriterId="{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}" InstanceId="{f7cfb7ba-697d-4598-9375-110844a810e8}" ConsistencyCheckResult="-2139618955" ConsistencyCheckResultDetailed="-2147212300"/&gt;&lt;/ComponentStatus&gt;</Data>
        <Data Name="ComponentInfo">&lt;ComponentInfo&gt;&lt;ComponentInfoItem WriterId="{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}" InstanceId="{f7cfb7ba-697d-4598-9375-110844a810e8}" Name="E397EC81-9B8C-435F-95C9-F9950638BF28" LogicalPath="" Caption="Online\VM" AppName="HyperV" hResult="-2139618955" hDetailedResult="-2147212300" IsFullBackup="1" FullBackupReason="0" Size="0" DataTransferred="0" TotalNoOfFiles="0" /&gt;&lt;ComponentInfoItem WriterId="{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}" InstanceId="{f7cfb7ba-697d-4598-9375-110844a810e8}" Name="Host Component" LogicalPath="" Caption="Host Component" AppName="HyperV" hResult="0" hDetailedResult="0" IsFullBackup="1" FullBackupReason="0" Size="398475" DataTransferred="0" TotalNoOfFiles="0" /&gt;&lt;/ComponentInfo&gt;</Data>
        <Data Name="SSBEnumerateStartTime">1601-01-01T00:00:00.000000000Z</Data>
        <Data Name="SSBEnumerateEndTime">1601-01-01T00:00:00.000000000Z</Data>
        <Data Name="SSBVhdCreationStartTime">1601-01-01T00:00:00.000000000Z</Data>
        <Data Name="SSBVhdCreationEndTime">1601-01-01T00:00:00.000000000Z</Data>
        <Data Name="SSBBackupStartTime">1601-01-01T00:00:00.000000000Z</Data>
        <Data Name="SSBBackupEndTime">1601-01-01T00:00:00.000000000Z</Data>
        <Data Name="SystemStateBackup">&lt;SystemState IsPresent="1" HResult="0" DetailedHResult="0" /&gt;</Data>
        <Data Name="BMR">&lt;BMR IsPresent="1" HResult="0" DetailedHResult="0" /&gt;</Data>
        <Data Name="VssFullBackup">true</Data>
        <Data Name="UserInputBMR">true</Data>
        <Data Name="UserInputSSB">true</Data>
        <Data Name="BackupSuccessLogPath">C:\Windows\Logs\WindowsServerBackup\Backup-02-08-2019_05-03-03.log</Data>
        <Data Name="BackupFailureLogPath">C:\Windows\Logs\WindowsServerBackup\Backup_Error-02-08-2019_05-03-03.log</Data>
        <Data Name="EnumerateBackupStartTime">&lt;TimesList&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="2019-08-02T05:42:20.414Z" /&gt;&lt;/TimesList&gt;</Data>
        <Data Name="EnumerateBackupEndTime">&lt;TimesList&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="2019-08-02T05:42:21.429Z" /&gt;&lt;/TimesList&gt;</Data>
        <Data Name="PruneBackupStartTime">&lt;TimesList&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;/TimesList&gt;</Data>
        <Data Name="PruneBackupEndTime">&lt;TimesList&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;Time Time="1601-01-01T00:00:00.000Z" /&gt;&lt;/TimesList&gt;</Data>
        <Data Name="BackupFlags">0xc</Data>
        <Data Name="ComponentInfoSummary">&lt;ComponentInfoSummary ComponentInfoArrayPresent="1" TotalComponents="2" SucceededComponents="1" /&gt;</Data>
      </EventData>
    </Event>

    Also:
    Writer Microsoft Hyper-V VSS Writer experienced some error during snapshot creation.  More info: .

    Log Name:      Application
    Source:        SPP
    Date:          8/2/2019 1:03:38 AM
    Event ID:      16387
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      Host.FQDN.FQDN.net
    Description:
    Writer Microsoft Hyper-V VSS Writer experienced some error during snapshot creation.  More info: .
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="SPP" />
        <EventID Qualifiers="0">16387</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2019-08-02T05:03:38.942365800Z" />
        <EventRecordID>1924</EventRecordID>
        <Channel>Application</Channel>
        <Computer>Host.FQDN.FQDN.net</Computer>
        <Security />
      </System>
      <EventData>
        <Data>Microsoft Hyper-V VSS Writer</Data>
        <Data>
        </Data>
        <Data>The writer experienced a partial failure. Check the component level error state for more information. (0x80042336)</Data>
        <Data>Unspecified error (0x80004005)</Data>
        <Binary>01000000B8120000991200000000000042BEB7C511CAC619E59C92030000000000000000</Binary>
      </EventData>
    </Event>

    I also found one of my VSS writers in a stable state but with an error:

    Writer name: 'Microsoft Hyper-V VSS Writer'
       Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
       Writer Instance Id: {f7cfb7ba-697d-4598-9375-110844a810e8}
       State: [1] Stable
       Last error: Unexpected error

    Friday, August 2, 2019 6:41 AM