none
VSS error on backup with two Exchange 2016 in DAG "[VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]." RRS feed

  • Question

  • Hello everybody,

    this are my two exchange servers:

    EX1: Exchange 2016 CU13 Version 15.1 (Build 1779.2) on Windows Server 2016 Version 1607 (Build 14393.3326)
    EX2: Exchange 2016 CU13 Version 15.1 (Build 1779.2) on Windows Server 2016 Version 1607 (Build 14393.3326)

    Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                                  Length    Length                             State
    ----                                          ------          --------- ----------- --------------------   ------------
    EX2_DB1\EX2                                   Mounted         0         0                                  Healthy
    EX1_DB1\EX2                                   Healthy         0         0           29.11.2019 09:09:54    Healthy
    EX_DB1_10GB\EX2                               Healthy         0         0           29.11.2019 09:07:39    Healthy
    EX_PF_DB1\EX2                                 Mounted         0         0                                  Healthy

    Name                           Server          Recovery        ReplicationType
    ----                           ------          --------        ---------------
    EX1_DB1                        EX1             False           Remote
    EX2_DB1                        EX2             False           Remote
    EX_DB1_10GB                    EX1             False           Remote
    EX_PF_DB1                      EX2             False           Remote


    We use Veeam for backups. All servers are in one backup job. The error only occurs  intermittent on ex1.


    Backups with warning contain only this message:

    25.11.2019 21:19:23 :: VSS: Backup job failed.
    Cannot notify writers about the 'BACKUP FINISH' event.
    A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{f20775fd-7aba-42f8-92d3-b80a17a8ce74}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]. 

    When the backup starts "MSExchangeRepl" event id 2140 and 2026are logged:

    "MSExchangeRepl" event id 2140
    The Microsoft Exchange Replication service VSS Writer encountered an exception in function Microsoft::Exchange::Cluster::ReplicaVssWriter::CReplicaVssWriterInterop::PrepareSnapshot. HResult -2147467259. Exception Microsoft.Mapi.MapiExceptionBackupInProgress: MapiExceptionBackupInProgress: Unable execute Prepare on snapshot. (hr=0x80004005, ec=1293)
    Diagnostic context:
        Lid: 1494    ---- Remote Context Beg ----
        Lid: 33832   StoreEc: 0x50D     
        Lid: 42621   StoreEc: 0x50D     
        Lid: 10786   dwParam: 0x0        Msg: 15.01.1779.007:EX1:14f782c4-aadf-458e-bb8e-68553d8a0320
        Lid: 51578   Guid: 14f782c4-aadf-458e-bb8e-68553d8a0320
        Lid: 1750    ---- Remote Context End ----
        Lid: 59216  
        Lid: 34640   StoreEc: 0x50D     
        Lid: 57114  
        Lid: 40730   StoreEc: 0x50D     
       bei Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
       bei Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)
       bei Microsoft.Mapi.ExRpcAdmin.ProcessSnapshotOperation(Guid mdbGuid, SnapshotOperationCode operationCode, UInt32 flags)
       bei Microsoft.Exchange.Cluster.Replay.StoreRpcController.<>c__DisplayClass29_0.<SnapshotPrepare>b__0()
       bei Microsoft.Exchange.Cluster.Shared.SafeRefCountedTimeoutWrapper.<>c__DisplayClass21_0.<ProtectedCallWithTimeout>b__0()
       bei Microsoft.Exchange.Data.HA.InvokeWithTimeout.Invoke(Action invokableAction, Action foregroundAction, TimeSpan invokeTimeout, Boolean sendWatsonReportNoThrow, Object cancelEvent)
       bei Microsoft.Exchange.Cluster.Shared.SafeRefCountedTimeoutWrapper.ProtectedCallWithTimeout(String operationName, TimeSpan timeout, Action operation)
       bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.SnapshotPrepare(Guid dbGuid, UInt32 flags)
       bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.PrepareSnapshot().



    "MSExchangeRepl" event id 2034
    The Microsoft Exchange Replication service VSS Writer (Instance 71904d88-eff5-455b-ac6a-68f07fb8c20b) failed with error 80004005 when preparing for snapshot.

    When backup is complete we get this errors:

    "MSExchangeRepl" event id 2140
    The Microsoft Exchange Replication service VSS Writer encountered an exception in function Microsoft::Exchange::Cluster::ReplicaVssWriter::CReplicaVssWriterInterop::BackupCompleteTruncateLogsComponents. HResult -3. Exception Microsoft.Mapi.MapiExceptionCallFailed: MapiExceptionCallFailed: Unable execute Truncate on snapshot. (hr=0x80004005, ec=1295)
    Diagnostic context:
        Lid: 1494    ---- Remote Context Beg ----
        Lid: 63528   StoreEc: 0x50F     
        Lid: 42621   StoreEc: 0x50F     
        Lid: 10786   dwParam: 0x0        Msg: 15.01.1779.007:EX1:14f782c4-aadf-458e-bb8e-68553d8a0320
        Lid: 51578   Guid: 14f782c4-aadf-458e-bb8e-68553d8a0320
        Lid: 1750    ---- Remote Context End ----
        Lid: 59216  
        Lid: 34640   StoreEc: 0x50F     
        Lid: 57114  
        Lid: 40730   StoreEc: 0x50F     
       bei Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
       bei Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)
       bei Microsoft.Mapi.ExRpcAdmin.ProcessSnapshotOperation(Guid mdbGuid, SnapshotOperationCode operationCode, UInt32 flags)
       bei Microsoft.Exchange.Cluster.Replay.StoreRpcController.<>c__DisplayClass32_0.<SnapshotTruncateLogInstance>b__0()
       bei Microsoft.Exchange.Cluster.Shared.SafeRefCountedTimeoutWrapper.<>c__DisplayClass21_0.<ProtectedCallWithTimeout>b__0()
       bei Microsoft.Exchange.Data.HA.InvokeWithTimeout.Invoke(Action invokableAction, Action foregroundAction, TimeSpan invokeTimeout, Boolean sendWatsonReportNoThrow, Object cancelEvent)
       bei Microsoft.Exchange.Cluster.Shared.SafeRefCountedTimeoutWrapper.ProtectedCallWithTimeout(String operationName, TimeSpan timeout, Action operation)
       bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.SnapshotTruncateLogInstance(Guid dbGuid, UInt32 flags)
       bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.BackupCompleteTruncateLogsReplica(ReplayConfiguration replica, Guid vssIdCurrentSnapshotSetId, Boolean fTruncateLogs)
       bei Microsoft.Exchange.Cluster.ReplicaVssWriter.CReplicaVssWriterInterop.BackupCompleteTruncateLogsComponents(IVssWriterComponents* pComponents, Guid vssIdCurrentSnapshotSetId, Boolean fTruncateLogs).


    "MSExchangeRepl" event id 2140
    The Microsoft Exchange Replication service VSS Writer (Instance 71904d88-eff5-455b-ac6a-68f07fb8c20b) failed with error FFFFFFFD when processing the backup completion event.


    "VSS" 8229

    Von einem VSS Writer wurde ein Ereignis mit dem Fehler "0x800423f3, Für den Generator wurde ein vorübergehender Fehler festgestellt. Wenn der Sicherungsvorgang wiederholt wird, tritt der Fehler möglicherweise nicht erneut auf." zurückgewiesen. Änderungen, die während der Verarbeitung des Ereignisses an den Writerkomponenten vorgenommen wurden, sind für den Anforderer nicht verfügbar. Zugehörige Ereignisse der Hostanwendung des VSS Writer finden Sie im Ereignisprotokoll. 

    Vorgang:
       BackupComplete-Ereignis

    Kontext:
       Ausführungskontext: Writer
       Generatorklassen-ID: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
       Generatorname: Microsoft Exchange Writer
       Generatorinstanzname: Exchange Replication Service
       Generatorinstanz-ID: {f20775fd-7aba-42f8-92d3-b80a17a8ce74}
       Befehlszeile: "C:\Program Files\Microsoft\Exchange Server\V15\Bin\msexchangerepl.exe"
       Prozess-ID: 12984

    I hope we can solve this prorblem. Thanks in advance

    Friday, November 29, 2019 8:40 AM

All replies

  • Check following or contact Veeam support,

    https://forums.veeam.com/veeam-backup-replication-f2/upgrade-to-v9-and-exchange-2013-dag-backup-warning-t32764.html

    You can run below command in cmd and check the status,

    vssadmin list writers

    Asper the output, check relevant service status. Then you can manually restart those services,

    VSS Writer Process (.exe) Service
    ASR Writer VSS Volume Shadow Copy
    BITS Writer BITS Background Intelligent Transfer Service
    COM+ REGDB Writer VSS Volume Shadow Copy
    DFS Replication Service Writer DFSR DFS Replication
    DHCP Jet Writer DHCPServer DHCP Server
    FRS Writer NtFrs File Replication
    FRSM Writer srmsvc File Server Resource Manager
    IIS Config Writer AppHostSvc Application Host Helper Service
    IIS Metabase Writer IISADMIN IIS Admin Service
    Microsoft Exchange Replica Writer MSExchangeRepl Microsoft Exchange Replication Service
    Microsoft Exchange Writer MSExchangeIS Microsoft Exchange Information Store
    Microsoft Hyper-V VSS Writer vmms Hyper-V Virtual Machine Management
    MSMQ Writer MSMQ Message Queuing
    MSSearch Service Writer WSearch Windows Search
    NTDS NTDS Active Directory Domain Services
    O_Search VSS Writer OSearch Office SharePoint Server Search
    O_Search 14 VSS Writer OSearch14 SharePoint Server Search 14
    Registry Writer VSS Volume Shadow Copy
    Shadow Copy Optimization Writer VSS Volume Shadow Copy
    SMS Writer SMSSITEVSSWRITER SMS_SITE_VSS_WRITER
    SPSearch VSS Writer SPSearch Windows SharePoint Services Search
    SPSearch4 VSS Writer SPSearch4 SharePoint Foundation Search V4
    SqlServerWriter SQLWriter SQL Server VSS Writer
    System Writer CryptSvc Cryptographic Services
    TermServLicensing TermServLicensing Remote Desktop Licensing
    WIDWriter WIDWriter Windows Internal Database VSS Writer
    WINS Jet Writer WINS Windows Internet Name Service (WINS)
    WMI Writer Winmgmt Windows Management Instrumentation

    Friday, November 29, 2019 8:47 AM
  • I know this post. But since there are no failover errors during backup, this imo not applicable.

    You think this is a 100% veeam error?

    C:\Windows\system32>vssadmin list writers
    vssadmin 1.1 - Verwaltungsbefehlszeilenprogramm des Volumeschattenkopie-Dienstes
    (C) Copyright 2001-2013 Microsoft Corp.

    Verfassername: "Task Scheduler Writer"
       Verfasserkennung: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
       Verfasserinstanzkennung: {1bddd48e-5052-49db-9b07-b96f96727e6b}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "VSS Metadata Store Writer"
       Verfasserkennung: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
       Verfasserinstanzkennung: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "Performance Counters Writer"
       Verfasserkennung: {0bada1de-01a9-4625-8278-69e735f39dd2}
       Verfasserinstanzkennung: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "System Writer"
       Verfasserkennung: {e8132975-6f93-4464-a53e-1050253ae220}
       Verfasserinstanzkennung: {1fc55e1f-12ed-492d-90ba-96358805b022}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "Cluster Shared Volume VSS Writer"
       Verfasserkennung: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
       Verfasserinstanzkennung: {d3e45ba6-faae-4044-97ad-9d266bb00699}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "ASR Writer"
       Verfasserkennung: {be000cbe-11fe-4426-9c58-531aa6355fc4}
       Verfasserinstanzkennung: {c8093753-b126-4266-89a4-fd6104d9da2a}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "Shadow Copy Optimization Writer"
       Verfasserkennung: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
       Verfasserinstanzkennung: {695cc125-b0ef-4ee2-acb2-c7f8a3975868}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "Microsoft Exchange Writer"
       Verfasserkennung: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
       Verfasserinstanzkennung: {f20775fd-7aba-42f8-92d3-b80a17a8ce74}
       Status: [1] Stabil
       Letzter Fehler: Wiederholbarer Fehler

    Verfassername: "Registry Writer"
       Verfasserkennung: {afbab4a2-367d-4d15-a586-71dbb18f8485}
       Verfasserinstanzkennung: {90470a2e-88bb-414b-9282-eabd691f96a8}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "COM+ REGDB Writer"
       Verfasserkennung: {542da469-d3e1-473c-9f4f-7847f01fc64f}
       Verfasserinstanzkennung: {c19d4967-5491-4c8a-97ad-44040e757ab8}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "BITS Writer"
       Verfasserkennung: {4969d978-be47-48b0-b100-f328f07ac1e0}
       Verfasserinstanzkennung: {c360818c-7434-4471-9edd-f5e7faa623be}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "WMI Writer"
       Verfasserkennung: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
       Verfasserinstanzkennung: {5904cc4f-3314-4822-9ed3-b7010cb30262}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "MSMQ Writer (MSMQ)"
       Verfasserkennung: {7e47b561-971a-46e6-96b9-696eeaa53b2a}
       Verfasserinstanzkennung: {9009411b-6995-4176-9bef-c6862c9f4696}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "IIS Metabase Writer"
       Verfasserkennung: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
       Verfasserinstanzkennung: {49435770-cccf-426f-99aa-2a40d523f7c9}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "IIS Config Writer"
       Verfasserkennung: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
       Verfasserinstanzkennung: {a16ba553-459b-476e-aac8-2a051141abbe}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Verfassername: "Cluster Database"
       Verfasserkennung: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
       Verfasserinstanzkennung: {836e6198-20fa-4803-acda-8fe2b38533ff}
       Status: [1] Stabil
       Letzter Fehler: Kein Fehler

    Sorry. I forgot this info. 



    • Edited by kschueler Friday, November 29, 2019 9:02 AM
    Friday, November 29, 2019 8:49 AM
  • Hi kschueler,

    The issue may be caused by a conflict between the VMware Tools and certain backup agents. The solution for us is to make sure the VMware Tools are installed *before* the backup agent. This also means when updating the Tools we need to uninstall the backup agent, update Tools, then reinstall the agent.

    Here's a possible resolution for you:

    Go to Task Manager, locate the task "ShadowCopyVolume(...)" that shows an error at latest status. Right click and choose delete. See if the issue persists.

    Regards,

    Beverly Gao


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    Monday, December 2, 2019 9:56 AM
  • Hi,
    I am writing here to confirm with you how the thing going now?
    If you need further help, please provide more detailed information, so that we can give more appropriate suggestions.

    Regards,

    Beverly Gao


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    Thursday, December 5, 2019 1:22 AM
  • Like Udara Kaushalya reccomended, I contacted veeam support for additional help. The supporter told me, that I need to put the exchange backup into an extra job. Since three backups no errors.

    The error was caused by a vss timeout. Since my system is during backup under heavy load, vss could not release vss snapshot fast enough, so the backup got a warning. This is the explanation so far.

    Thursday, December 5, 2019 7:20 AM
  • Hi kschueler,

    It seems the issue has been resolved. You can mark your reply as answer, this will make answer searching in the forum easier and be beneficial to other community members as well.

    Regards,

    Beverly Gao


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    Tuesday, December 10, 2019 9:53 AM