locked
November 2014 Update Rollup - KB3000853 - VSS Backups Broken RRS feed

  • Question

  • A number of our Windows 2012 Servers installed KB3000853 over the last two days.

    VSS backups for applications, such as Exchange 2013, Lync 2013, SQL 2012, hosted on 2012 servers are no longer working.

    We get errors such as:

    SQL:

    EventId 12366 - VSS - An unhandled exception was encountered while processing a VSS writer event callback method. The VSS writer infrastructure is in an unstable state. Restart the service or application that hosts the writer.

     Writer name:            SqlServerWriter
     Writer id:              {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
     Writer instance:        {035a4544-2f3f-49a7-8760-0752e40aaa26}
     Process command line:   "C:\Program Files\Microsoft SQL Server\90\Shared\sqlwriter.exe"
     Process ID:             1428
     Writer operation:       1014
     Writer state:           1
     Exception code:         0xe06d7363
     Exception location:     02

    EventId 12301 - VSS - Volume Shadow Copy Service error: Writer SqlServerWriter did not respond to a GatherWriterStatus call.

    Operation:
       Gather writers' status
       Executing Asynchronous Operation

    Context:
       Current State: GatherWriterStatus

    The SQL VSS writer disappears from the list of writers (vssadmin list writers) after a backup has been attempted.  Restarting the SQL VSS Writer service puts it back in the list of writers but it will fail again on the next backup.

    Exchange:

    EventId 2034 - MSExchangeRepl - The Microsoft Exchange Replication service VSS Writer (Instance 1a4e455f-c584-4544-b01c-62387e1a1945) failed with error FFFFFFFD when processing the backup completion event.

    EventId 2140 - MSExchangeRepl -  The Microsoft Exchange Replication service VSS Writer encountered an exception in function Microsoft::Exchange::Cluster::ReplicaVssWriter::CReplicaVssWriterInterop::BackupCompleteTruncateLogsComponents. HResult -3. Exception Microsoft.Mapi.MapiExceptionCallFailed: MapiExceptionCallFailed: Unable execute Truncate on snapshot. (hr=0x80004005, ec=1295)

    Exchange VSS writer is in an error state after the backup is attempted.  Restarting the VSS servers and Exchange Replication services bring the VSS writer into a healthy state, but backups still fail.

    Restarting the servers doesn't help.

    VSSSVC.exe on the servers was updated by KB3000853 to version 6.2.9200.17128.  We believe this is the source of the problems at the moment, although we haven't had a chance to remove the update yet.

    We have opened a case with PSS.


    • Edited by DJL Thursday, November 13, 2014 12:26 PM
    Thursday, November 13, 2014 12:25 PM

Answers

  • Please see http://blogs.msdn.com/b/taylorb/archive/2014/11/19/backup-jobs-failing-post-nov-rollup-install.aspx  where Taylor Brown (PM at Hyper-V Product Group) provides a workaround.

    Hope this helps,

    Didier

    Wednesday, November 19, 2014 10:20 PM
  • The KB 3000853 package was updated today:

    KB3000853 was re-released on November 25, 2014 with the correct version of KB2996928. This update can be obtained from Windows Update or Microsoft Download Center. 

    Tuesday, November 25, 2014 7:52 PM

All replies

  • We're having the same issues. I was able to temporarily fix the SQL errors and get a proper backup by updating the logon account for the SQL Server VSS Writer service to a domain admin account, but I would rather it be on its default as a local service.

    I have not been able to fix the Exchange error.


    • Edited by Cypher131 Thursday, November 13, 2014 4:05 PM
    Thursday, November 13, 2014 4:01 PM
  • Cypher131 - many thanks for the info! That's worked on our SQL server too! As you say not a great long term solution but it's great to have backups working again.

    We've just managed to backup Exchange 2013. We ran the following diskshadow commands to create a shadow simply to flush the log files as we were going to run out of space over night, but then DPM began backing up again!

    diskshadow
    add volume d:
    add volume e:
    begin backup
    create
    end backup

    Volumes d: and e: are our database and logs drives respectively



    • Edited by DJL Thursday, November 13, 2014 7:45 PM
    Thursday, November 13, 2014 4:20 PM
  • Uninstalling the update fixed the issue as well. In my opinion - better temporary solution than giving service Domain Admin privileges


    Janis Berzins / Senior Infrastructure Solutions Consultant

    Friday, November 14, 2014 7:09 AM
  • Any news from PSS? We appear to have the same issue with Exchange. Thanks.
    Saturday, November 15, 2014 5:14 PM
  • I searched for KB3000853 and here is the link:

    November 2014 update rollup for Windows RT, Windows 8, and Windows Server 2012
    http://support.microsoft.com/kb/3000853/en-us

    It contains several updates/hotfixes and the following one specifically related to VSS Backup:

    Backup task fails with a time-out error in Windows Server 2012 or Windows Server 2008 R2
    http://support.microsoft.com/kb/2996928

    However currently there is no report about either KB3000853 or KB2996928 could cause VSS writers issue. 

    In current situation, run VSS writer as domain admin should be a quick workaround until a solution is published.

    And if you decided to uninstall the rollup update, since it is for fixing following issues, you can also choose to install the updates below separately if specific issues occurs on your server or client system:

    3004905 Windows Hyper-V improvement for Linux virtual machines that have file systems that are larger than 2 TB
    3004542 Windows Server 2012-based cluster responds to requests slowly
    3004540 Memory usage becomes high when you run an application in Windows 8 or Windows Server 2012
    2995478 Wmiprvse.exe memory leak when a program queries disk or partition storage information in Windows Server 2012
    2970215 Microcode update for Intel processors to improve the reliability of Windows Server
    2891362 A file copy operation fails when files or folders have long paths in Windows Explorer
    3002653 Update to support AAC and LATM format audios in Windows 8.1 or Windows 8
    2996928 Backup task fails with a time-out error in Windows Server 2012 or Windows Server 2008 R2

    I'll log this issue down and report it. If there is any feedback I'll update this thread as well.

    Thank you all for your time to report this issue!


    If you have any feedback on our support, please send to tnfsl@microsoft.com.

    • Proposed as answer by NinoRCTN Thursday, November 20, 2014 6:28 AM
    Monday, November 17, 2014 5:13 AM
  • Hi,

    have same problem, several servers (SQL, file and Exchange 2013), uninstalling the KB3000853 solve it.

    (These servers have GPT-disks)

    Regards, J

    Monday, November 17, 2014 9:22 AM
  • Same problem here. One interesting thing though is that it is not all of the databases that are failing.

    Monday, November 17, 2014 1:38 PM
  • I had the same problem with Windows 2012 and Exchange 2013. Backup complete but logs not truncated. After several hours of troubleshooting I removed KB3000853 and Backup was fully functional with logs getting truncated.
    Monday, November 17, 2014 2:11 PM
  • PSS are monitoring this thread and are aware of the problem.  If you're having the same problem please join the thread. 

    We have been advised to uninstall KB3000853 and wait for a fix.



    • Edited by DJL Monday, November 17, 2014 2:35 PM
    Monday, November 17, 2014 2:11 PM
  • I got the Exchange writer error using Windows backup and BackupAssist. Removing the KB resolve the problem.
    Monday, November 17, 2014 2:22 PM
  • We have the same issue here. In addition to the Exchange writer, this is also affecting our Backup Exec (I know...) resulting in "failed" backups because the writer became unresponsive.
    Monday, November 17, 2014 3:46 PM
  • We had the exact same issue on one of our DPM2012 SP1 servers and uninstalling KB3000853 solved the problem immediately. 

    Another DPM2012 SP1 server we have that hadn't installed KB3000853 did not experience this. 


    • Edited by DMB_UK Monday, November 17, 2014 4:54 PM
    Monday, November 17, 2014 4:53 PM
  • Good evening

    We have also had this problem since installing the same update last week. Setting the 'SQL Server VSS Writer' service to a domain admin account does not seem to have resolved the issue.

    I will try uninstalling the update next and will report back on whether that resolves the problem.

    Thanks

    Monday, November 17, 2014 5:34 PM
  • We had the exact same issue on one of our DPM2012 SP1 servers and uninstalling KB3000853 solved the problem immediately. 

    Another DPM2012 SP1 server we have that hadn't installed KB3000853 did not experience this. 


    I would suggest that you didn't have the error on bother DPM servers because they are perhaps protecting differing data sources? Not all the protection jobs failed for all data sources on the single DPM server that I run. Some jobs failed and some did not. Specifically Exchange Mailbox database related jobs and SQL database related jobs failed while system state protection jobs did not (the system state backup job I run on an SQL server where the protection job for the databases it hosts failed, ran successfully).

    I hope that makes sense!!

    Monday, November 17, 2014 5:38 PM
  • Since last Wednesday, my Veaam backup job has been failing, on all VMs spread across three VM hosts, with the error "Unable to allocate processing resources. Errors: Failed to create snapshot for vm..."

    I have been working with Veeam to no avail.

    Then I saw this thread. I uninstalled KB3000853 from one of my failing VM hosts and the backup completed successfully for the first time since prior to November's Windows Update release.

    Monday, November 17, 2014 8:27 PM
  • Hello,

    Same error here with 5 SQL-Servers.
    On some of them all dbs had the error and on some only a few dbs.

    I tried both workarounds (deinstallation and dom-admin for sql-vss-writer) with success.

    greetings from germany,

    torsten

    Tuesday, November 18, 2014 12:36 PM
  • Hi,

    my secondary DPM couldn't backup the database of my primary DPM anymore. SQL VSS Error 12366

    Uninstalling KB3000853 resolved the issue.


    This posting is provided AS IS with no warranties.

    Tuesday, November 18, 2014 1:21 PM
  • Has anyone managed to get this working on Exchange by changing a service login account? I switched both MS Shadow Copy Provider and MS Exchange Server Extension for Windows Server Backup to a domain admin account but am still not seeing logs being flushed. Thanks.
    Tuesday, November 18, 2014 9:33 PM
  • I have a couple of 2012R2 servers that are having the same issue and they don't have KB3000853 installed. Anyone else have this problem too?
    Tuesday, November 18, 2014 9:34 PM
  • I uninstalled the KB and things are working fine again. Hopefully there will be a hotfix or other revision shortly.
    Wednesday, November 19, 2014 2:17 AM
  • Just adding our voice to this as requested, we have experienced similar issues with backing up Exchange 2010 on Windows Server 2012 with Backup Exec, uninstalling this update resolved our problem.

    Wednesday, November 19, 2014 11:24 AM
  • Here's another "Me too".  I've had one Exchange 2013 cu6 DB (two servers in a DAG, Barracuda LBs in front) that started having backup issues over the weekend. After restarts of both the Exchange server and the DPM 2012 R2 backup server not working to fix it, by yesterday I had 3 DBS (out of 12) that would no longer back up nor complete their consistency checks.

    I was sent this link by an associate, and after uninstalling KB3000853 this morning, backups are now working again.

    In DPM 2012 R2, the error message is displayed as follows:

    Recovery point creation jobs for Exchange Mailbox Database DATABASENAME on SERVERNAME.my.domain have been failing. The number of failed recovery point creation jobs = 1.
     If the data source protected has some dependent data sources (like a SharePoint Farm), then click on the Error Details to view the list of dependent data sources for which recovery point creation failed. (ID 3114)

    DPM encountered a retryable VSS error. (ID 30112 Details: VssError:The writer's timeout expired between the Freeze and Thaw events.
     (0x800423F2))

    Wednesday, November 19, 2014 6:01 PM
  • Please see http://blogs.msdn.com/b/taylorb/archive/2014/11/19/backup-jobs-failing-post-nov-rollup-install.aspx  where Taylor Brown (PM at Hyper-V Product Group) provides a workaround.

    Hope this helps,

    Didier

    Wednesday, November 19, 2014 10:20 PM
  • Thank you Didier for updating the blog.

    As the blog mentioned, Microsoft already knows the issue and please apply KB2996928 as a workaround until an official solution is published. 

    http://support.microsoft.com/kb/2996928/


    If you have any feedback on our support, please send to tnfsl@microsoft.com.

    • Proposed as answer by EugeneLeitanMVP Thursday, November 20, 2014 6:14 AM
    Thursday, November 20, 2014 5:27 AM
  • I had the same problem with Windows 2012 and Exchange 2010.  KB3000853

    Have a nice day !!!

    Thursday, November 20, 2014 6:11 AM
  • Hi guys

    I can confirm that installing KB2996928 on affected servers resolves the issue without the need to uninstall KB3000853.

    http://support.microsoft.com/kb/3000853

    http://support.microsoft.com/kb/2996928/

    Regards

    Thursday, November 20, 2014 4:55 PM
  • You mentioned Backup Exec so I'd like to help out.

     Are there any specific errors you are getting in the job log?  also what are the versions of Backup Exec and Exchange?

     Make sure there aren't any utility jobs running at the same time as the backup since this can cause conflicts with the writers. Also check that any Backup Exec processes are excluded from antivirus scans.


    Lenora Moss Technical Support Engineer, SMB Partner Support, Symantec Corporation www.symantec.com

    Friday, November 21, 2014 1:14 AM
  • Yes Backup Exec also (we have 2014 SP1 hf225092). Problem was solved by uninstalling KB300853. I did'n wait hotfix KB2996928.

    Some logs, this error was random, not every backup job, but almost every second:

    BackupV-79-57344-65233 - Snapshot Technology: Initialization failure on: "\\2012server or exchange2013\System?State". Snapshot technology used: Microsoft Volume Shadow Copy Service (VSS).

    Snapshot technology error (0xE000FED1): A failure occurred querying the Writer status. See the job log for details about the error.

    Check the Windows Event Viewer for details.

    Writer Name: Background Intelligent Transfer Service, Writer ID: {4969D978-BE47-48B0-B100-F328F07AC1E0}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Stable (1).

    V-79-57344-65233 - Snapshot Technology: Initialization failure on: "\\2012fileserver\System?State". Snapshot technology used: Microsoft Volume Shadow Copy Service (VSS).

    Snapshot technology error (0xE000FED1): A failure occurred querying the Writer status. See the job log for details about the error.

    Check the Windows Event Viewer for details.

    Writer Name: Automated System Recovery, Writer ID: {BE000CBE-11FE-4426-9C58-531AA6355FC4}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Stable (1).

    Writer Name: Background Intelligent Transfer Service, Writer ID: {4969D978-BE47-48B0-B100-F328F07AC1E0}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Stable (1).

    Writer Name: COM+ Class Registration Database, Writer ID: {542DA469-D3E1-473C-9F4F-7847F01FC64F}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Stable (1).

    Writer Name: Registry, Writer ID: {AFBAB4A2-367D-4D15-A586-71DBB18F8485}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Stable (1).

    Writer Name: System Files, Writer ID: {E8132975-6F93-4464-A53E-1050253AE220}, Last error: The VSS Writer timed out (0x800423f2), State: Failed during prepare backup operation (7).

    Writer Name: Windows Management Instrumentation, Writer ID: {A6AD56C2-B509-4E6C-BB19-49D8F43532F0}, Last error: The VSS Writer timed out (0x800423f2), State: Failed during prepare backup operation (7).


    • Edited by jyrppa Friday, November 21, 2014 5:52 AM
    Friday, November 21, 2014 5:34 AM
  • I agree with George Mather1 !

    I installed KB2996928 (http://support.microsoft.com/kb/2996928/) on Exchange 2010 Windows Server 2012 and restart server. After i performed backup and got success.

    Log from Exchange 2010 Windows Server 2012

    "

    Exchange VSS Writer (instance ffde3d3d-a189-4bd3-9065-e16b96274cbb:15) has processed the backup completion event successfully.

    "

    Use DPM 2012 R2 UR4.


    Have a nice day !!!

    Friday, November 21, 2014 5:47 AM
  • I am also affected by this issue.  I can confirm that installing KB2996928 on top of KB3000853 does indeed fix the problem.
    Friday, November 21, 2014 2:38 PM
  • KB 3000853 is WS 2012, so it is expected that you would not have it installed on WS 2012 R2.
    Friday, November 21, 2014 5:06 PM
  • The Hotfix KB2996928 resolved the issue for my Systems. VSS Backups working.

    btw. KB2996928 can imported into WSUS. Saves me a lot of time patching my systems


    This posting is provided AS IS with no warranties.

    Monday, November 24, 2014 9:33 AM
  • Hi,

    Installing KB2996928 on top of KB3000853 fixed my problem too. W2012/E2013 SP1.

    MS, please include the correct version fo KB2996928 in the KB3000853 package!

    I installed this 48hs ago from Windows Update and It was still the one with the bug.

    HP Data Protector error log:

    [Major] From: OB2BAR_VSSBAR@server "MS Exchange 2010+ Server"  Time: 24/11/2014 11:39:31 p.m.
    [145:575]  Writer 'Microsoft Exchange Writer' failed to prepare files for backup:
         Reported state: VSS_WS_STABLE
         Expected state: VSS_WS_WAITING_FOR_BACKUP_COMPLETE
         Failure code:  0x0

    Thanks

    Tuesday, November 25, 2014 2:55 AM
  • The KB 3000853 package was updated today:

    KB3000853 was re-released on November 25, 2014 with the correct version of KB2996928. This update can be obtained from Windows Update or Microsoft Download Center. 

    Tuesday, November 25, 2014 7:52 PM
  • Hi guys

    I can confirm that installing KB2996928 on affected servers resolves the issue without the need to uninstall KB3000853.

    http://support.microsoft.com/kb/3000853

    http://support.microsoft.com/kb/2996928/

    Regards


    We had the same problem in a customer environment with System Center Data Protection Manager (DPM) 2012 R2 UR4. After installing the KB2996928-V2 on the two Exchange Servers and restarting the Servers the problem was solved.
    Friday, November 28, 2014 1:28 PM
  • Hi

    I had the same issue on my Exchange server, doing dpm backups would run to a point and then fail. Uninstalling the update KB3000853 and rebooting returns the VSS writer back to a working state.


    Hope this helps. Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Friday, November 28, 2014 9:18 PM