none
Error 30112 when backing up clustered Hyper-V VM RRS feed

  • Question

  • We have a Hyper-V R2 cluster with DPM 2010 doing the VM backups at the host level. All VMs are backing up and restoring fine except two. On those VMs the following error occurs: (I changed the VM and server names)


    Description:    The back up to tape job failed for the following reason: (ID 3311)

    There are no significant events where he tells me to look, just the same VSS events as in a successful backup.

    These 2 VMs do have an extra VHD with over 200GB attached. The second one was backing up fine until recently. Most VMs have dynamically expanding disks. These 2 have them.

    Error details below.

    \Backup Using Child Partition Snapshot\*VM1 (*VM1*.Cluster.Domain) DPM encountered a retryable VSS error. (ID 30112 Details: VssError:The writer experienced a transient error.  If the backup process is retried,
    the error may not reoccur.
    (0x800423F3))
    Check the Application Event Log on Clusternode2.domain for the cause of the failure. Fix the cause and retry the operation.
    For more information on this error, go to http://go.microsoft.com/fwlink/?LinkId=132612.
    \Backup Using Child Partition Snapshot\*VM2* (VM2.Cluster.domain) DPM encountered a retryable VSS error. (ID 30112 Details: VssError:The writer experienced a transient error.  If the backup process is retried,
    the error may not reoccur.
    (0x800423F3))

    Check the Application Event Log on Clusternode1.Domain for the cause of the failure. Fix the cause and retry the operation.
    For more information on this error, go to http://go.microsoft.com/fwlink/?LinkId=132612.

    • Moved by Parag Agrawal Wednesday, August 4, 2010 12:03 PM Moving this to DPM Hyper-V forum for better track-ability and response (From:Data Protection Manager)
    Wednesday, July 28, 2010 3:45 PM

Answers

  • The retryable VSS error can also occur if there is a problem with one of the VSS writers within the VM itself (e.g. writer in a failed/retryable state). Have you executed the "vssadmin.exe list writers" command within the problem VM?
    Cheers, Tyler F [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by Estrelo Wednesday, September 8, 2010 12:32 AM
    Friday, September 3, 2010 8:07 PM
    Moderator
  • It could be the SQLServerWriter Service. Can you turn off the SQL writer service and take a backup to assert/eliminate this being the faulty writer. Are there any additional events in the application event log of the VM where the SQL server writer service is runinng?


    Thanks Shyama Hembram[MSFT] This posting is provided AS IS, with no warranties, and confers no rights.
    • Marked as answer by Estrelo Wednesday, September 8, 2010 12:33 AM
    Tuesday, September 7, 2010 6:44 PM
    Moderator

All replies

  • Hi

    Since DPM uses the VSS for Hyper-V it happens that the VSS comes into a retyable state. To find out if your VSS for Hyper-V is in an other state than stable open an elivated dos promt and type "vssadmin list writers" without ""

    Is your VSS writer for Hyper-V stable? If not reinitiate your VSS, the best thing is to reboot the Hyper-V host but there is other ways also.

    BR

    Robert Hedblom


    Check out my DPM blog http://robertanddpm.blogspot.com
    Wednesday, July 28, 2010 6:19 PM
    Moderator
  • Hello Robert.

    It could be that, but this has been going on for some days now, the backups are daily, the error is daily with one of the VMs, occasional with the other, and all other VMs are backing up fine. I did notice that both VMs with problems have 2 disks, and in both cases disk 1 is on CSV1 and disk 2 is in CSV2. I guess I probably just need to switch those VHDs around.

    Will report back.

    Update:

    It's not the issue stated above. I have moved the 2nd VHD of one on the problematic VMs to the same CSV as the 1st disk. Backup errors out the same way. 

    The error happens at the beginning of the backup. I can't find any relevant errors on the host machine, just a generic backup failed, ID 10102

     

    Wednesday, July 28, 2010 10:16 PM
  • One more thing. I was reading your blog (thank you for starting one on DPM!) and I did notice when reading your description that my backups do not run 3 at a time as they were supposed to according to your description. My SAN is an MSA2000 connected over DP SAS cable to both hosts.
    Thursday, July 29, 2010 8:02 AM
  • Are those VMs running win2k3 by any chance. Also do you have enough free space on the VHDs to take backup. 

    They may not be the cause of the problem, but it would be nice to have some more context on why these 2 particular VMs

     

    also if you can search through the archives of this technet (DPM technet), you might actually come across similar problem. Somewhere i remember it had something to do with some essential Hyper-V components missing on the VM. 

    Thanks,

    Parag

    This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, July 29, 2010 1:25 PM
  • Hello Parag,

    One VM is a Server 2008 R2, other one is an SBS. All updated with the latest integration components after the upgrade to DPM 2010 final. Backup is made to a HP Disk2Disk with deduping, currently at 50% capacity. DPM thinks it's a MSL G3 Series (4x48) / LTO 3 full with 400GB and 800GB LTO3 tapes. Total size on the VHDs:  VM1 with 244GB+53,8GB=298GB and VM2 with 173GB+67,5GB=241GB

    I don't think it's a space issue.

    Oh, and I only have tape backups configured, I would guess it only does one at a time because of that, so that's out too...

     

     

    Friday, July 30, 2010 1:10 AM
  • The retryable VSS error can also occur if there is a problem with one of the VSS writers within the VM itself (e.g. writer in a failed/retryable state). Have you executed the "vssadmin.exe list writers" command within the problem VM?
    Cheers, Tyler F [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by Estrelo Wednesday, September 8, 2010 12:32 AM
    Friday, September 3, 2010 8:07 PM
    Moderator
  • That could be it... Here's the info he gave back:

    <!-- /* Font Definitions */ @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-520092929 1073786111 9 0 415 0;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0cm; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:Calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi; mso-fareast-language:EN-US;} p.MsoPlainText, li.MsoPlainText, div.MsoPlainText {mso-style-noshow:yes; mso-style-priority:99; mso-style-link:"Plain Text Char"; margin:0cm; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; mso-bidi-font-size:10.5pt; font-family:"Calibri","sans-serif"; mso-fareast-font-family:Calibri; mso-fareast-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi; mso-fareast-language:EN-US;} span.PlainTextChar {mso-style-name:"Plain Text Char"; mso-style-noshow:yes; mso-style-priority:99; mso-style-unhide:no; mso-style-locked:yes; mso-style-link:"Plain Text"; mso-bidi-font-size:10.5pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-hansi-font-family:Calibri;} .MsoChpDefault {mso-style-type:export-only; mso-default-props:yes; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:Calibri; mso-fareast-theme-font:minor-latin; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi; mso-fareast-language:EN-US;} @page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 3.0cm 70.85pt 3.0cm; mso-header-margin:36.0pt; mso-footer-margin:36.0pt; mso-paper-source:0;} div.WordSection1 {page:WordSection1;} -->

    Writer name: 'Task Scheduler Writer'

        Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}

        Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}

        State: [1] Stable

        Last error: No error

     

    Writer name: 'VSS Metadata Store Writer'

         Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}

        Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}

        State: [1] Stable

        Last error: No error

     

    Writer name: 'Performance Counters Writer'

        Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}

        Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}

        State: [1] Stable

        Last error: No error

     

    Writer name: 'SqlServerWriter'

        Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}

        Writer Instance Id: {7a49fba0-50db-4d75-b60f-ad5613d77b48}

        State: [8] Failed

        Last error: Non-retryable error

     

    Writer name: 'System Writer'

        Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}

        Writer Instance Id: {37262343-0e9c-4dcd-a983-b1446f83004a}

        State: [5] Waiting for completion

        Last error: No error

     

    Writer name: 'ASR Writer'

        Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}

        Writer Instance Id: {5f08cda7-90d7-42d1-be38-53898c530d25}

        State: [1] Stable

        Last error: No error

     

    Writer name: 'WMI Writer'

        Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}

        Writer Instance Id: {a8068876-2b4b-4ba5-adae-4e27c4f5a7bd}

        State: [5] Waiting for completion

        Last error: No error

     

    Writer name: 'Registry Writer'

        Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}

        Writer Instance Id: {d63f1867-38b6-434d-9c88-1dc96b99d9d4}

        State: [1] Stable

        Last error: No error

     

    Writer name: 'COM+ REGDB Writer'

        Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}

        Writer Instance Id: {1ae59e83-085a-4f49-91aa-42289936741e}

        State: [1] Stable

        Last error: No error

     

    Writer name: 'Shadow Copy Optimization Writer'

        Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}

        Writer Instance Id: {e54cbe77-ea6e-436f-8445-fc592a0c5934}

        State: [1] Stable

        Last error: No error

     

     

    Sunday, September 5, 2010 5:40 PM
  • Is the Integration component updated inside the Guest? If not can you give it a try and post back if backup continues to fail with this error.


    Thanks Shyama Hembram[MSFT] This posting is provided AS IS, with no warranties, and confers no rights.
    Monday, September 6, 2010 7:30 PM
    Moderator
  • Yes, of course, that was one of the first things I've tried. The integration components install but no restart is required. They ask for a restart when they update. The last one was after the jump from DPM beta to final I think.

     

    Could it be one of these two? I don't know much about VSS writers...

    Writer name: 'SqlServerWriter'

        Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}

        Writer Instance Id: {7a49fba0-50db-4d75-b60f-ad5613d77b48}

        State: [8] Failed

        Last error: Non-retryable error

     

     Writer name: 'System Writer'

        Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}

        Writer Instance Id: {37262343-0e9c-4dcd-a983-b1446f83004a}

        State: [5] Waiting for completion

        Last error: No error

     

    Monday, September 6, 2010 10:32 PM
  • It could be the SQLServerWriter Service. Can you turn off the SQL writer service and take a backup to assert/eliminate this being the faulty writer. Are there any additional events in the application event log of the VM where the SQL server writer service is runinng?


    Thanks Shyama Hembram[MSFT] This posting is provided AS IS, with no warranties, and confers no rights.
    • Marked as answer by Estrelo Wednesday, September 8, 2010 12:33 AM
    Tuesday, September 7, 2010 6:44 PM
    Moderator
  • Hi

    Look into the event logs of the server that has SQL installed. Maby the disk is full where you located your transactionlogs, this will make the VSS jump into an error state.

    There is also something that trying to use your System Writer that is used when making backup of system state's. If possible, reboot the server with the VSS error, verify with vssadmin list writers command from an elevated DOS promt that all your VSS'es is in a stable state, retry the job in DPM or create a new recovery point.

    Get back with info =)

    BR

    Robert Hedblom


    Check out my DPM blog @ http://robertanddpm.blogspot.com
    Tuesday, September 7, 2010 9:00 PM
    Moderator
  • I think it might be sorted, there were a couple of issues keeping the Sql server writer from starting. I have to wait on some other backup jobs before I can test it. Will post back.

     

    EDIT: Still not solved. I get error 24583 on the Server.

    Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013
    Error state: 1, Severity: 16
    Source: Microsoft SQL Server Native Client 10.0
    Error message: BACKUP DATABASE is terminating abnormally.
    SQLSTATE: 42000, Native Error: 3271
    Error state: 1, Severity: 16
    Source: Microsoft SQL Server Native Client 10.0
    Error message: A nonrecoverable I/O error occurred on file "{FAD141D9-B36D-4078-AE8F-81F336AF1595}3:" 995(failed to retrieve text for this error. Reason: 15105).

    Wednesday, September 8, 2010 12:36 AM
  • Hi,

    I'm getting the same error 24583 when creating a bare metal VSS full backup of W2k8 R2 Hyper-V host including all volumes, specifically on running virtual Forefront TMG 2010 (SQL Server 2008 Express). According to http://troubleshootingsql.com/2010/12/06/volume-shadow-barfs-with-0x80040e14-code/ I've tried increase max worker threads, but WSB is still failing. I am not able to succesfully backup the whole Hyper-V for months!

    But DO NOT STOP SQL Server VSS Write Service because you wouldn't be able to recover any SQL Server database files from such a Volume Shadow Backup.

    There is another thread (http://social.technet.microsoft.com/Forums/en-US/sqldatabaseengine/thread/4faff249-0639-4898-918c-6b0a5b1f9017) also without resolution :(

    Any new suggestions?

    Tuesday, December 21, 2010 3:07 PM
  • I've managed to successfuly finish backup by increasing max worker threads to double of database count (and a bit :) with
    EXEC sp_configure 'max worker threads', <count>
    To be sure and get the exact number, after a failed snapshot use
    SELECT status,COUNT(*) AS counts FROM sys.dm_os_schedulers GROUP BY status
    Saturday, January 8, 2011 1:37 PM