none
DPM 2012 R2 Error 0x809909E2 backing up Hyper-V VM on a SOFS Cluster RRS feed

  • Question

  • Here is the architecture:

    Hyper-V VM (2012 R2)

    Hyper-V Host (2012 R2)

    -------- SMB Share -----

    2012 R2 Scale out file Server (2 Nodes)

    The cluster validates fine. The error DPM reports in the console is this:

    Failure occurred while adding one or more of the volumes involved in backup operation to snapshot set. Please check the event log on HOST to troubleshoot the issue. (ID 30290 Details: Internal error code: 0x809909E2)

    Here is the Host's DPMRACurr log, reporting its unable to successfully retrieve the snapshots:


    2E28    2DA4    09/11    23:06:39.755    31    vssbaserequestor.cpp(194)    [0000000000E71420]    63DEF0BE-D932-4038-B442-098D692DEEBD    NORMAL    CVssBaseRequestor::StartGatherWriterMetadata [0000000000E71420]
    2E28    2DA4    09/11    23:06:42.783    31    vssbaserequestor.cpp(944)    [0000000000E71420]        NORMAL    QueryStatus returned 0x4230a, Releasing VssAsync [0000000001E8F5A0]
    2E28    2DA4    09/11    23:06:42.815    22    hypervutil.cpp(493)            NORMAL    VMId: 2C20DA32-A22B-4BC9-B3C8-44AC6F406B3F, ConfigFileName: \\hypervsofs\hypervstorage\vm1\bf200x67\Virtual Machines\2C20DA32-A22B-4BC9-B3C8-44AC6F406B3F.xml
    2E28    2DA4    09/11    23:06:42.830    31    vsssnapshotrequestor.cpp(1727)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::[0000000000E71420], Found matching vss logical path: , component name: 2C20DA32-A22B-4BC9-B3C8-44AC6F406B3F, for writer id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
    2E28    2DA4    09/11    23:06:42.830    31    hypervwriterhelperplugin.cpp(213)            NORMAL    Component Info, bstrLogicalPath:, bstrComponentName:2C20DA32-A22B-4BC9-B3C8-44AC6F406B3F, eComponentType:2
    2E28    2DA4    09/11    23:06:42.830    31    hypervwriterhelperplugin.cpp(171)            WARNING    Error:0, Failed to read user choice of backup
    2E28    2DA4    09/11    23:06:42.830    31    hypervwriterhelperplugin.cpp(283)            NORMAL    bAllowOfflineBackup[1] bLogicalPathContainsOnlineBackup[1]
    2E28    2DA4    09/11    23:06:42.830    31    vsssnapshotrequestor.cpp(252)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddComponentForSnapshot [0000000000E71420]
    2E28    2DA4    09/11    23:06:42.846    31    vsssnapshotrequestor.cpp(1925)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForFile [0000000000E71420]
    2E28    2DA4    09/11    23:06:42.893    31    vsssnapshotrequestor.cpp(459)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForSnapshot [0000000000E71420]
    2E28    2DA4    09/11    23:06:42.893    31    vsssnapshotrequestor.cpp(482)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor: AddVolumeForSnapshot - Marked volume \\HYPERVSOFS\HYPERVSTORAGE\ to be snapshot
    2E28    2DA4    09/11    23:06:42.893    31    vsssnapshotrequestor.cpp(1925)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForFile [0000000000E71420]
    2E28    2DA4    09/11    23:06:42.940    31    vsssnapshotrequestor.cpp(459)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForSnapshot [0000000000E71420]
    2E28    2DA4    09/11    23:06:42.940    31    vsssnapshotrequestor.cpp(1925)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForFile [0000000000E71420]
    2E28    2DA4    09/11    23:06:42.971    31    vsssnapshotrequestor.cpp(459)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForSnapshot [0000000000E71420]
    2E28    2DA4    09/11    23:06:42.971    31    vsssnapshotrequestor.cpp(1925)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForFile [0000000000E71420]
    2E28    2DA4    09/11    23:06:43.033    31    vsssnapshotrequestor.cpp(459)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForSnapshot [0000000000E71420]
    2E28    2DA4    09/11    23:06:43.033    31    vsssnapshotrequestor.cpp(1925)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForFile [0000000000E71420]
    2E28    2DA4    09/11    23:06:43.080    31    vsssnapshotrequestor.cpp(459)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForSnapshot [0000000000E71420]
    2E28    2DA4    09/11    23:06:43.080    31    vsssnapshotrequestor.cpp(1925)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForFile [0000000000E71420]
    2E28    2DA4    09/11    23:06:43.127    31    vsssnapshotrequestor.cpp(459)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::AddVolumeForSnapshot [0000000000E71420]
    2E28    2DA4    09/11    23:06:43.190    31    agentproxyhelper.cpp(67)            NORMAL    CSV share [\\HYPERVSOFS\HYPERVSTORAGE\], owner node [BF200S54]
    2E28    2DA4    09/11    23:06:43.205    31    hypervwriterhelperplugin.cpp(558)    [00000000011502C0]        NORMAL    Component 2C20DA32-A22B-4BC9-B3C8-44AC6F406B3F is a VM
    2E28    2DA4    09/11    23:06:43.221    31    vsssnapshotrequestor.cpp(544)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::StartPrepareForBackup [0000000000E71420]
    2E28    2DA4    09/11    23:06:43.236    31    vsssnapshotrequestor.cpp(1608)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::ReleaseVolumesForSnapshot [0000000000E71420]
    2E28    2DA4    09/11    23:06:43.236    31    vsssnapshotrequestor.cpp(1505)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::PrepareVolumesForSnapshot [0000000000E71420]
    2E28    2DA4    09/11    23:06:43.236    31    vsssnapshotrequestor.cpp(593)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor: Using provider {00000000-0000-0000-0000-000000000000} for volume \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:06:43.268    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:06:48.299    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:06:53.346    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:06:58.376    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:07:03.409    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:07:08.448    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:07:13.488    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:07:18.527    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:07:23.571    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:07:28.602    31    vsssnapshotrequestor.cpp(610)    [0000000000E71420]        NORMAL    Hr: = [0x8004230c] AddToSnapshotSet failed for Volume: \\HYPERVSOFS\HYPERVSTORAGE\
    2E28    2DA4    09/11    23:07:33.618    31    vsssessioncontext.cpp(143)    [000000000331EA20]        WARNING    <VSS_CONTEXT><COMPONENTS><COMPONENT><WriterId>{66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}</WriterId><WriterName></WriterName><LogicalPath></LogicalPath><ComponentName>2C20DA32-A22B-4BC9-B3C8-44AC6F406B3F</ComponentName><ComponentType>2</ComponentType></COMPONENT></COMPONENTS><SNAPSHOT_CONTEXT>          SelectComponent = 1,          PartialFileSupport = 0,          BootableState = 0,          BackupType = 1,          SnapshotContext = 0,          SnapshotAttributes = VSS_CTX_BACKUP          </SNAPSHOT_CONTEXT><SnapShotVolumes><Volume>              <Name>\\HYPERVSOFS\HYPERVSTORAGE\</Name>              <SnapshotPath>(null)</SnapshotPath>              <MountPoint>\\HYPERVSOFS\HYPERVSTORAGE\</MountPoint>              <MountPointArray></MountPointArray>              <SnapshotId>{00000000-0000-0000-0000-000000000000}</SnapshotId>              <ProviderId>{00000000-0000-0000-0000-000000000000}</ProviderId>              <IsHardwareProvider>0</IsHardwareProvider>              </Volume></SnapShotVolumes>
    2E28    2DA4    09/11    23:07:33.618    31    vsssessioncontext.cpp(143)    [000000000331EA20]        WARNING    </VSS_CONTEXT>
    2E28    2DA4    09/11    23:07:33.618    05    fsmtransition.cpp(111)    [0000000003323890]        WARNING    Failed: Hr: = [0x809909e2] HasEventErrorCode: completion: 0xa10c, signature: 0xaabbcc00
    2E28    2DA4    09/11    23:07:33.618    31    vsssnapshotrequestor.cpp(2056)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::DoFailureCleanup [0000000000E71420]
    2E28    2DA4    09/11    23:07:33.618    31    vsssnapshotrequestor.cpp(1608)    [0000000000E71420]        NORMAL    CVssSnapshotRequestor::ReleaseVolumesForSnapshot [0000000000E71420]
    2E28    2C98    09/11    23:07:33.618    31    aasubtask.cpp(906)    [000000000331B840]        WARNING    <?xml version="1.0"?>
    2E28    2C98    09/11    23:07:33.618    31    aasubtask.cpp(906)    [000000000331B840]        WARNING    <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2137454110" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="e8cf75b1-4e1e-4404-8536-b2abb128bcab" GuidWorkItem="ca282574-7bc3-4eec-b0da-6f6297707979" TETaskInstanceID="63def0be-d932-4038-b442-098d692deebd"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2137454110" DetailedSource="2"/><RAStatus><RAPreBackup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><BackupTime>0</BackupTime><DSStatus><ComponentName>2C20DA32-A22B-4BC9-B3C8-44AC6F406B3F</ComponentName><LogicalPath></LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>
    2E28    2C98    09/11    23:07:33.618    03    runtime.cpp(1376)    [0000000000E740F0]        FATAL    Subtask failure, sending status response XML=[<?xml version="1.0"?>
    2E28    2C98    09/11    23:07:33.618    03    runtime.cpp(1376)    [0000000000E740F0]        FATAL    <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2137454110" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="e8cf75b1-4e1e-4404-8536-b2abb128bcab" GuidWorkItem="ca282574-7bc3-4eec-b0da-6f6297707979" TETaskInstanceID="63def0be-d932-4038-b442-098d692deebd"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="30290" DetailedCode="-2137454110" DetailedSource="2"><Parameter Name="AgentTargetServer" Value="VMHOST"/></ErrorInfo><RAStatus><RAPreBackup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><BackupTime>0</BackupTime><DSStatus><ComponentName>2C20DA32-A22B-4BC9-B3C8-44AC6F406B3F</ComponentName><LogicalPath></LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>
    2E28    2C98    09/11    23:07:33.618    03    runtime.cpp(1376)    [0000000000E740F0]        FATAL    ]
    2E28    2C98    09/11    23:07:33.633    31    vsssnapshotrequestor.cpp(656)    [0000000000E71420]    63DEF0BE-D932-4038-B442-098D692DEEBD    WARNING    Failed: Hr: = [0x809909e2] : AddToSnapshotSet failed for volume : \\HYPERVSOFS\HYPERVSTORAGE\ with hr: 0x8004230C
    2E28    2C98    09/11    23:07:33.633    31    createsnapshotsubtask.cpp(1532)    [000000000331B840]    63DEF0BE-D932-4038-B442-098D692DEEBD    WARNING    Failed: Hr: = [0x809909e2] : Encountered Failure: : lVal : pSnapshotRequestor->StartPrepareForBackup(snapshotSetId, m_fUseSystemSoftwareProviderOnly)
    2E28    2C98    09/11    23:07:33.633    05    fsmstate.cpp(167)    [0000000003322B80]    63DEF0BE-D932-4038-B442-098D692DEEBD    WARNING    Failed: Hr: = [0x809909e2] : Encountered Failure: : lVal : pTransition->Execute(pEvent)
    2E28    2C98    09/11    23:07:33.633    05    genericfsm.cpp(225)    [000000000331BC00]    63DEF0BE-D932-4038-B442-098D692DEEBD    WARNING    Failed: Hr: = [0x809909e2] : Encountered Failure: : lVal : m_pCurrentState->SendEvent(pEvent, pNextState)
    :

    On the Scale out file server, the DPMRA service is acting like it can't find information about the cluster:

    2354    1FF0    09/11    23:00:48.497    05    genericstatus.cpp(1662)    [000000000167C700]    FC676096-29B2-4AB1-9D60-12EAE58031B8    NORMAL    majorVerNum[4], minorVerNum[2], buildNum[1217], hotfixNum[0]
    2354    1FF0    09/11    23:00:48.588    31    vssbaserequestor.cpp(51)    [0000000003963058]    29D2416C-EAE3-4D4F-9FD0-B05C4D1CA0F4    NORMAL    CVssBaseRequestor: constructor [0000000003963058]
    2354    1FF0    09/11    23:00:48.588    31    vsssnapshotrequestor.cpp(91)    [0000000003963058]    29D2416C-EAE3-4D4F-9FD0-B05C4D1CA0F4    NORMAL    CVssSnapshotRequestor::CVssSnapshotRequestor [0000000003963058]
    2354    1FF0    09/11    23:00:48.588    03    workitem.cpp(86)    [0000000001680F10]    29D2416C-EAE3-4D4F-9FD0-B05C4D1CA0F4    ACTIVITY    Idle Timer created with timeout = 390000
    2354    1FF0    09/11    23:00:48.603    31    vainquirysubtask.cpp(424)    [0000000003962D20]    29D2416C-EAE3-4D4F-9FD0-B05C4D1CA0F4    NORMAL    Writer metadata not needed for inquiry.
    2354    1FF0    09/11    23:00:48.603    61    inquirysubtask.cpp(1031)    [0000000003962D20]    29D2416C-EAE3-4D4F-9FD0-B05C4D1CA0F4    NORMAL    ==>CInquirySubTask::ExecuteInquiry
    2354    1FF0    09/11    23:00:48.603    61    necluster.cpp(84)    [00000000039626B0]    29D2416C-EAE3-4D4F-9FD0-B05C4D1CA0F4    NORMAL    ==>CNECluster::EnumerateClusterDetails
    235


    Thursday, September 11, 2014 11:22 PM

Answers

  • Hi,


    If trying to back up a Hyper-V VM data source that resides on the Server 2012 SMB, please check the following.

    Configure DPM backup for virtual machines using SMB storage
    http://technet.microsoft.com/en-us/library/hh757866.aspx#bkmk_protectvmwithsmb

    Make sure the Remote SMB file server prereqs are met:

    1) DPM Agent is required on all computers involved (Hyper-V server(s) and the SMB share server.
     2) You must Enable the "File Server VSS Agent Service" role on Remote SMB File server
     Add the role via Server manager\Tools\Add Role.  On the server role selection, expand:


          File and iScsi Services
                File and iScsi Services
                      File Server VSS Agent Service


    3) Hyper-V nodes and DPM server machine$ account should have full permissions on the specific Remote SMB File share.
     4) The cluster service account should have full permission on the specific Remote SMB file share
       NOTE: Accounts should be explicitly added and not embedded in other accounts as odd behavior has been observed when using groups. 
             This is not a requirement but simply something we have seen.


    If you are using a hardware VSS provider, please add the following key on the hyper-V nodes and SMB server so DPM will use system provider and see if that fixes the issue.

    HKLM\Software\Microsoft\Microsoft Data Protection Manager\Agent\UseSystemSoftwareProvider


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, September 12, 2014 10:12 AM
    Moderator

All replies

  • Hi,


    If trying to back up a Hyper-V VM data source that resides on the Server 2012 SMB, please check the following.

    Configure DPM backup for virtual machines using SMB storage
    http://technet.microsoft.com/en-us/library/hh757866.aspx#bkmk_protectvmwithsmb

    Make sure the Remote SMB file server prereqs are met:

    1) DPM Agent is required on all computers involved (Hyper-V server(s) and the SMB share server.
     2) You must Enable the "File Server VSS Agent Service" role on Remote SMB File server
     Add the role via Server manager\Tools\Add Role.  On the server role selection, expand:


          File and iScsi Services
                File and iScsi Services
                      File Server VSS Agent Service


    3) Hyper-V nodes and DPM server machine$ account should have full permissions on the specific Remote SMB File share.
     4) The cluster service account should have full permission on the specific Remote SMB file share
       NOTE: Accounts should be explicitly added and not embedded in other accounts as odd behavior has been observed when using groups. 
             This is not a requirement but simply something we have seen.


    If you are using a hardware VSS provider, please add the following key on the hyper-V nodes and SMB server so DPM will use system provider and see if that fixes the issue.

    HKLM\Software\Microsoft\Microsoft Data Protection Manager\Agent\UseSystemSoftwareProvider


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, September 12, 2014 10:12 AM
    Moderator
  • Ahh, well it was one of two things - I didn't have the VSS Agent installed on the SMB Cluster and I also explicitly added the cluster service name to the share permissions.

    That is also not the first time I've had issues with the Hyper-V servers behaving badly connecting to backend SMB Shares when using groups for NTFS permissions instead of explicit computer accounts.


    Thanks!

    Friday, September 12, 2014 5:08 PM