none
Error 199 trying to protect SoFS Share with DPM RRS feed

  • Question

  • I'm trying to protect a Scale Out File Server in a 2 node cluster using DPM (same issue on both 2016 and 2019). However when trying to add the shares to the protection group, i receive Error ID 199 when DPM tries to enumerate the cluster group.

    I get the following for each file share

    DPM was unable to protect the members in the resource group<ObjectName>. This resource group contains resources that have dependencies that are not part of this resource group.

    Where objectname is the share name. The storage for the shares is in Cluster Shared Storage and doesn't have any dependency options when i view it in the Cluster Manager so i can't check the dependencies as the comments for error 199 would recommend.

    Any ideas how i can address this, i'm pretty sure backup of a SoFS share is supported by DPM.

    Wednesday, May 29, 2019 2:29 PM

Answers

All replies

  • Hello James,

    What kind of workloads are you storing on your Scale-Out File Server (SoFS) share?

    Problems with backing up SoFS shares usually tend to be due to permission issues.

    Can you check the following logs for any more information?

    Cluster nodes log:

    • %ProgramFiles%\Microsoft Data Protection Manager\DPM\Temp\DPMRACurr.errlog

    DPM server log:

    • %ProgramFiles%\Microsoft System Center\DPM\DPM\Temp\MSDPMCurr.errlog

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, May 29, 2019 2:49 PM
  • It's used to store web application files and IIS configuration data. Essentially it's just a file server however

    I've checked the logs and i can't see anything that may shed any more light on the problem.

    The shares aren't in the protection group as i can't see them when viewing the cluster group, i just receive the 199 errors when trying to expand the group to add them to the protection group.

    The shares themselves are working fine otherwise.

    Wednesday, May 29, 2019 4:39 PM
  • I've just tried again and checked the logs, i've been able to find some details which may be relevant, although they don't mention the shares specifically or the error 199

    PAAS-FILES-N1 is one of the nodes in the cluster and CLS-PAAS-FILES is the cluster name

    17B0    1F98    05/29    16:54:33.059    01    TaskExecutor.cs(692)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Setting up Fsm: verb[9c494a3e-9680-4dc4-8e3a-885f9959c2e4]
    17B0    1F98    05/29    16:54:33.075    01    TaskExecutor.cs(487)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    ExecuteTask(job=69faf761-36d4-40db-94b2-3d2d726c69cf) - Starting task instance
    17B0    1F4C    05/29    16:54:33.075    02    EventManager.cs(98)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Publishing event from TaskInstance.cs(411): TaskStart, [TaskID=e07e85c9-af69-4cfe-80a7-c4e12e72b2a2]
    17B0    1F4C    05/29    16:54:33.075    01    TaskInstance.cs(447)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Task: Initial transition enqueued, startState='Init State'
    17B0    1F98    05/29    16:54:33.075    68    CMCatalog.cs(71)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Server paas-appfiles.CLS-PAAS-FILES.Jaama.co.uk ;IsClustered = True; Cluster name = CLS-PAAS-FILES.Jaama.co.uk
    17B0    1F98    05/29    16:54:33.091    68    RunInquiryCommonTypes.cs(1361)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Adding inquiry workitem e07e85c9-af69-4cfe-80a7-c4e12e72b2a2 of server PAAS-FILES-N2.Jaama.co.uk with commandInstanceId 346135f1-efce-4ccb-9965-10ce702eea60 to RunningAcInquiries HashTable
    17B0    119C    05/29    16:54:33.294    68    RunInquiry.cs(467)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Agent Inquiry started on server PAAS-FILES-N2.Jaama.co.uk
    17B0    1F98    05/29    16:54:33.528    68    RunInquiry.cs(492)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Agent Inquiry Finished on server PAAS-FILES-N2.Jaama.co.uk
    17B0    1F98    05/29    16:54:33.528    68    RunInquiryCommonTypes.cs(1372)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Deleting inquiry workitem e07e85c9-af69-4cfe-80a7-c4e12e72b2a2 commandInstanceId 346135f1-efce-4ccb-9965-10ce702eea60 from RunningAcInquiries HashTable
    17B0    1F98    05/29    16:54:33.544    68    BranchedVolumeInquiry.cs(68)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    BranchedVolumeInquiry::TriggerVolumeInquiry:: CMTE triggered Volume Inquiry on server [PAAS-FILES-N1.Jaama.co.uk]
    17B0    1F98    05/29    16:54:33.559    68    RunInquiryCommonTypes.cs(1361)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Adding inquiry workitem e07e85c9-af69-4cfe-80a7-c4e12e72b2a2 of server PAAS-FILES-N1.Jaama.co.uk with commandInstanceId facd1c96-7af9-4211-a880-d6f15b196204 to RunningAcInquiries HashTable
    17B0    03B4    05/29    16:54:33.575    04    cmdproc.cpp(2055)    [000000001D970060]    E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Failed: Hr: = [0x80070005] CCommandProcessor::CCIE failed on server PAAS-FILES-N1.Jaama.co.uk, mqi.hr = 0x80070005, No further retry
    17B0    03B4    05/29    16:54:33.575    04    cmdproc.cpp(1879)    [000000001D970060]    E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Attempted CreateInstance with (fUseSpnWithRealm=false). Now retrying with SPNWithRealm
    17B0    03B4    05/29    16:54:33.591    04    cmdproc.cpp(2055)    [000000001D970060]    E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Failed: Hr: = [0x80070005] CCommandProcessor::CCIE failed on server PAAS-FILES-N1.Jaama.co.uk, mqi.hr = 0x80070005, No further retry
    17B0    03B4    05/29    16:54:33.591    04    cmdproc.cpp(2067)    [000000001D970060]    E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Failed: Hr: = [0x80070005] : Encountered Failure: : lVal : hr
    17B0    03B4    05/29    16:54:33.591    04    cmdproc.cpp(2665)    [000000001D970060]    E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    CCommandProcessor::SendOutboundCommand this:[000000001D970060], ServerName: PAAS-FILES-N1.Jaama.co.uk
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL    Task: Received tiemout message from CmdProc '<?xml version="1.0" encoding="utf-16"?>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL    <TECommand xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/2003/dls/Commands.xsd">
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL      <Command xmlns="http://schemas.microsoft.com/2003/dls/GenericCommand.xsd">
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <CommandInstanceID>facd1c96-7af9-4211-a880-d6f15b196204</CommandInstanceID>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <AgentTypeID>da6aa17a-d61c-4e9c-8cea-db25dea52a95</AgentTypeID>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <WorkItemID>e07e85c9-af69-4cfe-80a7-c4e12e72b2a2</WorkItemID>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <TETaskInstanceID>e07e85c9-af69-4cfe-80a7-c4e12e72b2a2</TETaskInstanceID>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <ServerIdFilter>00000000-0000-0000-0000-000000000000</ServerIdFilter>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <VerbIndexFilter>63</VerbIndexFilter>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <DatasourceIndexFilter>0</DatasourceIndexFilter>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL      </Command>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL      <ACInquiry Depth="Subtree" Flag="TopLevelPhysical" SchemaID="SVFF" InquiryType="Resultset" TraverseMountPoints="true" ReplicaInquiry="false">
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <StartNode xmlns="http://schemas.microsoft.com/2003/dls/CmteCommands.xsd">
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL          <JunctionlessPath xmlns="http://schemas.microsoft.com/2003/dls/CmteCommonTypes.xsd">
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL            <PathEntry Type="Domain" Value="Jaama.co.uk" />
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL            <PathEntry Type="Server" Value="PAAS-FILES-N1.Jaama.co.uk" />
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL          </JunctionlessPath>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL          <StartNodeRelativePath xmlns="http://schemas.microsoft.com/2003/dls/CmteCommonTypes.xsd">
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL            <PathEntry Type="Domain" Value="Jaama.co.uk" />
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL            <PathEntry Type="Server" Value="PAAS-FILES-N1.Jaama.co.uk" />
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL          </StartNodeRelativePath>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        </StartNode>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <RequestedType xmlns="http://schemas.microsoft.com/2003/dls/CmteCommands.xsd">Volume</RequestedType>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <RequestedAttribute xmlns="http://schemas.microsoft.com/2003/dls/CmteCommands.xsd">VolumeLabel</RequestedAttribute>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <RequestedAttribute xmlns="http://schemas.microsoft.com/2003/dls/CmteCommands.xsd">ReadOnlyVolume</RequestedAttribute>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <RequestedAttribute xmlns="http://schemas.microsoft.com/2003/dls/CmteCommands.xsd">BootVolume</RequestedAttribute>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <RequestedAttribute xmlns="http://schemas.microsoft.com/2003/dls/CmteCommands.xsd">DataSource</RequestedAttribute>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <RequestedAttribute xmlns="http://schemas.microsoft.com/2003/dls/CmteCommands.xsd">IsClustered</RequestedAttribute>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL        <RequestedAttribute xmlns="http://schemas.microsoft.com/2003/dls/CmteCommands.xsd">ClusteredToVN</RequestedAttribute>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL      </ACInquiry>
    17B0    24CC    05/29    16:54:33.591    01    TaskInstance.cs(256)            NORMAL    </TECommand>' Error CmdProcAccessDenied
    17B0    24CC    05/29    16:54:33.591    04    cmdproc.cpp(2067)    [000000001D970060]    E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Failed: Hr: = [0x80070005] : Encountered Failure: : lVal : hr
    17B0    24CC    05/29    16:54:33.591    04    cmdproc.cpp(1891)    [000000001D970060]    E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Failed: Hr: = [0x80070005] : CreateInstance Failed (fUseSpnWithRealm=true) : lVal : hr
    17B0    24CC    05/29    16:54:33.591    04    cmdproc.cpp(2421)    [000000001D970060]    E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Failed: Hr: = [0x80070005] : Encountered Failure: : lVal : CreateInstance( strCmdTarget, clsidTarget, hrDLS, (IUnknown **)&pAgentCommand, (pCommand->GetSenderToken() == 0), pCommand->IsNonDomainAgent(), fIsNonADMachine, cmdTargetIP )
    17B0    119C    05/29    16:54:33.591    68    RunInquiry.cs(514)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Agent Inquiry timed out on server PAAS-FILES-N1.Jaama.co.uk
    17B0    119C    05/29    16:54:33.591    68    InquiryContext.cs(398)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Enqueued message with errorcode 32,Access is denied (0x80070005)
    17B0    119C    05/29    16:54:33.591    68    RunInquiryCommonTypes.cs(1372)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Deleting inquiry workitem e07e85c9-af69-4cfe-80a7-c4e12e72b2a2 commandInstanceId facd1c96-7af9-4211-a880-d6f15b196204 from RunningAcInquiries HashTable
    17B0    0704    05/29    16:54:33.591    68    RunInquiryCommonTypes.cs(1007)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Task trail Errocode:PrtAccessDenied,-2147024891
    17B0    0704    05/29    16:54:33.606    68    BranchedVolumeInquiry.cs(122)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Volume inquiry failed for error code PrtAccessDenied; -2147024891; WindowsHResult
    17B0    0704    05/29    16:54:33.606    01    TaskInstance.cs(803)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    Task Diagnostic Information - <?xml version="1.0" encoding="utf-16"?>
    17B0    0704    05/29    16:54:33.606    01    TaskInstance.cs(803)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    <TaskExecutionContext>
    17B0    0704    05/29    16:54:33.606    01    TaskInstance.cs(803)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING      <TEVerb>RunDataSourceInquiryID</TEVerb>
    17B0    0704    05/29    16:54:33.606    01    TaskInstance.cs(803)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING      <TEErrorState>FindDataSourceRunVolumeInquiry.InquiryWaitForResults</TEErrorState>
    17B0    0704    05/29    16:54:33.606    01    TaskInstance.cs(803)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING      <TEErrorDetails>&lt;?xml version="1.0" encoding="utf-16"?&gt;
    17B0    0704    05/29    16:54:33.606    01    TaskInstance.cs(803)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    &lt;q1:ErrorInfo ErrorCode="32" DetailedCode="-2147024891" DetailedSource="2" ExceptionDetails="" xmlns:q1="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd"&gt;
    17B0    0704    05/29    16:54:33.606    01    TaskInstance.cs(803)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING      &lt;q1:Parameter Name="servername" Value="PAAS-FILES-N1.Jaama.co.uk" /&gt;
    17B0    0704    05/29    16:54:33.606    01    TaskInstance.cs(803)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    &lt;/q1:ErrorInfo&gt;</TEErrorDetails>
    17B0    0704    05/29    16:54:33.606    01    TaskInstance.cs(803)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    WARNING    </TaskExecutionContext>
    17B0    0704    05/29    16:54:33.606    02    EventManager.cs(98)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    Publishing event from TaskInstance.cs(828): TaskStop, [TaskID=e07e85c9-af69-4cfe-80a7-c4e12e72b2a2]
    17B0    0704    05/29    16:54:33.606    01    TaskExecutor.cs(843)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    FATAL    Task stopped (state=Failed, result=PrtAccessDenied; -2147024891; WindowsHResult), search "Task Diagnostic Information" for details.
    17B0    0704    05/29    16:54:33.606    16    ActiveJob.cs(745)            WARNING     Fail: Task 'e07e85c9-af69-4cfe-80a7-c4e12e72b2a2' failed with error during execution.
    17B0    0704    05/29    16:54:33.606    16    Task.cs(235)            NORMAL     Changing task state from 'GenerateWorkplan' -> 'Failed' (e07e85c9-af69-4cfe-80a7-c4e12e72b2a2)
    17B0    0704    05/29    16:54:33.606    16    ActiveJob.cs(904)            NORMAL     Dispatching RetireJob.
    17B0    0704    05/29    16:54:33.606    16    ActiveJob.cs(1081)            NORMAL     Changing state of job: '69faf761-36d4-40db-94b2-3d2d726c69cf' from 'Execute' -> 'Retire'
    17B0    0704    05/29    16:54:33.606    02    EventManager.cs(98)            NORMAL    Publishing event from ActiveJob.cs(1100): JobStateChange, [JobID=69faf761-36d4-40db-94b2-3d2d726c69cf]
    17B0    0704    05/29    16:54:33.606    16    Task.cs(235)            NORMAL     Changing task state from 'Failed' -> 'Retire' (e07e85c9-af69-4cfe-80a7-c4e12e72b2a2)
    17B0    0704    05/29    16:54:33.622    01    TaskExecutor.cs(179)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    NORMAL    RetireWorkplan(job=69faf761-36d4-40db-94b2-3d2d726c69cf) - Calling derived RetireWorkplan
    17B0    0704    05/29    16:54:33.622    01    TaskExecutor.cs(225)        E07E85C9-AF69-4CFE-80A7-C4E12E72B2A2    ACTIVITY    RetireWorkplan(job=69faf761-36d4-40db-94b2-3d2d726c69cf) - Retiring task: Failed
    17B0    0704    05/29    16:54:33.622    16    ActiveJob.cs(1201)            NORMAL     Processing end of job: 69faf761-36d4-40db-94b2-3d2d726c69cf
    17B0    0704    05/29    16:54:33.622    68    ActivityWindow.cs(101)            NORMAL     RemoveWait wait : 69faf761-36d4-40db-94b2-3d2d726c69cf
    17B0    0704    05/29    16:54:33.622    16    ActiveJob.cs(1224)            NORMAL     Final task state=Retire(e07e85c9-af69-4cfe-80a7-c4e12e72b2a2)
    17B0    0704    05/29    16:54:33.622    16    Task.cs(235)            NORMAL     Changing task state from 'Retire' -> 'Failed' (e07e85c9-af69-4cfe-80a7-c4e12e72b2a2)
    17B0    0704    05/29    16:54:33.622    16    ActiveJob.cs(1073)            FATAL     Changing state of job: '69faf761-36d4-40db-94b2-3d2d726c69cf' from 'Retire' -> 'Failed'
    17B0    0704    05/29    16:54:33.622    02    EventManager.cs(98)            NORMAL    Publishing event from ActiveJob.cs(1100): JobStateChange, [JobID=69faf761-36d4-40db-94b2-3d2d726c69cf]

    It's showing a number of CCIE failed on server messages, the servers are all part of the same domain and i've only seen references to this error for non-domain machines

    Wednesday, May 29, 2019 5:04 PM
  • Come to think of it just now, DPM only supports the protection of Hyper-V virtual machines on Cluster Shared Volumes (CSVs). Protecting other workloads hosted on CSVs isn't supported.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, May 29, 2019 11:09 PM
  • It looks like you're right, I was under the impression it was SQL on CSVs that wasn't supported but it seems that Hyper-V is the only supported workload on CSVs.

    For anyone else finding this issue, I have found a suitable workaround that seemst to work fine in my situation (your mileage may vary). I've set the preferred node to a specific node in the cluster and use DPM to backup the files by protecting the C:\ClusterStorage\Volume1, Volume2 etc. locations on that Node.

    Thursday, May 30, 2019 10:47 AM