locked
DPM 2010 - Backup domain controller of untrusted forest... RRS feed

  • Question

  • I understand that DPM 2010 is supposed to be able to backup servers in untrusted domains, this seems to work well for workstations and application servers.  When I try to backup a domain controller for an untrusted domain, DPM lists "Error" in the agent status column for that domain controller. 

    On the actual DPM server I am not seeing any errors in the event log... also I am not 100% certain what to look for in MSDPMCurr.errlog but it mostly shows normal events happening with very few warnings mixed in... and there is only one other server being backed up currently on this DPM instance.

    On the domain controller I am trying to backup I get the following error:  DPMRA Event 84 - "A DPM agent failed to communicate with the DPM service on sv-dpm1.domain.local because access is denied. Make sure that sv-dpm1.domain.local has DCOM launch and access permissions for the computer running the DPM agent (Error code: 0x80070005, full name: sv-dpm1.domain.local)."  In the DPMRACurr.errlog there are a few warnings though...

    12A4    12B8    07/19    14:43:12.980    22    agentutils.cpp(2624)            WARNING    Failed: Hr: = [0x80070645] : U: lVal : hr
    ------------ a bunch of normal events excluded here....
    12A4    131C    07/19    14:43:15.371    04    cmdproc.cpp(2017)    [0000000000481DB0]    F6B11086-F24E-4EBC-93CA-A742DC4511BD    WARNING    Failed: Hr: = [0x80070005] : F: lVal : hr
    12A4    131C    07/19    14:43:15.371    04    cmdproc.cpp(2242)    [0000000000481DB0]    F6B11086-F24E-4EBC-93CA-A742DC4511BD    WARNING    Failed: Hr: = [0x80070005] : F: lVal : CreateInstance( strCmdTarget, clsidTarget, hrDLS, (IUnknown **)&pAgentCommand, (pCommand->GetSenderToken() == 0), pCommand->IsNonDomainAgent(), fIsNonADMachine, cmdTargetIP )
    12A4    131C    07/19    14:43:15.371    04    cmdproc.cpp(2482)    [0000000000481DB0]    F6B11086-F24E-4EBC-93CA-A742DC4511BD    WARNING    CCommandProcessor::SendOutboundCommand this:[0000000000481DB0], ServerName: sv-dpm1.domain.local
    12A4    131C    07/19    14:43:15.371    04    cmdproc.cpp(2579)    [0000000000481DB0]    F6B11086-F24E-4EBC-93CA-A742DC4511BD    WARNING    Logging event for error: 257, detailed: 0x80070005

     

    The thing that is strange is that I am not configuring anything on the domain controller so it is properly configuring things so that DPMRA knows what DPM server to communicate with.  The service is also set to manual like usual and if I stop the service and then click "Refresh Information" on the DPM server, it actually launches the DPMRA service, so clearly the credentials are working fine. 

    This seems like it is something having to do with backing up a domain controller for an untrusted domain only because I can backup other servers in that domain (that aren't domain controllers) using the same DPM server without an issue. 

    Any advice on next steps to debug the issue would be appreciated...

    • Moved by Praveen D [MSFT] Tuesday, July 20, 2010 5:43 AM Moving to DPM workgroup protection Forum (From:Data Protection Manager)
    Monday, July 19, 2010 2:57 PM

Answers

  • The support incident is still open, (Many thanks to the Sr. Escalation Engineer Steve L. of MSFT who has been helping with this issue) however at this point it looks like the issue is a bug of sorts within DPM (or possibly a design decision by the developers, unsure at this time which one.)

    The basic problem is due to the authentication of the dpm agent not passing the correct information to the dpm server when it tries to authenticate because of the way our domain forest and the untrusted forest is setup.

    We are having a problem because we have the following configuration:

    dpm-sv1.domain1.local - DPM server in the DOMAIN1.LOCAL domain
    dc1.other.local - Untrusted domain controller we are trying to backup in the OTHER.LOCAL domain
    dc2.other.domain1.local - Domain controller which is in a child domain of DOMAIN1.LOCAL which has the same name as the untrusted domain controller we are trying to backup (OTHER.DOMAIN1.LOCAL)

    Basically, when the dpm agent tries to communicate with the dpm server, it negotiates the credentials rather than passing them explicitly, so when it discovers that it can use OTHER\dpmagent as the credentials, it tries that, and then obviously fails because the user was setup as DC1\dpmagent not OTHER.DOMAIN1.LOCAL\dpmagent.

     

    Workaround:
    This issue has been escalated up to development and hopefully they will have a patch or some way to properly work around, but for the time being...

    I came up with a workaround that may work for some people, although it is not a fix, and it is not something that can be relied on in all scenarios for multiple reasons. 

    If your DPM server lives in a domain (DOMAIN_A.LOCAL) and that domain has a child domain (DOMAIN_B.DOMAIN_A.LOCAL) which has the same name as the domain (DOMAIN_B.LOCAL) for an untrusted server you are trying to protect; if you have access to the DOMAIN_B.DOMAIN_A.LOCAL domain to create users, you can do the following:

     - Run setdpmserver and attach-nondomainserver.ps1 like you would normally (remember the username and password you used)
     - Open active directory users and computers on a domain controller for the DOMAIN_B.DOMAIN_A.LOCAL domain
     - Create a new user and give it the same name and password you specified in the earlier dpm commands
     - On the DPM server, open up local users and groups and add the new user to the following local groups:
        1) Distributed COM Users
        2) DPMRADmTrustedMachines
        3) MSDPMTrustedMachines

    Now when you open up the dpm management console, and go to the agents tab of the management section, you should be able to refresh the untrusted machine in question and get a status OK.

    If you do not have permission to the child domain where the user needs to be created, this obviously will not work, nor is it an elegant solution, but it is a solution nonetheless.

     

    I will update again once I get word back about the status of the ticket.

    Update 2010-12-16: My support case was recently closed with Microsoft and I wanted to update here. Unfortunately, they are not going to be fixing this issue because the development team feels the risk of creating new issues are greater than the need to resolve this. Hopefully this will work properly in the next version of DPM. :-/

    • Marked as answer by tcnolan Tuesday, July 27, 2010 3:13 PM
    • Edited by tcnolan Thursday, December 16, 2010 6:02 AM update
    Tuesday, July 27, 2010 3:12 PM

All replies

  • Ability to start/stop the DPMRA service do not guarantee that agent is configured correctly. Have you followed the below steps while attaching untrusted domain, domain controller machine? http://technet.microsoft.com/en-us/library/ff634193.aspx If not can you please go through them once? Is this domain controller is a readonly domain controller by any chance?
    Thanks, Praveen D [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Monday, July 19, 2010 5:04 PM
  • Hi Praveen, I have actually gone through those steps numerous times, albeit from memory, so I will go through the steps one-by-one and hope that I simply missed something... I will update once I do that a little later today.  To the question about the RODC, that is not the case in this situation, although I am curious if there is something to keep in mind for that kind of scenario if we ever had a mixed role server that was also an RODC that needed to be backed up...

    Thanks for the reply!

    Monday, July 19, 2010 6:44 PM
  • I went through the steps one by one, and I still have the same situation happening.

     

    I first removed the server from dpm by running: .\Remove-ProductionServer.ps1 -dpmservername sv-dpm1 -psname sv-dc1.client.local

    On the server I was looking to protect I ran: setdpmserver -dpmservername sv-dpm1.domain.local -isnondomainserver -username dpm-sv-dc1
    (which ran successfully)

    On the dpm server I ran: .\Attach-NonDomainServer.ps1 -dpmservername sv-dpm1 -psname sv-dc1.client.local -username dpm-sv-dc1 -password ****
    (also ran successfully)

    After running these commands, the agent status is still showing as "Error" and I am seeing the same error in the eventlog on the sv-dc1 server about DCOM launch permissions.

     

    Here is the full source of the error in the event log...

    Log Name:  Application
    Source:  DPMRA
    Date:   7/19/2010 3:09:17 PM
    Event ID:  84
    Task Category: None
    Level:   Error
    Keywords:  Classic
    User:   SYSTEM
    Computer:  sv-dc1.client.local
    Description:
    A DPM agent failed to communicate with the DPM service on 
    sv-dpm1.domain.local because access is denied. Make sure that 
    sv-dpm1.domain.local has DCOM launch and access permissions 
    for the computer running the DPM agent (Error code: 
    0x80070005, full name: sv-dpm1.domain.local).
    
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
     <System>
     <Provider Name="DPMRA" />
     <EventID Qualifiers="0">84</EventID>
     <Level>2</Level>
     <Task>0</Task>
     <Keywords>0x80000000000000</Keywords>
     <TimeCreated SystemTime="2010-07-19T19:09:17.000000000Z" />
     <EventRecordID>2957</EventRecordID>
     <Channel>Application</Channel>
     <Computer>sv-dc1.client.local</Computer>
     <Security UserID="S-1-5-18" />
     </System>
     <EventData>
     <Data>sv-dpm1.domain.local</Data>
     <Data>0x80070005</Data>
     <Data>sv-dpm1.domain.local</Data>
     </EventData>
    </Event>

    What would the next steps be?

    Monday, July 19, 2010 7:11 PM
  • Hi tcnolan!

    Since the logs is pointing to the DCOM object verify on the affected DC that the DPM server has the right security previliges.

    If your trying to backup workloads of another domain its a good thing of having a two-way trust between the domains for DPM to work at it's best. I have configured only protection for non-domain-attatched clients and got that to work, but I think that the easiest solution is a trust between the domains if you don't se that as a security risk.

    How does your domain trusts look like?

    BR

    Robert Hedblom

    http://robertanddpm.blogspot.com/


    Check out my DPM blog http://robertanddpm.blogspot.com
    Monday, July 19, 2010 10:15 PM
  • Unfortunately setting up a two-way trust is an absolute impossibility for these forests which is what I was hoping to take care of with DPM 2010. 

     

    I am really curious if anyone else is able to backup a domain controller in an untrusted domain... if we are unable to get this working we will likely move completely off DPM and to another product from a different vendor, it simply isn't worth the headache to stay first party with this.  Hopefully someone has a suggestion or two though...

    Tuesday, July 20, 2010 2:46 AM
  • I have done the following now and still no luck...

    - Remove untrusted domain controller entry from DPM by issuing Remove-ProductionServer.ps1
    - Uninstall DPMRA from untrusted domain controller
    - Reboot DPM server (I know the rebooting is a bit extraneous...)
    - Reboot untrusted domain controller
    - Install DPMRA agent
    - Execute SetDpmServer on untrusted domain controller
    - Execute Attach-NonDomainServer.ps1 on DPM server
    - Reboot untrusted domain controller
    - Reboot DPM server

    I am totally at a loss now... I followed the directions to the T and it still won't work.  The thing that is confusing is that the DCOM errors seem like they are calling for a trust relationship because it is asking to make sure the DPM server has launch permissions on the domain controller, but if I understand correctly, DPM (for NonDomainServers) is using a user token rather than a machine token to authenticate.

    Side note: Interestingly enough... I setup an entirely different server as a test, running as a domain controller in its own forest and I was able to add this to the DPM server following all the steps above (without the reboots) without an issue, so therefore I would have to believe the problem isn't with the DPM server itself, but with the DPMRA.

    Tuesday, July 20, 2010 4:42 AM
  • I have never tested a DC in an untrusted domain, but it seems odd that this would work as the process creates a local account which DC's shouldn't have.  But, you obviously got this to work on a different untrusted DC, so I'll throw that opinion out the window.

    Past that, check the following groups on the DC as the accoutn you create should be added:

    Distributed COM users
    DPMRADCOMTrustedMachines
    DPMRADmTrustedMachines

    I would also use dcomcnfg to look at the DPMRA permissions and compare the DC you got working for testing to the one that is failing.


    Thanks, Chris Bu - MSFT This posting is provided "AS IS" with no warranties, and confers no rights
    Tuesday, July 20, 2010 4:10 PM
  • Hi Chris,

    Thanks for the follow-up.  I checked the permissions, the group membership, dcomcnfg, all after doing a fresh install of DPMRA again, and it still has the same problem. Comparing it to the test domain controller I am not seeing anything that is different, at least not that I have checked.  Could there be settings in group policy that may be interfering?  This domain controller is a pretty basic setup so this shouldnt be a factor, but I am not even sure where to look at this point.  The logs all point to a problem with DCOM, but DCOM looks fine...

    Thanks!

    Tuesday, July 20, 2010 7:08 PM
  • Is the next step to just open an advanced incident with Microsoft?  I would hate to burn $300 on an issue that is just a setting I haven't found yet in all of my searching...
    Thursday, July 22, 2010 6:17 PM
  • Please go ahead and open a support incident, as this might need a thorough investigation using the logs.
    Thanks, Praveen D [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by Praveen D [MSFT] Friday, July 23, 2010 10:29 AM
    • Unmarked as answer by tcnolan Tuesday, July 27, 2010 3:13 PM
    Friday, July 23, 2010 10:29 AM
  • I have opened a support incident and will update the thread with the resolution in case anyone else is having the same problem...
    Friday, July 23, 2010 2:59 PM
  • The support incident is still open, (Many thanks to the Sr. Escalation Engineer Steve L. of MSFT who has been helping with this issue) however at this point it looks like the issue is a bug of sorts within DPM (or possibly a design decision by the developers, unsure at this time which one.)

    The basic problem is due to the authentication of the dpm agent not passing the correct information to the dpm server when it tries to authenticate because of the way our domain forest and the untrusted forest is setup.

    We are having a problem because we have the following configuration:

    dpm-sv1.domain1.local - DPM server in the DOMAIN1.LOCAL domain
    dc1.other.local - Untrusted domain controller we are trying to backup in the OTHER.LOCAL domain
    dc2.other.domain1.local - Domain controller which is in a child domain of DOMAIN1.LOCAL which has the same name as the untrusted domain controller we are trying to backup (OTHER.DOMAIN1.LOCAL)

    Basically, when the dpm agent tries to communicate with the dpm server, it negotiates the credentials rather than passing them explicitly, so when it discovers that it can use OTHER\dpmagent as the credentials, it tries that, and then obviously fails because the user was setup as DC1\dpmagent not OTHER.DOMAIN1.LOCAL\dpmagent.

     

    Workaround:
    This issue has been escalated up to development and hopefully they will have a patch or some way to properly work around, but for the time being...

    I came up with a workaround that may work for some people, although it is not a fix, and it is not something that can be relied on in all scenarios for multiple reasons. 

    If your DPM server lives in a domain (DOMAIN_A.LOCAL) and that domain has a child domain (DOMAIN_B.DOMAIN_A.LOCAL) which has the same name as the domain (DOMAIN_B.LOCAL) for an untrusted server you are trying to protect; if you have access to the DOMAIN_B.DOMAIN_A.LOCAL domain to create users, you can do the following:

     - Run setdpmserver and attach-nondomainserver.ps1 like you would normally (remember the username and password you used)
     - Open active directory users and computers on a domain controller for the DOMAIN_B.DOMAIN_A.LOCAL domain
     - Create a new user and give it the same name and password you specified in the earlier dpm commands
     - On the DPM server, open up local users and groups and add the new user to the following local groups:
        1) Distributed COM Users
        2) DPMRADmTrustedMachines
        3) MSDPMTrustedMachines

    Now when you open up the dpm management console, and go to the agents tab of the management section, you should be able to refresh the untrusted machine in question and get a status OK.

    If you do not have permission to the child domain where the user needs to be created, this obviously will not work, nor is it an elegant solution, but it is a solution nonetheless.

     

    I will update again once I get word back about the status of the ticket.

    Update 2010-12-16: My support case was recently closed with Microsoft and I wanted to update here. Unfortunately, they are not going to be fixing this issue because the development team feels the risk of creating new issues are greater than the need to resolve this. Hopefully this will work properly in the next version of DPM. :-/

    • Marked as answer by tcnolan Tuesday, July 27, 2010 3:13 PM
    • Edited by tcnolan Thursday, December 16, 2010 6:02 AM update
    Tuesday, July 27, 2010 3:12 PM
  • We're running into this same issue. Any word if this will be fixed in a hotfix?


    Jeff Graves, ORCS Web, Inc.
    Tuesday, August 17, 2010 1:53 PM
  • Still waiting for final word from escalation team. It's in the hands of the developers now though at least.

    Have you tried the work around I suggested? If you are having the same problem you should be able to make due with the workaround at least until a hotfix or whatever the outcome ends up being happens.
    blog TinyInt.Com | work Logicworks.Net
    Tuesday, August 17, 2010 2:59 PM
  • Actually - looks like my issue might be slightly different. I'll spin up a new thread. Thanks for the reply.
    Jeff Graves, ORCS Web, Inc.
    Tuesday, August 17, 2010 5:40 PM
  • hi, did this ever get resolved, i have the same setup and have the same problem.  Did you manage to get this working in the end?

    it seems when i try to add attach the agent it does not like the credentials.  I have tried different password to make sure it was not a typo but it must be related to the fact that the dc does not have local accounts and groups

    Thanks


    phill
    Wednesday, August 25, 2010 1:32 AM
  • Hi Phill, I still have the MS support incident opened and I believe it has been escalated all the way to the dev team at this point but no official resolution. If your setup is the same with the forest structure, try the workaround I have posted. It's not a problem with the DC not having local accounts, it is actually because the credentials being negotiated are for the wrong domain. read my full post marked as an answer, it is the only way around this I have been able to come up with, and works fine. Once the support case is closed I will post an update with resolution included.
    blog TinyInt.Com | work Logicworks.Net
    Wednesday, August 25, 2010 1:24 PM
  • HI There. While this issue is slightly different in nature than mine, I belive mine to b more basic: Domain1.local has a dpm server ( DPMServer) Domain2.local is another Domain where the servers are that I want to protect. I have a 2 way, selective, trust in place, where Domain2.local trusts Domain1.local, but not the other way around. The two domain controllers in Domain2.local are not able to be backed up by DPM. What I have found is that running the SetDpmServer -dpmservername dpmserver.domain1.local -IsNonDomainServer -Username command on the DC, creates the user account as per normal. The problem is that this: the user creation process, when run on a member server, puts the user into a local users group on a member server, however on a DC, there is no such animal; so it compensates by making a domain user in the USERS container.... This invariably throws up an error to do with DCOM launch and access permissions. I, too have a Support incident opened up with Microsoft, and am questioning how they can say you can back up untrusted domains, when you can't include the domain controllers in that protection ??? It just does not makes sense to me....
    Monday, December 6, 2010 5:35 PM
  • @Rick, it sounds like you are having a separate issue.  I actually have multiple domain controllers in multiple untrusted domains being protected by a single DPM server in one instance and it works fine.  When I was working on my case with MSFT, one of the things we checked were DCOM permissions, but these are pretty straight forward... did you have any luck with them yet?

    By the way, as far as the user creation is concerned, if you are protecting multiple domain controllers for the same domain, you will need to use different user names for each one.  This was a "gotcha" that I encountered while working through untrusted DC protection, although it doesn't seem to be an issue _every_ time.  I like to use the following very simple naming convention though for the usernames to avoid conflict/confusion: dpmsa-<servername>


    blog TinyInt.Com | work Logicworks.Net
    • Proposed as answer by phillbl Wednesday, March 16, 2011 9:11 PM
    • Unproposed as answer by phillbl Wednesday, March 16, 2011 9:11 PM
    Thursday, December 16, 2010 6:10 AM
  • sorry for the late reply!..... fixed it in the end.  realized that the sites in AD sites and services were not setup properly... created sites and double checked.  seemed to be an issue when a computer was in a different office on a different network but had authenticated to a server in a different office.  once i seperated the servers and got users logging on using their local dc everything started to work.

     


    phill
    • Proposed as answer by phillbl Wednesday, March 16, 2011 9:11 PM
    • Unproposed as answer by tcnolan Thursday, March 17, 2011 3:13 PM
    Wednesday, March 16, 2011 9:11 PM
  • HI There. While this issue is slightly different in nature than mine, I belive mine to b more basic: Domain1.local has a dpm server ( DPMServer) Domain2.local is another Domain where the servers are that I want to protect. I have a 2 way, selective, trust in place, where Domain2.local trusts Domain1.local, but not the other way around. The two domain controllers in Domain2.local are not able to be backed up by DPM. What I have found is that running the SetDpmServer -dpmservername dpmserver.domain1.local -IsNonDomainServer -Username command on the DC, creates the user account as per normal. The problem is that this: the user creation process, when run on a member server, puts the user into a local users group on a member server, however on a DC, there is no such animal; so it compensates by making a domain user in the USERS container.... This invariably throws up an error to do with DCOM launch and access permissions. I, too have a Support incident opened up with Microsoft, and am questioning how they can say you can back up untrusted domains, when you can't include the domain controllers in that protection ??? It just does not makes sense to me....

    I know this is years old but did you ever get this fixed?  I have the same problem with backing up a DC in a remote domain
    Wednesday, March 27, 2013 3:17 PM
  • decided to backup the domain controller using windows backup to a member server.  Then used DPM to backup the member server.

    Looks like it's the only way it can be done

    Monday, June 17, 2013 2:01 PM