none
DPM 2010 backup of System State fails - but WSB backup of System State works

    Question

  • My problem is similar to that described in the thread "DPM Backup of System State failed" posted Thursday, April 30, 2009 11:43 AM.

    In DPM alert:
    "DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 546, WSB Error Code:  0x80780079). (ID 30229 Details: Internal error code: 0x809909FB)"

    The affected machine is running WSS (Windows Storage Server) 2008 SP2 and has WSB (Windows Server Backup) installed.  On that server, when I run

    wbadmin start systemstatebackup -backuptarget:C:

    The system state backup succeeds and there are are no errors.   However if DPM runs the system state backup as per the error seen in DPM above, on the WSS2008 server there is an error in the WSB logs that says:

    Log Name:      Microsoft-Windows-Backup
    Source:        Microsoft-Windows-Backup
    Date:          7/13/2011 11:16:58 AM
    Event ID:      19
    Task Category: None
    Level:         Error
    Keywords:     
    User:          SYSTEM
    Computer:      WSS2008
    Description:
    Backup attempted at '7/13/2011 10:16:58 AM' failed to start, error code '2155348089'.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Backup" Guid="{1db28f2e-8f80-4027-8c5a-a11f7f10f62d}" />
        <EventID>19</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x4000000000000000</Keywords>
        <TimeCreated SystemTime="2011-07-13T10:16:58.498Z" />
        <EventRecordID>10</EventRecordID>
        <Correlation />
        <Execution ProcessID="2332" ThreadID="1976" />
        <Channel>Microsoft-Windows-Backup</Channel>
        <Computer>WSS2008</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
        <Data Name="HRESULT">2155348089</Data>
        <Data Name="BackupTime">2011-07-13T10:16:58.498Z</Data>
        <Data Name="BackupTarget">D:</Data>
        <Data Name="BackupTime">2011-07-13T10:16:58.498Z</Data>
        <Data Name="HRESULT">2155348089</Data>
        <Data Name="NumOfVolumes">0</Data>
        <Data Name="VolumeNames">
        </Data>
      </EventData>
    </Event>

    The other thread suggests checking VSS writers but I have been through the output of DISKSHADOW LIST WRITERS and I cannot find any paths that match patterns such as "\..\" or "\\" or "/" that might indicate an invalid path in the registry.

    Can anyone help?

    Thanks


    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    • Moved by MarcReynolds Thursday, July 14, 2011 12:40 PM (From:Data Protection Manager)
    Thursday, July 14, 2011 8:11 AM

Answers

All replies

  • Hi,

    Note the error shows: <Data Name="BackupTarget">D:</Data>

    Please change the location of the backup file to C: by modifying the PSDatasourceConfig.xml and see if that works:

    Change the Location of the Backup File
    http://technet.microsoft.com/en-us/library/bb809015.aspx

     


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, July 15, 2011 12:09 AM
  • Hi,

    You are correct Mike, when I try:

    wbadmin start systemstatebackup -backuptarget:D:

    I get:

    Error while preparing for backup to your backup storage location.
    Internal Error:  Could not create diff area file on backup target volume.
    The specified backup storage location has the shadow copy storage residing on an
    other volume.

    Thank you for pointing me in the right direction, however the link and instructions you refer to are for DPM 2007, and I am using DPM 2010.  Your instructions were:

    To change the location of the system state backup file

    1. On the protected computer, open PSDatasourceConfig.xml in an XML or text editor. PSDatasourceConfig.xml is typically located at install path\Program Files\Microsoft Data Protection Manager\DPM\Datasources.

    2. Change the <FilesToProtect> value from %systemdrive% to the desired location.

    3. Save the file.

    4. On the DPM server, if there is a protection group protecting the system state of the protected computer in step 1, run a consistency check.

    5. The consistency check will fail and generate an alert. Perform the recommended actions in the alert as follows:

      1. In the alert details, click the Modify protection group link, and then step through the wizard.
      2. Perform a consistency check.

    However my PSDatasourceConfig.xml file has no value inside its <FilesToProtect> tags.  Here is my file:

      <?xml version="1.0" encoding="utf-8" ?>
    - <PSDatasourcesConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.microsoft.com/2003/dls/PSDatasourceConfig.xsd">
    - <DatasourceConfig>
      <WriterId>7B2E4C3D-876F-4535-B5C9-B344D4F05120</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer</LogicalPath>
      <ComponentName>DisconnectedClient</ComponentName>
      <FilesToProtect />
      <FilesToExclude />
      <FileSpecRecursive>true</FileSpecRecursive>
      <Size>1024</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
    - <DatasourceConfig>
      <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer</LogicalPath>
      <ComponentName>System Protection</ComponentName>
      <FilesToProtect>C:\WindowsImageBackup\*</FilesToProtect>
      <Size>51200</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
    - <DatasourceConfig>
      <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer\System Protection</LogicalPath>
      <ComponentName>System State</ComponentName>
      <FilesToProtect />
      <Size>10240</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
    - <DatasourceConfig>
      <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer\System Protection</LogicalPath>
      <ComponentName>Bare Metal Recovery</ComponentName>
      <FilesToProtect />
      <Size>51200</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
      </PSDatasourcesConfig>
     
    Do I change the <FilesToProtect/> I marked green above to
    <FilesToProtect>%systemdrive%</FilesToProtect>
    or
    <FilesToProtect>C:</FilesToProtect>
    because for all my servers C: would always be a valid place to do the System State Backup, and %systemdrive% is always C: on these protected servers.  I will try this tonight.

    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Friday, July 15, 2011 9:36 AM
  • Hi,

     

    Hm - it looks like it's already configured to be on C: - which is strange.

     <FilesToProtect>C:\WindowsImageBackup\*</FilesToProtect>

    Another option would be to move the shadow storage for D: back onto D: using vssadmin.

    C:\>vssadmin add shadowstorage /for=D: /on=D: /maxsize=10%


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, July 15, 2011 3:10 PM
  • It looks like it is still trying to go to D: as seen in the Error event on my WSS2008 server:

    Log Name:      Microsoft-Windows-Backup
    Source:        Microsoft-Windows-Backup
    Date:          7/14/2011 4:33:50 AM
    Event ID:      19
    Task Category: None
    Level:         Error
    Keywords:     
    User:          SYSTEM
    Computer:      WSS2008
    Description:
    Backup attempted at '7/14/2011 3:33:50 AM' failed to start, error code '2155348089'.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Backup" Guid="{1db28f2e-8f80-4027-8c5a-a11f7f10f62d}" />
        <EventID>19</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x4000000000000000</Keywords>
        <TimeCreated SystemTime="2011-07-14T03:33:50.143Z" />
        <EventRecordID>11</EventRecordID>
        <Correlation />
        <Execution ProcessID="5068" ThreadID="4992" />
        <Channel>Microsoft-Windows-Backup</Channel>
        <Computer>WSS2008</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
        <Data Name="HRESULT">2155348089</Data>
        <Data Name="BackupTime">2011-07-14T03:33:50.143Z</Data>
        <Data Name="BackupTarget">D:</Data>
        <Data Name="BackupTime">2011-07-14T03:33:50.143Z</Data>
        <Data Name="HRESULT">2155348089</Data>
        <Data Name="NumOfVolumes">0</Data>
        <Data Name="VolumeNames">
        </Data>
      </EventData>
    </Event>

    This is my current PSDatasourceConfig.xml file:

      <?xml version="1.0" encoding="utf-8" ?>
    - <PSDatasourcesConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.microsoft.com/2003/dls/PSDatasourceConfig.xsd">
    - <DatasourceConfig>
      <WriterId>7B2E4C3D-876F-4535-B5C9-B344D4F05120</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer</LogicalPath>
      <ComponentName>DisconnectedClient</ComponentName>
      <FilesToProtect />
      <FilesToExclude />
      <FileSpecRecursive>true</FileSpecRecursive>
      <Size>1024</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
    - <DatasourceConfig>
      <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer</LogicalPath>
      <ComponentName>System Protection</ComponentName>
      <FilesToProtect>C:\WindowsImageBackup\*</FilesToProtect>
      <Size>51200</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
    - <DatasourceConfig>
      <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer\System Protection</LogicalPath>
      <ComponentName>System State</ComponentName>
      <FilesToProtect>C:</FilesToProtect>
      <Size>10240</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
    - <DatasourceConfig>
      <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer\System Protection</LogicalPath>
      <ComponentName>Bare Metal Recovery</ComponentName>
      <FilesToProtect />
      <Size>51200</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
      </PSDatasourcesConfig>
     
    Any ideas?

    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Tuesday, July 19, 2011 1:29 PM
  • I moved my shadow copies of D: back to D: and tried again. It fails now for another reason:

    C:\Windows\system32>wbadmin start systemstatebackup -backuptarget:D:
    wbadmin 1.0 - Backup command-line tool
    (C) Copyright 2004 Microsoft Corp.

    Starting System State Backup [7/28/2011 4:06 PM]
    Retrieving volume information...

    This would backup the system state from volume(s) OS(C:) to D:.
    Do you want to start the backup operation?
    [Y] Yes [N] No Y

    Creating the shadow copy of volumes requested for backup.
    Identifying system state files to backup (This may take a few minutes)...
    Found (1573) files
    Found (5632) files
    Found (11304) files
    Found (16165) files
    Found (21382) files
    Found (26871) files
    Found (40908) files
    Found (49726) files
    Found (60192) files
    Found (72822) files
    Found (86516) files
    Found (97840) files
    Found (97973) files

    Summary of backup:
    ------------------

    Backup of system state failed [7/28/2011 4:09 PM]

    Log of files successfully backed up
    'C:\Windows\Logs\WindowsServerBackup\SystemStateBackup 28-07-2011 16-06-35.log'

    Log of files for which backup failed
    'C:\Windows\Logs\WindowsServerBackup\SystemStateBackup_Error 28-07-2011 16-06-35
    .log'


    There was a failure in preparing the backup image of one of the volumes in the b
    ackup set.

    Failed to format media.

    My problem is still unresolved.


    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Thursday, July 28, 2011 3:13 PM
  • Interestingly, after a reboot of my WSS2008 server yesterday, the System State Backup suceeded last night (onto the D: drive), so if you get the error "Failed to format media" I would try a reboot.

    Still no solution from Microsoft as to why I cannot select the drive that the System State Backup is saved to.  The suggested changes to PSDatasourceConfig.xml did not work.  Now I am forced to have it on my D: drive and because of that I have to keep the shadow copies of my D: drive on the same drive, which is not what we wanted.  But at least now we have a working backup of this server.


    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Friday, July 29, 2011 8:11 AM
  • Hi,

    Concerning the problem with the path to the backup drive check the following:

    Make sure the installpath is correct in the registry under: 
        HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Setup

    Make sure SYSTEM account has full control of the file:
        C:\Program Files\Microsoft Data Protection Manager\DPM\Datasources\PSDataSourceConfig.xml

    Make sure the C:\Program Files\Microsoft DPM\DPM\bin\LHSystemStateBackup.cmd file is not hardcoded to use D: - it should read as follows to accept the variable %1 which is read from the PSDataSourceConfig.xml to point to C:\WindowsImageBackup

       rem Start a wbadmin.exe to create a systemstate backup
       start  /WAIT %SystemRoot%\system32\wbadmin.exe start systemstatebackup -quiet -backuptarget:%1

     


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Monday, August 01, 2011 12:34 AM
  • I've just checked all of those items.  They all appear as you said they should above.

    It gives me a workaround in that C:\Program Files\Microsoft DPM\DPM\bin\LHSystemStateBackup.cmd file could be hardcoded to use C: by changing that line to

       start  /WAIT %SystemRoot%\system32\wbadmin.exe start systemstatebackup -quiet -backuptarget:C:

    This would work for this particular installation of DPM because in this case I do want the System State Backup to always got to C:.  I will try it tonight and post here if it works.

    The way I understand it is that we can either
    (a) Let DPM decide which drive to use (one with most free space)
    (b) hardcode the drive used for ALL protected servers
    and that we cannot specify/hardcode drives to use on a per-server basis, i.e. you cannot force the use of C: on SERVER1 and force the use of D: on SERVER2.


    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Monday, August 01, 2011 9:31 AM
  • I think I would troubleshoot why the setting in the PSDataSourceConfig.xml is not being honored.  Did you follow ALL the steps outlined in the technet article - did you get the alert and follow the link ?

     

    To change the location of the system state backup file

    1. On the protected computer, open PSDatasourceConfig.xml in an XML or text editor. PSDatasourceConfig.xml is typically located at install path\Program Files\Microsoft Data Protection Manager\DPM\Datasources.

    2. Change the <FilesToProtect> value from %systemdrive% to the desired location.

    3. Save the file.

    4. On the DPM server, if there is a protection group protecting the system state of the protected computer in step 1, run a consistency check.

    5. The consistency check will fail and generate an alert. Perform the recommended actions in the alert as follows:

      1. In the alert details, click the Modify protection group link, and then step through the wizard.

      2. Perform a consistency check.

     

    By Default, DPM will use the drive with the most free space "at the time of protection" to store the systemstate backup files.  Once configured, it will not change, however, by you modifying the PSDatasourceConfig.xml on each machine, you have direct control over where it gets moved to.


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Monday, August 01, 2011 3:07 PM
  • I had not run the consistency check before.  I have just tried to run it on both my protection groups.  I right click on my Protection Group and select "run consistency check".  However no "Consistency Check" job appears under "Monitoring -> Jobs" (under a filter I created to show all such jobs) and no alert was generated.  I got a message saying "Consistency check would be performed for all those members which are in the inconsisitent state in the selected protection group."  However under the Protection tab, no members were in the "Inconsistent" state.  Does this mean that the consistency check is not running because DPM decides it is not needed? 

    Interestingly, despite the fact that I changed C:\Program Files\Microsoft DPM\DPM\bin\LHSystemStateBackup.cmd to use C: by changing that line to

       start  /WAIT %SystemRoot%\system32\wbadmin.exe start systemstatebackup -quiet -backuptarget:C:

    ...it still shows in the event log that it went to D: on my WSS2008 server.


    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Wednesday, August 03, 2011 10:44 AM
  • Hi,

    1) The CC ran at the PG level will only run a CC against a datasource that is inconsistent.  A CC ran on a data source will run unconditionally.

    2)  It seems that the C:\Program Files\Microsoft DPM\DPM\bin\LHSystemStateBackup.cmd file is not the one being used, Are there any other copies of that file ?   If you rename it, does the DPM backup fail ? 


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, August 03, 2011 5:37 PM
  • 1) The CC ran at the PG level will only run a CC against a datasource that is inconsistent.  A CC ran on a data source will run unconditionally.

    How do I run a CC against a data source?  On the protection tab if I select the item "Computer\System Protection\System State" for that server, in the action pane the "perform consistency check" option is greyed out.  Same if I right-click on that item.

    As regards LHSystemStateBackup.cmd I will rename it tonight and will see if the System State backup fails.


    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Thursday, August 04, 2011 12:38 PM
  • Hi,

    The only condition that I can think of that would make the option to run a CC be grayed out is if the data souce is in inactive protection.  


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, August 04, 2011 4:51 PM
  • After renaming LHSystemStateBackup.cmd to LHSystemStateBackup_renamed.cmd my system state backup failed with the following error:

    Computer\System Protection (DPMServer)

    DPM failed to create the backup. If you are backing up only System State, verify if there is enough free space on the protected computer to store the System State backup.  On protected computers running Windows Server2008, verify that Windows Server Backup (WSB) is installed and that it is not performing any other backup or recovery task. (ID 30214 Details: Internal error code: 0x809909FB) You can see the errors reported by WSB in the Eventlog on the protected computer DPMServer. For resolution actions and more information on these errors, go to http://technet.microsoft.com/en-us/library/cc734488(WS.10).aspx.

     

    This is my DPM server, not the WSS2008 target I was trying to backup, so I now realise that each server on which an agent is deployed has its own PSDatasourceConfig.xml and LHSystemStateBackup.cmd and you meant I should change the files on the server with the agent where the problem was. That at least explains that.

    Still do not understand why I cannot run a CC manually on a datasource and I can find no documentaion on running a CC manually in DPM 2010 on Technet.  I only have 2 protection groups and both are in use.


    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Friday, August 05, 2011 10:46 AM
  • This is PSDatasourceConfig.xml the from the protected computer, as you can see D: was specified explicitly:

      <?xml version="1.0" encoding="utf-8" ?>
    - <PSDatasourcesConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.microsoft.com/2003/dls/PSDatasourceConfig.xsd">
    - <DatasourceConfig>
      <WriterId>7B2E4C3D-876F-4535-B5C9-B344D4F05120</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer</LogicalPath>
      <ComponentName>DisconnectedClient</ComponentName>
      <FilesToProtect />
      <FilesToExclude />
      <FileSpecRecursive>true</FileSpecRecursive>
      <Size>1024</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
    - <DatasourceConfig>
      <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer</LogicalPath>
      <ComponentName>System Protection</ComponentName>
      <FilesToProtect>D:\WindowsImageBackup\*</FilesToProtect>
      <Size>51200</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
    - <DatasourceConfig>
      <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer\System Protection</LogicalPath>
      <ComponentName>System State</ComponentName>
      <FilesToProtect />
      <Size>10240</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
    - <DatasourceConfig>
      <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>
      <Version>01.00.0000.00</Version>
      <VssWriterInvolved>false</VssWriterInvolved>
      <LogicalPath>Computer\System Protection</LogicalPath>
      <ComponentName>Bare Metal Recovery</ComponentName>
      <FilesToProtect />
      <Size>51200</Size>
      <UseDRWithCC>true</UseDRWithCC>
      </DatasourceConfig>
      </PSDatasourcesConfig>
     
    I have changed that to D: to C: now and will reply Monday if that has fixed it.
     
    My apologies for any confusion!!

    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    • Proposed as answer by ShaneB. _ Tuesday, August 16, 2011 3:39 PM
    Friday, August 05, 2011 10:56 AM
  • Hello,

    Does this issue still exist?


    Thanks
    Shane


    Tuesday, August 09, 2011 10:35 PM
  • The System State backups are now going to C:, and it is succeeding.  The issue is resolved.

    Thanks,

    Richard


    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Wednesday, August 10, 2011 8:04 AM
  • Hello,

    Does this issue still exists?


    Thanks
    Shane
    Friday, August 12, 2011 1:21 PM
  • Hello,

    Does this issue still exists?


    Thanks
    Shane


    No, it is resolved.  Mike gave me the correct solution.  Case closed!

    Thanks


    Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
    Friday, August 12, 2011 1:30 PM
  •    I have the same problem - System state consistency checks adn recoery points fail on the DPM server's System State, but is suscessful using WSB. In addition, many of my client server recovery points started failing at the same time - same error.  The protection agent on TRGAZDPM01.ryland.com was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)

     My PSDataSourceConfig.xml file does not have the wrong drive.  The storage pool isn't anywhere near full. I am getting LOTS of .CRASH files from the DPMRA.  Here are pertinent pieces of an error log:

    xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2137454160" Reason="Error" CommandID="RAGetVolumeStatistics" CommandInstanceID="67629372-2b19-476a-aa50-9e5fd3932af7" GuidWorkItem="cd0a882f-f0c2-479d-a1e8-ba1a0e2308b3" TETaskInstanceID="d4e97f4e-2326-42c3-b79b-50489d27dc51"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="2001" DetailedCode="-2137454160" DetailedSource="2"><Parameter Name="AgentTargetServer" Value="TRGAZDPM01.ryland.com"/></ErrorInfo></Status>

    106C 1708 08/25 03:10:57.859    03   runtime.cpp(1599)    [000000000049C000]     D4E97F4E-2326-42C3-B79B-50489D27DC51 FATAL ]

    106C 1708 08/25 03:10:57.859    05   defaultsubtask.cpp(680)    [0000000000551640]     D4E97F4E-2326-42C3-B79B-50489D27DC51 WARNING    Failed: Hr: = [0x809909b0] : F: lVal : CommandReceived(pAgentOvl)

    106C 1708 08/25 03:10:57.869    29   radefaultsubtask.cpp(356)  [0000000000551640]     D4E97F4E-2326-42C3-B79B-50489D27DC51 WARNING    Failed: Hr: = [0x809909b0] : CRADefaultSubTask: WorkitemID does not exist, {CD0A882F-F0C2-479D-A1E8-BA1A0E2308B3}

    106C 1708 08/25 03:10:57.869    05   defaultsubtask.cpp(459)    [0000000000551640]     D4E97F4E-2326-42C3-B79B-50489D27DC51 WARNING    Failed: Hr: = [0x809909b0] : F: lVal : CommandReceivedSpecific(pCommand, pOvl)

    106C 1708 08/25 03:10:57.869    03   runtime.cpp(1599)    [000000000049C000]     D4E97F4E-2326-42C3-B79B-50489D27DC51 FATAL Subtask failure, sending status response XML=[<?xml version="1.0"?>

    106C 1708 08/25 03:10:57.869    03   runtime.cpp(1599)    [000000000049C000]     D4E97F4E-2326-42C3-B79B-50489D27DC51 FATAL <Status

    AND:

         [000000000050D480]         NORMAL     CreateNewAcceptSocket called for address family: 2

    106C 16E0 08/25 03:15:49.225    20   cc_base.cpp(913)                NORMAL     DM: - Spotted QoS IPv6, Index:4

    106C 16E0 08/25 03:15:49.225    20   cc_base.cpp(908)                NORMAL     DM: - Spotted QoS IPv4, Index:5

    106C 16E0 08/25 03:15:49.225    20   cc_base.cpp(1036)               NORMAL     Created Socket with Family: 2, QoS Index: 5

    106C 16E0 08/25 03:15:49.225    20   cc_base.cpp(761)     [0000000000553070]        NORMAL     DM: - EndPoint:000000000050D480, Family: 2, Indx:5, Port:0

    106C 1650 08/25 03:15:59.139    03   runtime.cpp(1652)    [000000000049C000]        NORMAL     CDLSRuntime::ProcessIdleTimeout

    106C 1650 08/25 03:15:59.139    03   runtime.cpp(646)     [000000000049C000]        NORMAL     CDLSRuntime::Uninitialize, bForce: 0

    106C 1650 08/25 03:15:59.139    05   genericagent.cpp(264) [0000000000499960]        NORMAL     Agent Can Shutdown if there is only default wokitem active[1]

    106C 1650 08/25 03:15:59.139    29   dpmra.cpp(360)  [0000000000499960]        NORMAL     CDPMRA::Shutting down dpmra, force-shutdown :yes

    106C 1650 08/25 03:15:59.139    03   workitem.cpp(391)               NORMAL     Timing out WI [000000000045F150], WI GUID = {B71B4544-7067-4A30-B5FB-BA320B10D82A}, ..last DM activity happened 67340670msec back, WI Idle Timeout = 390000msec

    106C 1650 08/25 03:15:59.139    22   genericthreadpool.cpp(664) [000000000054E930]           NORMAL     CGenericThreadPool: Waiting for threads to exit

    106C 16E0 08/25 03:16:01.139    20   cc_base.cpp(1226)    [0000000000555360]        NORMAL     DM: Aborting  http state machine for reason 0

    106C 07F4 08/25 03:16:01.139    20   cc_base.cpp(1226)    [000000000050D620]        NORMAL     DM: Aborting  http state machine for reason 0

    106C 1024 08/25 03:16:01.140    20   cc_base.cpp(1226)    [000000000050F950]        NORMAL     DM: Aborting  http state machine for reason 0

    106C 1788 08/25 03:16:01.140    20   cc_base.cpp(1226)    [0000000000512CC0]        NORMAL     DM: Aborting  http state machine for reason 0

    106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [0000000000512CC0]        NORMAL     DM: Going to timeout connection

    106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [0000000000519590]        NORMAL     DM: Going to timeout connection

    106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [000000000051B880]        NORMAL     DM: Going to timeout connection

    106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [000000000051DB70]        NORMAL     DM: Going to timeout connection

    106C 07F4 08/25 03:16:01.140    20   cc_base.cpp(1226)    [000000000051FE60]        NORMAL     DM: Aborting  http state machine for reason 0

    106C 16E0 08/25 03:16:01.140    20   cc_base.cpp(1226)    [0000000000553070]        NORMAL     DM: Aborting  http state machine for reason 7471221

    106C 1024 08/25 03:16:01.140    20   cc_base.cpp(1226)    [0000000000522150]        NORMAL     DM: Aborting  http state machine for reason 0

    106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [000000000051FE60]        NORMAL     DM: Going to timeout connection

    106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [0000000000522150]        NORMAL     DM: Going to timeout connection

    106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [0000000000553070]        NORMAL     DM: Going to timeout connection

    106C 1788 08/25 03:16:01.140    20   cc_base.cpp(1226)    [000000000051DB70]        NORMAL     DM: Aborting  http state machine for reason 0

    106C 1024 08/25 03:16:01.140    20   cc_base.cpp(1226)    [0000000000519590]        NORMAL     DM: Aborting  http state machine for reason 0

    106C 07F4 08/25 03:16:01.140    20   cc_base.cpp(1226)    [000000000051B880]        NORMAL     DM: Aborting  http state machine for reason 0

    106C 1650 08/25 03:16:01.142    22   genericthreadpool.cpp(664) [000000000049C000]           NORMAL     CGenericThreadPool: Waiting for threads to exit

    106C 11D4 08/25 03:16:03.143    03   timer.cpp(513)  [0000000000524F68]        ACTIVITY   Shutting down timer thread.

    106C 1650 08/25 03:16:03.143    03   service.cpp(81)            ACTIVITY     CService::StopThisService

    106C 1650 08/25 03:16:03.143    03   service.cpp(281)     [000000000022FA00]        ACTIVITY   CService::StopService

    This just started last Friday - everything was smooth until then, and I am not aware of any changes made to the DPM server. 

    Thursday, August 25, 2011 3:50 AM