DPM 2010 backup of System State fails - but WSB backup of System State works

All replies

• 

  

  0

  Sign in to vote

  Hi,

  Note the error shows: <Data Name="BackupTarget">D:</Data>

  Please change the location of the backup file to C: by modifying the PSDatasourceConfig.xml and see if that works:

  Change the Location of the Backup File
  http://technet.microsoft.com/en-us/library/bb809015.aspx

   

  ---

  Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

  • Proposed as answer by Mike JacquetMicrosoft employee Friday, July 15, 2011 12:09 AM
  • Marked as answer by Richard J Gillis Wednesday, August 10, 2011 8:02 AM

  Friday, July 15, 2011 12:09 AM
• 

  

  0

  Sign in to vote

  Hi,

  You are correct Mike, when I try:
  
  wbadmin start systemstatebackup -backuptarget:D:
  
  I get:
  
  Error while preparing for backup to your backup storage location.
  Internal Error:  Could not create diff area file on backup target volume.
  The specified backup storage location has the shadow copy storage residing on an
  other volume.
  
  Thank you for pointing me in the right direction, however the link and instructions you refer to are for DPM 2007, and I am using DPM 2010.  Your instructions were:

  To change the location of the system state backup file

  1. On the protected computer, open PSDatasourceConfig.xml in an XML or text editor. PSDatasourceConfig.xml is typically located at install path\Program Files\Microsoft Data Protection Manager\DPM\Datasources.

  2. Change the <FilesToProtect> value from %systemdrive% to the desired location.

  3. Save the file.

  4. On the DPM server, if there is a protection group protecting the system state of the protected computer in step 1, run a consistency check.

  5. The consistency check will fail and generate an alert. Perform the recommended actions in the alert as follows:

     1. In the alert details, click the Modify protection group link, and then step through the wizard.
     2. Perform a consistency check.

  However my PSDatasourceConfig.xml file has no value inside its <FilesToProtect> tags.  Here is my file:

    <?xml version="1.0" encoding="utf-8" ?>

  - <PSDatasourcesConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.microsoft.com/2003/dls/PSDatasourceConfig.xsd">

  - <DatasourceConfig>

    <WriterId>7B2E4C3D-876F-4535-B5C9-B344D4F05120</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer</LogicalPath>

    <ComponentName>DisconnectedClient</ComponentName>

    <FilesToProtect />

    <FilesToExclude />

    <FileSpecRecursive>true</FileSpecRecursive>

    <Size>1024</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

  - <DatasourceConfig>

    <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer</LogicalPath>

    <ComponentName>System Protection</ComponentName>

    <FilesToProtect>C:\WindowsImageBackup\*</FilesToProtect>

    <Size>51200</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

  - <DatasourceConfig>

    <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer\System Protection</LogicalPath>

    <ComponentName>System State</ComponentName>

    <FilesToProtect />

    <Size>10240</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

  - <DatasourceConfig>

    <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer\System Protection</LogicalPath>

    <ComponentName>Bare Metal Recovery</ComponentName>

    <FilesToProtect />

    <Size>51200</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

    </PSDatasourcesConfig>

   

  Do I change the <FilesToProtect/> I marked green above to

  <FilesToProtect>%systemdrive%</FilesToProtect>

  or

  <FilesToProtect>C:</FilesToProtect>

  because for all my servers C: would always be a valid place to do the System State Backup, and %systemdrive% is always C: on these protected servers.  I will try this tonight.

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  Friday, July 15, 2011 9:36 AM
• 

  

  0

  Sign in to vote

  Hi,

   

  Hm - it looks like it's already configured to be on C: - which is strange.

   <FilesToProtect>C:\WindowsImageBackup\*</FilesToProtect>

  Another option would be to move the shadow storage for D: back onto D: using vssadmin.

  C:\>vssadmin add shadowstorage /for=D: /on=D: /maxsize=10%

  ---

  Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

  Friday, July 15, 2011 3:10 PM
• 

  

  0

  Sign in to vote

  It looks like it is still trying to go to D: as seen in the Error event on my WSS2008 server:

  Log Name:      Microsoft-Windows-Backup
  Source:        Microsoft-Windows-Backup
  Date:          7/14/2011 4:33:50 AM
  Event ID:      19
  Task Category: None
  Level:         Error
  Keywords:     
  User:          SYSTEM
  Computer:      WSS2008
  Description:
  Backup attempted at '7/14/2011 3:33:50 AM' failed to start, error code '2155348089'.
  Event Xml:
  <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
      <Provider Name="Microsoft-Windows-Backup" Guid="{1db28f2e-8f80-4027-8c5a-a11f7f10f62d}" />
      <EventID>19</EventID>
      <Version>0</Version>
      <Level>2</Level>
      <Task>0</Task>
      <Opcode>0</Opcode>
      <Keywords>0x4000000000000000</Keywords>
      <TimeCreated SystemTime="2011-07-14T03:33:50.143Z" />
      <EventRecordID>11</EventRecordID>
      <Correlation />
      <Execution ProcessID="5068" ThreadID="4992" />
      <Channel>Microsoft-Windows-Backup</Channel>
      <Computer>WSS2008</Computer>
      <Security UserID="S-1-5-18" />
    </System>
    <EventData>
      <Data Name="HRESULT">2155348089</Data>
      <Data Name="BackupTime">2011-07-14T03:33:50.143Z</Data>
      <Data Name="BackupTarget">D:</Data>
      <Data Name="BackupTime">2011-07-14T03:33:50.143Z</Data>
      <Data Name="HRESULT">2155348089</Data>
      <Data Name="NumOfVolumes">0</Data>
      <Data Name="VolumeNames">
      </Data>
    </EventData>
  </Event>

  This is my current PSDatasourceConfig.xml file:

    <?xml version="1.0" encoding="utf-8" ?>

  - <PSDatasourcesConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.microsoft.com/2003/dls/PSDatasourceConfig.xsd">

  - <DatasourceConfig>

    <WriterId>7B2E4C3D-876F-4535-B5C9-B344D4F05120</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer</LogicalPath>

    <ComponentName>DisconnectedClient</ComponentName>

    <FilesToProtect />

    <FilesToExclude />

    <FileSpecRecursive>true</FileSpecRecursive>

    <Size>1024</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

  - <DatasourceConfig>

    <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer</LogicalPath>

    <ComponentName>System Protection</ComponentName>

    <FilesToProtect>C:\WindowsImageBackup\*</FilesToProtect>

    <Size>51200</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

  - <DatasourceConfig>

    <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer\System Protection</LogicalPath>

    <ComponentName>System State</ComponentName>

    <FilesToProtect>C:</FilesToProtect>

    <Size>10240</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

  - <DatasourceConfig>

    <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer\System Protection</LogicalPath>

    <ComponentName>Bare Metal Recovery</ComponentName>

    <FilesToProtect />

    <Size>51200</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

    </PSDatasourcesConfig>

   

  Any ideas?

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  Tuesday, July 19, 2011 1:29 PM
• 

  

  0

  Sign in to vote

  I moved my shadow copies of D: back to D: and tried again. It fails now for another reason:

  C:\Windows\system32>wbadmin start systemstatebackup -backuptarget:D:
  wbadmin 1.0 - Backup command-line tool
  (C) Copyright 2004 Microsoft Corp.

  Starting System State Backup [7/28/2011 4:06 PM]
  Retrieving volume information...

  This would backup the system state from volume(s) OS(C:) to D:.
  Do you want to start the backup operation?
  [Y] Yes [N] No Y

  Creating the shadow copy of volumes requested for backup.
  Identifying system state files to backup (This may take a few minutes)...
  Found (1573) files
  Found (5632) files
  Found (11304) files
  Found (16165) files
  Found (21382) files
  Found (26871) files
  Found (40908) files
  Found (49726) files
  Found (60192) files
  Found (72822) files
  Found (86516) files
  Found (97840) files
  Found (97973) files

  Summary of backup:
  ------------------

  Backup of system state failed [7/28/2011 4:09 PM]

  Log of files successfully backed up
  'C:\Windows\Logs\WindowsServerBackup\SystemStateBackup 28-07-2011 16-06-35.log'

  Log of files for which backup failed
  'C:\Windows\Logs\WindowsServerBackup\SystemStateBackup_Error 28-07-2011 16-06-35
  .log'

  
  There was a failure in preparing the backup image of one of the volumes in the b
  ackup set.

  Failed to format media.

  My problem is still unresolved.

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  Thursday, July 28, 2011 3:13 PM
• 

  

  0

  Sign in to vote

  Interestingly, after a reboot of my WSS2008 server yesterday, the System State Backup suceeded last night (onto the D: drive), so if you get the error "Failed to format media" I would try a reboot.

  Still no solution from Microsoft as to why I cannot select the drive that the System State Backup is saved to.  The suggested changes to PSDatasourceConfig.xml did not work.  Now I am forced to have it on my D: drive and because of that I have to keep the shadow copies of my D: drive on the same drive, which is not what we wanted.  But at least now we have a working backup of this server.

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  Friday, July 29, 2011 8:11 AM
• 

  

  0

  Sign in to vote

  Hi,

  Concerning the problem with the path to the backup drive check the following:

  Make sure the installpath is correct in the registry under: 
      HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Setup

  Make sure SYSTEM account has full control of the file:
      C:\Program Files\Microsoft Data Protection Manager\DPM\Datasources\PSDataSourceConfig.xml

  Make sure the C:\Program Files\Microsoft DPM\DPM\bin\LHSystemStateBackup.cmd file is not hardcoded to use D: - it should read as follows to accept the variable %1 which is read from the PSDataSourceConfig.xml to point to C:\WindowsImageBackup
  
     rem Start a wbadmin.exe to create a systemstate backup
     start  /WAIT %SystemRoot%\system32\wbadmin.exe start systemstatebackup -quiet -backuptarget:%1

   

  ---

  Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

  Monday, August 1, 2011 12:34 AM
• 

  

  0

  Sign in to vote

  I've just checked all of those items.  They all appear as you said they should above.

  It gives me a workaround in that C:\Program Files\Microsoft DPM\DPM\bin\LHSystemStateBackup.cmd file could be hardcoded to use C: by changing that line to
  
     start  /WAIT %SystemRoot%\system32\wbadmin.exe start systemstatebackup -quiet -backuptarget:C:

  This would work for this particular installation of DPM because in this case I do want the System State Backup to always got to C:.  I will try it tonight and post here if it works.

  The way I understand it is that we can either
  (a) Let DPM decide which drive to use (one with most free space)
  (b) hardcode the drive used for ALL protected servers
  and that we cannot specify/hardcode drives to use on a per-server basis, i.e. you cannot force the use of C: on SERVER1 and force the use of D: on SERVER2.

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  Monday, August 1, 2011 9:31 AM
• 

  

  0

  Sign in to vote

  I think I would troubleshoot why the setting in the PSDataSourceConfig.xml is not being honored.  Did you follow ALL the steps outlined in the technet article - did you get the alert and follow the link ?

   

  To change the location of the system state backup file

  1. On the protected computer, open PSDatasourceConfig.xml in an XML or text editor. PSDatasourceConfig.xml is typically located at install path\Program Files\Microsoft Data Protection Manager\DPM\Datasources.

  2. Change the <FilesToProtect> value from %systemdrive% to the desired location.

  3. Save the file.

  4. On the DPM server, if there is a protection group protecting the system state of the protected computer in step 1, run a consistency check.

  5. The consistency check will fail and generate an alert. Perform the recommended actions in the alert as follows:

     1. In the alert details, click the Modify protection group link, and then step through the wizard.
        
        
     2. Perform a consistency check.
        
        

   

  By Default, DPM will use the drive with the most free space "at the time of protection" to store the systemstate backup files.  Once configured, it will not change, however, by you modifying the PSDatasourceConfig.xml on each machine, you have direct control over where it gets moved to.

  ---

  Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

  Monday, August 1, 2011 3:07 PM
• 

  

  0

  Sign in to vote

  I had not run the consistency check before.  I have just tried to run it on both my protection groups.  I right click on my Protection Group and select "run consistency check".  However no "Consistency Check" job appears under "Monitoring -> Jobs" (under a filter I created to show all such jobs) and no alert was generated.  I got a message saying "Consistency check would be performed for all those members which are in the inconsisitent state in the selected protection group."  However under the Protection tab, no members were in the "Inconsistent" state.  Does this mean that the consistency check is not running because DPM decides it is not needed? 

  Interestingly, despite the fact that I changed C:\Program Files\Microsoft DPM\DPM\bin\LHSystemStateBackup.cmd to use C: by changing that line to
  
     start  /WAIT %SystemRoot%\system32\wbadmin.exe start systemstatebackup -quiet -backuptarget:C:

  ...it still shows in the event log that it went to D: on my WSS2008 server.

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  Wednesday, August 3, 2011 10:44 AM
• 

  

  0

  Sign in to vote

  Hi,

  1) The CC ran at the PG level will only run a CC against a datasource that is inconsistent.  A CC ran on a data source will run unconditionally.

  2)  It seems that the C:\Program Files\Microsoft DPM\DPM\bin\LHSystemStateBackup.cmd file is not the one being used, Are there any other copies of that file ?   If you rename it, does the DPM backup fail ? 

  ---

  Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

  Wednesday, August 3, 2011 5:37 PM
• 

  

  0

  Sign in to vote

  1) The CC ran at the PG level will only run a CC against a datasource that is inconsistent.  A CC ran on a data source will run unconditionally.

  How do I run a CC against a data source?  On the protection tab if I select the item "Computer\System Protection\System State" for that server, in the action pane the "perform consistency check" option is greyed out.  Same if I right-click on that item.

  As regards LHSystemStateBackup.cmd I will rename it tonight and will see if the System State backup fails.

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  Thursday, August 4, 2011 12:38 PM
• 

  

  0

  Sign in to vote

  Hi,

  The only condition that I can think of that would make the option to run a CC be grayed out is if the data souce is in inactive protection.  

  ---

  Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

  Thursday, August 4, 2011 4:51 PM
• 

  

  0

  Sign in to vote

  After renaming LHSystemStateBackup.cmd to LHSystemStateBackup_renamed.cmd my system state backup failed with the following error:

  Computer\System Protection (DPMServer)

  DPM failed to create the backup. If you are backing up only System State, verify if there is enough free space on the protected computer to store the System State backup.  On protected computers running Windows Server2008, verify that Windows Server Backup (WSB) is installed and that it is not performing any other backup or recovery task. (ID 30214 Details: Internal error code: 0x809909FB)

  You can see the errors reported by WSB in the Eventlog on the protected computer DPMServer. For resolution actions and more information on these errors, go to http://technet.microsoft.com/en-us/library/cc734488(WS.10).aspx.

   

  This is my DPM server, not the WSS2008 target I was trying to backup, so I now realise that each server on which an agent is deployed has its own PSDatasourceConfig.xml and LHSystemStateBackup.cmd and you meant I should change the files on the server with the agent where the problem was. That at least explains that.

  Still do not understand why I cannot run a CC manually on a datasource and I can find no documentaion on running a CC manually in DPM 2010 on Technet.  I only have 2 protection groups and both are in use.

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK
  

  Friday, August 5, 2011 10:46 AM
• 

  

  0

  Sign in to vote

  This is PSDatasourceConfig.xml the from the protected computer, as you can see D: was specified explicitly:

    <?xml version="1.0" encoding="utf-8" ?>

  - <PSDatasourcesConfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.microsoft.com/2003/dls/PSDatasourceConfig.xsd">

  - <DatasourceConfig>

    <WriterId>7B2E4C3D-876F-4535-B5C9-B344D4F05120</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer</LogicalPath>

    <ComponentName>DisconnectedClient</ComponentName>

    <FilesToProtect />

    <FilesToExclude />

    <FileSpecRecursive>true</FileSpecRecursive>

    <Size>1024</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

  - <DatasourceConfig>

    <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer</LogicalPath>

    <ComponentName>System Protection</ComponentName>

    <FilesToProtect>D:\WindowsImageBackup\*</FilesToProtect>

    <Size>51200</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

  - <DatasourceConfig>

    <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer\System Protection</LogicalPath>

    <ComponentName>System State</ComponentName>

    <FilesToProtect />

    <Size>10240</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

  - <DatasourceConfig>

    <WriterId>8c3d00f9-3ce9-4563-b373-19837bc2835e</WriterId>

    <Version>01.00.0000.00</Version>

    <VssWriterInvolved>false</VssWriterInvolved>

    <LogicalPath>Computer\System Protection</LogicalPath>

    <ComponentName>Bare Metal Recovery</ComponentName>

    <FilesToProtect />

    <Size>51200</Size>

    <UseDRWithCC>true</UseDRWithCC>

    </DatasourceConfig>

    </PSDatasourcesConfig>

   

  I have changed that to D: to C: now and will reply Monday if that has fixed it.

   

  My apologies for any confusion!!

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  • Proposed as answer by ShaneB. _ Tuesday, August 16, 2011 3:39 PM

  Friday, August 5, 2011 10:56 AM
• 

  

  0

  Sign in to vote

  Hello,
  
  Does this issue still exist?

  
  Thanks
  Shane

  
  

  Tuesday, August 9, 2011 10:35 PM
• 

  

  0

  Sign in to vote

  The System State backups are now going to C:, and it is succeeding.  The issue is resolved.

  Thanks,

  Richard

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  Wednesday, August 10, 2011 8:04 AM
• 

  

  0

  Sign in to vote

  Hello,
  
  Does this issue still exists?
  
  
  Thanks
  Shane

  Friday, August 12, 2011 1:21 PM
• 

  

  0

  Sign in to vote

  Hello,
  
  Does this issue still exists?
  
  
  Thanks
  Shane

  
  No, it is resolved.  Mike gave me the correct solution.  Case closed!

  Thanks

  ---

  Richard Gillis, MCITP Server 2008 Administrator in Birmingham UK

  Friday, August 12, 2011 1:30 PM
• 

  

  0

  Sign in to vote

     I have the same problem - System state consistency checks adn recoery points fail on the DPM server's System State, but is suscessful using WSB. In addition, many of my client server recovery points started failing at the same time - same error.  The protection agent on TRGAZDPM01.ryland.com was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)

   My PSDataSourceConfig.xml file does not have the wrong drive.  The storage pool isn't anywhere near full. I am getting LOTS of .CRASH files from the DPMRA.  Here are pertinent pieces of an error log:

  xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2137454160" Reason="Error" CommandID="RAGetVolumeStatistics" CommandInstanceID="67629372-2b19-476a-aa50-9e5fd3932af7" GuidWorkItem="cd0a882f-f0c2-479d-a1e8-ba1a0e2308b3" TETaskInstanceID="d4e97f4e-2326-42c3-b79b-50489d27dc51"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="2001" DetailedCode="-2137454160" DetailedSource="2"><Parameter Name="AgentTargetServer" Value="TRGAZDPM01.ryland.com"/></ErrorInfo></Status>

  106C 1708 08/25 03:10:57.859    03   runtime.cpp(1599)    [000000000049C000]     D4E97F4E-2326-42C3-B79B-50489D27DC51 FATAL ]

  106C 1708 08/25 03:10:57.859    05   defaultsubtask.cpp(680)    [0000000000551640]     D4E97F4E-2326-42C3-B79B-50489D27DC51 WARNING    Failed: Hr: = [0x809909b0] : F: lVal : CommandReceived(pAgentOvl)

  106C 1708 08/25 03:10:57.869    29   radefaultsubtask.cpp(356)  [0000000000551640]     D4E97F4E-2326-42C3-B79B-50489D27DC51 WARNING    Failed: Hr: = [0x809909b0] : CRADefaultSubTask: WorkitemID does not exist, {CD0A882F-F0C2-479D-A1E8-BA1A0E2308B3}

  106C 1708 08/25 03:10:57.869    05   defaultsubtask.cpp(459)    [0000000000551640]     D4E97F4E-2326-42C3-B79B-50489D27DC51 WARNING    Failed: Hr: = [0x809909b0] : F: lVal : CommandReceivedSpecific(pCommand, pOvl)

  106C 1708 08/25 03:10:57.869    03   runtime.cpp(1599)    [000000000049C000]     D4E97F4E-2326-42C3-B79B-50489D27DC51 FATAL Subtask failure, sending status response XML=[<?xml version="1.0"?>

  106C 1708 08/25 03:10:57.869    03   runtime.cpp(1599)    [000000000049C000]     D4E97F4E-2326-42C3-B79B-50489D27DC51 FATAL <Status

  AND:

       [000000000050D480]         NORMAL     CreateNewAcceptSocket called for address family: 2

  106C 16E0 08/25 03:15:49.225    20   cc_base.cpp(913)                NORMAL     DM: - Spotted QoS IPv6, Index:4

  106C 16E0 08/25 03:15:49.225    20   cc_base.cpp(908)                NORMAL     DM: - Spotted QoS IPv4, Index:5

  106C 16E0 08/25 03:15:49.225    20   cc_base.cpp(1036)               NORMAL     Created Socket with Family: 2, QoS Index: 5

  106C 16E0 08/25 03:15:49.225    20   cc_base.cpp(761)     [0000000000553070]        NORMAL     DM: - EndPoint:000000000050D480, Family: 2, Indx:5, Port:0

  106C 1650 08/25 03:15:59.139    03   runtime.cpp(1652)    [000000000049C000]        NORMAL     CDLSRuntime::ProcessIdleTimeout

  106C 1650 08/25 03:15:59.139    03   runtime.cpp(646)     [000000000049C000]        NORMAL     CDLSRuntime::Uninitialize, bForce: 0

  106C 1650 08/25 03:15:59.139    05   genericagent.cpp(264) [0000000000499960]        NORMAL     Agent Can Shutdown if there is only default wokitem active[1]

  106C 1650 08/25 03:15:59.139    29   dpmra.cpp(360)  [0000000000499960]        NORMAL     CDPMRA::Shutting down dpmra, force-shutdown :yes

  106C 1650 08/25 03:15:59.139    03   workitem.cpp(391)               NORMAL     Timing out WI [000000000045F150], WI GUID = {B71B4544-7067-4A30-B5FB-BA320B10D82A}, ..last DM activity happened 67340670msec back, WI Idle Timeout = 390000msec

  106C 1650 08/25 03:15:59.139    22   genericthreadpool.cpp(664) [000000000054E930]           NORMAL     CGenericThreadPool: Waiting for threads to exit

  106C 16E0 08/25 03:16:01.139    20   cc_base.cpp(1226)    [0000000000555360]        NORMAL     DM: Aborting  http state machine for reason 0

  106C 07F4 08/25 03:16:01.139    20   cc_base.cpp(1226)    [000000000050D620]        NORMAL     DM: Aborting  http state machine for reason 0

  106C 1024 08/25 03:16:01.140    20   cc_base.cpp(1226)    [000000000050F950]        NORMAL     DM: Aborting  http state machine for reason 0

  106C 1788 08/25 03:16:01.140    20   cc_base.cpp(1226)    [0000000000512CC0]        NORMAL     DM: Aborting  http state machine for reason 0

  106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [0000000000512CC0]        NORMAL     DM: Going to timeout connection

  106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [0000000000519590]        NORMAL     DM: Going to timeout connection

  106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [000000000051B880]        NORMAL     DM: Going to timeout connection

  106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [000000000051DB70]        NORMAL     DM: Going to timeout connection

  106C 07F4 08/25 03:16:01.140    20   cc_base.cpp(1226)    [000000000051FE60]        NORMAL     DM: Aborting  http state machine for reason 0

  106C 16E0 08/25 03:16:01.140    20   cc_base.cpp(1226)    [0000000000553070]        NORMAL     DM: Aborting  http state machine for reason 7471221

  106C 1024 08/25 03:16:01.140    20   cc_base.cpp(1226)    [0000000000522150]        NORMAL     DM: Aborting  http state machine for reason 0

  106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [000000000051FE60]        NORMAL     DM: Going to timeout connection

  106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [0000000000522150]        NORMAL     DM: Going to timeout connection

  106C 1650 08/25 03:16:01.140    20   cc_base.cpp(388)     [0000000000553070]        NORMAL     DM: Going to timeout connection

  106C 1788 08/25 03:16:01.140    20   cc_base.cpp(1226)    [000000000051DB70]        NORMAL     DM: Aborting  http state machine for reason 0

  106C 1024 08/25 03:16:01.140    20   cc_base.cpp(1226)    [0000000000519590]        NORMAL     DM: Aborting  http state machine for reason 0

  106C 07F4 08/25 03:16:01.140    20   cc_base.cpp(1226)    [000000000051B880]        NORMAL     DM: Aborting  http state machine for reason 0

  106C 1650 08/25 03:16:01.142    22   genericthreadpool.cpp(664) [000000000049C000]           NORMAL     CGenericThreadPool: Waiting for threads to exit

  106C 11D4 08/25 03:16:03.143    03   timer.cpp(513)  [0000000000524F68]        ACTIVITY   Shutting down timer thread.

  106C 1650 08/25 03:16:03.143    03   service.cpp(81)            ACTIVITY     CService::StopThisService

  106C 1650 08/25 03:16:03.143    03   service.cpp(281)     [000000000022FA00]        ACTIVITY   CService::StopService

  This just started last Friday - everything was smooth until then, and I am not aware of any changes made to the DPM server. 

  Thursday, August 25, 2011 3:50 AM