none
DPM 2012 R2 - Failed to clean up data RRS feed

  • Question

  • Hi,

    I'm hoping someone might be able to help me.

    I've recently been getting an error on one of our backup jobs, I've tried searching around for solutions but no-one seems to have had the exact same issue that I'm experiencing.

    The backup job is a shadow copy file backup of a complete volume.

    It's been working for the past few weeks but I'm now getting the error below when I try to run a consistency check. 

    People have reported issues similar which suggest executing the pruneshadowcopiesdpm2010.ps1 which I've tried but this seems to make no difference.

    The protection group is short term disk protection with a retention period of 14 days with file recovery points created at 11am, 1.30pm, 3.30pm and 8pm.  The job is currently reporting 32 recovery points which leads me to believe it shouldn't need to clean up any yet anyway.

    Any suggestions would be very much appreciated.

    -----------------------------

    Type: Consistency check
    Status: Failed
    Description: DPM failed to clean up data of old incremental backups on the replica for Volume D:\ on nas01.nwhs.local. Synchronization will fail until the replica cleanup succeeds. (ID 30134 Details: Cannot create a file when that file already exists (0x800700B7))
     More information
    End time: 29/04/2014 18:48:03
    Start time: 29/04/2014 18:48:02
    Time elapsed: 00:00:01
    Data transferred: 0 MB
    Cluster node -
    Source details: D:\
    Protection group: User Data
    Items scanned: 0
    Items fixed: 0

    Tuesday, April 29, 2014 6:05 PM

All replies

  • Having the same issue, ran the powerr shell script same issue.
    Thursday, May 1, 2014 6:16 AM
  • I've done a little more research and I've found that initially a synchronization of data failed with the following error.

    "Number of files skipped for synchronization due to errors has exceeded the maximum allowed limit of 100 files on this data source (ID 32538 Details: Internal error code: 0x809909FE)"

    This made the backup become inconsistent so it automatically ran a consistency check which it says completed with a warning, the warning being...

    "An unexpected error occurred while the job was running. (ID 104 Details: Cannot create a file when that file already exists (0x800700B7))"

    This is a very similar error to the error I now get when trying to run further consistency checks.  This makes me think it's this initial problem that's still outstanding.

    Does anyone know where I might be able to identify the file causing the issue? I've checked the logs that I can find and I can't see where it specifically mentions a problem file.

    Many thanks.

    Thursday, May 1, 2014 9:56 AM
  • RIS,

    Can you please review this thread to see if there may an issue with files on the protected server.  You may also want to run a chkdsk against the datasource volume. 

    http://social.technet.microsoft.com/Forums/en-US/f7de60cd-841f-4d22-862e-aaf3723717c7/dpm-could-not-log-files-that-were-skipped-during-backup-to-volumexxxxxfailedfilesdat-id?forum=dataprotectionmanager

    Thanks

    Andy


    Regards, A.Nadar, This posting is provided "AS IS" with no warranties, and confers no rights.

    Saturday, May 3, 2014 3:55 AM
  • RIS,

    Can you please review this thread to see if there may an issue with files on the protected server.  You may also want to run a chkdsk against the datasource volume. 

    Thanks

    Andy


    Regards, A.Nadar, This posting is provided "AS IS" with no warranties, and confers no rights.

    Hi Andy,

    Thanks for your suggestion.

    I've read through that thread and followed the advice to no effect.

    I've also run a chkdsk which completed successfully but again hasn't had any effect.

    Currently it's reporting that the replica is not consistent but when clicking to run a consistency check it fails immediately with the following error.

    DPM failed to clean up data of old incremental backups on the replica for Volume D:\ on nas01.nwhs.local. Synchronization will fail until the replica cleanup succeeds. (ID 30134 Details: Cannot create a file when that file already exists (0x800700B7))

    Any further suggestions would be appreciated.

    Thanks,

    Chris

    Wednesday, May 14, 2014 11:11 AM
  • RIS,

    What version of DPM are you using?

    What datasource are you protecting?  Is this system state? IF it is then the next step will be to check wbadmin on the local server to see if it succeeds. 

    Past that we need to verify DPM has antivirus exclusions.  Are you running Sophos by chance?

    Here is a link for reference also

    http://social.technet.microsoft.com/Forums/en-US/1eeb0ebe-9fc4-4550-9b6e-53e1d528b23c/constant-replica-is-inconsistent-errors-id-112-cannot-create-a-file-when-that-file-already?forum=dataprotectionmanager

    Thanks


    Regards, A.Nadar, This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, May 14, 2014 7:33 PM
  • Hi Andy,

    We're running DPM 2012 R2 and it's protecting a volume from our Storage Server 2012 server.  This is currently 1.38 TB worth of data and the DPM server has allocated 2.2 TB of space and still has over 3 TB of unallocated space.

    All of the other backups are completing as expected including other volumes from this same server.

    We do run Sophos Antivirus but it does have exceptions set and the volumes that are being protected do not store any Sophos related data.

    I've found that if I completely remove the protection group including the backed up data and then recreate it, it will work again as expected but only for about a week before coming back up with this same error.

    Thanks for your help so far - its really frustrating because when it works, it works great especially with the end-user-recovery support but I'm at a loss as to the root cause of this problem.

    Chris

    Thursday, May 15, 2014 10:25 AM
  • Hi,

    any news on this? We experience the same error ("DPM failed to clean up data of old incremental backups on the replica for Volume D:\ .... Synchronization will fail until the replica cleanup succeeds. (ID 30134 Details: Access denied (0x80070005)").

    I do not understand how it is possible to get "access denied". Must be a bug as the backup worked perfectly for a week. We already recreated the protection group but to no avail. This is very annoying as this concerns our file server where we need to regularly restore files. 

    Best regards,

    Ben

    Monday, July 28, 2014 10:42 AM
  • RIS,

    The access denied can come if there is already a lock on the file( like Antivirus). 

    Can you let me know what version of DPM2012R2?   if you goto about DPM you it will give you exact version 4.x.xxxx.x.  You stated you had AV exclusions set

    "We do run Sophos Antivirus but it does have exceptions set and the volumes that are being protected do not store any Sophos related data."

    I want to ensure all are set from - http://technet.microsoft.com/en-us/library/hh757911.aspx - if you can test disabling AV that may help.

    Can you check the event logs on DPM server and PS during the failure to see if there is anything else running that may lock the files?

    There may also be a "failedfiles.dat" that may due to unsupported files see link below for assistance troubleshooting that - http://social.technet.microsoft.com/Forums/en-US/f7de60cd-841f-4d22-862e-aaf3723717c7/dpm-could-not-log-files-that-were-skipped-during-backup-to-volumexxxxxfailedfilesdat-id?forum=dataprotectionmanager

    How to read failedfiles.dat if there is one -

    http://social.technet.microsoft.com/Forums/en-US/dataprotectionmanager/thread/27c48816-dabb-41a0-b800-6e1ac2fc1ed5

    Also what is the OS version of the DPM server?  Are you doing any dedupe\ODX\Thin provisioning on the protected server? 

    Ultimately we may need a ticket to determine what is locking out the file during the failure but please let me know the answers to the above and I will see if that helps.  Thanks



    Regards, A.Nadar, This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, July 28, 2014 3:30 PM
  • Hello Andy,

    we have configured AV (SCEP 2012 R2) as stated in the article that does not make any difference. We noticed that it helps to reboot the server. After the reboot backup works normally but only for one day.

    We use DPM 2012 R2 with UR2 (4.2.1235.0) on a Windows Server 2012 R2. Before UR2 we did not have any problems.

    On the file server we run file deduplication (Windows Server 2012 R2) and AV (SCEP 2012 R2).

    Thanks in advance for your help!

    Best regards

    Ben

    Thursday, July 31, 2014 12:55 PM
  • Hello @all,

    I think I found the solution for my case - the problem was caused by AV scan. During the scans, DPM is unable o access the replica volumes being scanned so the backup job fails with "Access denied". I reconfigured AV to not do a full scan of the DPM volumes.

    Just to clarify - is it enough to add \\?\Volume* to the path exclusions? Or do I have to do it otherwise?

    Best regards,

    Ben

    Monday, August 4, 2014 9:05 AM
  • Ben,

    From the Antivirus exceptions excluding the DPMRA.exe process should exempt the replicas from being scanned.  It is required to exempt the process explicitly and may require a restart of the DPMRA service to take effect.

    http://technet.microsoft.com/en-us/library/hh757911.aspx

    From the article:

    Replica and transfer logs—To do this, disable real-time monitoring of dpmra.exe, which is located in the folder Program Files\Microsoft Data Protection Manager\DPM\bin. Real-time monitoring degrades performance because the antivirus software scans the replicas each time DPM synchronizes with the protected server, and scans all affected files each time DPM applies changes to the replicas.

    Excluding the \\?\Volume* may end up excluding more volume than just the ones DPM creates but if DPM is the only application it may suffice. 

    You may also try excluding the Program Files\Microsoft Data Protection Manager\DPM\Volumes or the Program Files\Microsoft Data Protection Manager\DPM\Volumes\replica directory as a test

    Ultimately if the exclusions listed in the article do not resolve the issue you may want to open a support case.  Thank you 

    Andy


    Regards, A.Nadar, This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, August 6, 2014 3:42 PM
  • Also please try excluding the DPMRA.exe on the protected server also as a test.  Thank you

    Regards, A.Nadar, This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, August 6, 2014 3:43 PM
  • Hey Chris,

    Any luck?

    I am also experiencing the same issue....

    If you have found the solution then please share with us....

    Thursday, August 6, 2020 1:07 AM