none
Tape Backup Failure - ID 3311 RRS feed

  • Question

  • We are using DPM 2010 on Sever 2008 R2. We use a Powervault LT4000 with two LTO-5 drives. All tape library functionality seems to work and we are able to backup other protection groups. However, one of our main file servers protection group, approx 1.5TB of data, the tape backup fails about 300GB into the job with the 'The back up to tape failed for the following reasons: (ID 3311). Error Details: An unexpected error occurred while the job was running. (ID 104 Details: The system file %1 has become corrupt and has been replaced (0x8007027A)). This is a brand new server we just migrated to and this was the first tape backup job attempted on this protection group. The inital scheduled job failed and all 'Run backup to tape job again' attempts have failed.

    Note: We are using Tape Colocation
    • Edited by aktechsup Tuesday, December 6, 2011 5:46 PM
    Tuesday, December 6, 2011 5:43 PM

All replies

  • Hi,

    From research, that error code (0x8007027A) should only occur for files that are on a volume that has Single Instance Store (SIS) enabled. Please run chkdsk /f driveletter:    Where driveletter: is the drive letter of the protected volume on the protected server giving the error when trying to back it up.


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, December 7, 2011 6:01 AM
    Moderator
  • Thank you.  I forgot to put in a response.  We did find out there were multiple errors going on which did have to do with SIS.  Performed the chkdsk /f on the file server with SIS enabled.  This fixed some issues although I had to open a case to fix another issue where the system was becoming unresponsive when synchronizing/consitancy check of the same file server.  Turns out it was another issue with SIS.  Fortunately, soon afterwards a hotfix was available and now both my tape backups and system issues have been resolved.
    Tuesday, March 27, 2012 4:48 PM
  • Hi,

    Thanks for the update.

    Also the SIS issue is covered in this anouncement that was posted a few weeks ago.

    http://social.technet.microsoft.com/Forums/en-US/dataprotectionmanager/thread/ceddece4-d140-4f52-bddf-ce7e6825adb0


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, March 27, 2012 5:29 PM
    Moderator
  • Well, I would like to mark an Answer but the problem has returned.  I have run chkdsk /f on the volume and it didn't report anything wrong.  I can almost guarantee that this still has something to do with the sis data on the file server that is being replicated to the dpm replica.  Then, the tape backup fails with the same:

    An unexpected error occurred while the job was running. (ID 104 Details: The system file %1 has become corrupt and has been replaced (0x8007027A)).

    I have multiple volumes on our Server 2003 Storage Server all with SIS enabled.  All volumes are being protected with DPM and each volume has their own protection group setup with backup/recovery points set to different days to avoid any conflicts/overload.  However, the original volume I reported having the issue above is the same volume that is once again having an issue backing up to tape only.  I have the protection group setup to keep up to 30 days on disk and then once a month it goes to tape and that is what is failing.  All regular recovery points and full Sync's work fine.  Just the tape backup and just on this protection group.  Since other recovery groups on the same server work fine and they are actually larger volumes; this pretty much eliminates a throttling issue, tape issue, etc...  It must have something to do with the data on this one volumen but if the chkdsk /f doesn't do anything to fix it then I am at a loss on how to identify the problem.  I am going to attempt yet another chkdsk /f this weekend and then a full consitancy check and attempt the tape backup.  If that doesn't work is my only option to open a case?

    The SIS issue you mentioned above was a fix for the server becoming unresponsive and I did have to open a case and work with getting logs and memory dumps to Microsoft before that hotfix became availalbe.  So, at least the DPM server is functioning well other than this one issue.

    One more thing, I do have co-location turned and that was working well for the last 3-4 months.  Could this be part of the issue? Is there a way to disable co-location for just a specific protection group or only enable co-location for specific protection groups?
    • Edited by aktechsup Thursday, June 7, 2012 4:21 PM Add additional note/question at end
    Thursday, June 7, 2012 4:16 PM
  • Hi,

    LT Tape backups are done from the replica volume, so run chkdsk /F against the replica volume on the DPM server.  You can get the path to the replica from the details of the data source, then use mountvol.exe to get the volume guid associated with that path, then run:

    chkdsk /f \\?\Volume{GUID}\

    Once chkdsk is complete, make a new recovery point then see if long term backup jobs succeed.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, June 7, 2012 7:04 PM
    Moderator
  • Thanks for the reply.  I already attempted that but may not have created a new recovery point before attempting the tape backup. However, the chkdsk didn't show any errors.
    Thursday, June 7, 2012 7:16 PM
  • I did the chkdsk on the replica which shows no errors show below.  I then created a new restore point which worked and then started a new recovery point to tape - long term.  It did backup more than before but still resulted in the same error.  Any other thoughts?

    C:\Windows\system32>chkdsk /f \\?\Volume{d1f73612-1b75-11e1-a6ee-001018ba9546}
    The type of the file system is NTFS.

    Chkdsk cannot run because the volume is in use by another
    process.  Chkdsk may run if this volume is dismounted first.
    ALL OPENED HANDLES TO THIS VOLUME WOULD THEN BE INVALID.
    Would you like to force a dismount on this volume? (Y/N) Y
    Volume dismounted.  All opened handles to this volume are now invalid.
    Volume label is DPM-vol_61fe24b7-9b2d-4963-.

    CHKDSK is verifying files (stage 1 of 3)...
      1789184 file records processed.
    File verification completed.
      12571 large file records processed.
      0 bad file records processed.
      0 EA records processed.
      366963 reparse records processed.
    CHKDSK is verifying indexes (stage 2 of 3)...
      2038912 index entries processed.
    Index verification completed.
      0 unindexed files scanned.
      0 unindexed files recovered.
    CHKDSK is verifying security descriptors (stage 3 of 3)...
      1789184 file SDs/SIDs processed.
    Security descriptor verification completed.
      124864 data files processed.
    Windows has checked the file system and found no problems.

    1570283519 KB total disk space.
    1142114984 KB in 1645498 files.
        806356 KB in 124866 indexes.
             0 KB in bad sectors.
       1905559 KB in use by the system.
         65536 KB occupied by the log file.
     425456620 KB available on disk.

          4096 bytes in each allocation unit.
     392570879 total allocation units on disk.
     106364155 allocation units available on disk.

    Thursday, June 7, 2012 9:43 PM
  • Hi,

    There should be some details in the dpmracurr.errlog on the DPM server after a tape backup failure.  It may contain the problematic file name and if so, delete it from the replica, then do a consistency check so DPM brings it back over from the PS, then try a new RP followed by a tape backup.

    OOF for 2 weeks, good luck.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, June 8, 2012 1:35 AM
    Moderator