none
Improve backup time with millions of little files RRS feed

  • Question

  • Hello guys,

    I would need some help to improve our current backup time which is really long:

    Context:

    - DPM 2012 R2 (4.2.1273.0) correctly installed and configured to backup disks on tapes as long-term backups.

    - 5 protection groups corresponding to 5 local disks (physically mounted with iSCSI from our SAN)

    - We want each disk to be backuped on a tape every month.

    - These disks contain a lot of little files (~1kb) with an average size of 5TB for each disk

    Problem is :

    - Backups are working but take a long time and are often in errors because of this (Server restarting, misc. errors, etc...) For example,  it took 216 hours to backup only 2,7GB. Then I have some backups that need more than 13 days to be completed.

    - When I look into my Tape Library (Dell PowerVault TL2000) I can see my drive is busy writing only half of the time and idle the other half. Then I think the drive is not the root cause of my long times.

    Configuration :

    I tried to configure DPM the best I could with all these settings :

    - Check backup for data integrity not asked

    - ONly Compress data chosen (no encryption)

    - Enable on-the-wire compression (Tested with and without, no particular changes on the time taken to backup)

    - [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Agent]
    "UseShortErase"=dword:00000000
    "RSMCompatMode"=dword:0000000d : 13 (Advised in multiple posts for TL2000 tape library)
    "BufferQueueSize"=dword:00000014 : To try to improve backup time

    - [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Agent\ClientProtection]
    "WaitInMSPerRequestForClientRead"=dword:00000030 : To try to improve backup time

    Questions :

    -  Has one of you any suggestion to improve the speed of my backups?

    - Is there any way to backup these disks as a block and not to read all the files? (I don't need to restore a single file but a full disk in case of emergency). This would definitely improve the time needed to backup these disks since DPM looks every single file to store some informations in its database and I don't need this.

    Thanks !

    Tim


    Monday, August 10, 2015 12:04 PM

All replies

  • Probably the only thing you can do is to improve the performance of your DPM disk subsystem. There is no way to prestage DPM backup data to a container file and then write this more efficiently to tape. It would be great if DPM had that, but I doubt it is even under consideration. Tape drives are under a lot of stress if you cannot sustain a minimal data rate. They go into start/stop and rewind a lot which causes a very high wear on the tape and the drive itself. With each new iteration of tape technology this issue is only getting worse in my opinion.
    Tuesday, August 11, 2015 10:05 AM
  • Hi,

    Can you convert your HDD to VHDX and do backup on VHDX file ?

    Tuesday, August 11, 2015 1:47 PM
  • I'll look in this direction. If I understand in that case the VHDX will be copied to the tape as block-data and it should improve the speed right?
    Tuesday, August 11, 2015 3:16 PM
  • I don't know if that improve speed, but I think you can try.


    • Edited by Gerardas Tuesday, August 11, 2015 6:56 PM
    Tuesday, August 11, 2015 6:56 PM
  • Hi Tim,

    I'm just curious if you ever found a solution to help improve the backup times of your large file server?  I'm experiencing a similar scenario.  I dont think I have it as bad (about 70 hours for 7TB) but am looking for ideas to improve it.

    I was thinking an active - active file server cluster, but need a lab to test and see how DPM handles that... and a lab is something I dont have.

    Wednesday, March 7, 2018 8:39 PM