none
How to make Replica Creation transfer more agressively. RRS feed

  • Question

  • I need to get DPM 2012 to be more aggressive in transferring data.

    I have a file server with 35.5Tb I am trying to protect with DPM 2012.

    I have a 10GB/s Network dedicated to backup traffic between the servers so Network is no issue. When I robocopy between the file server and the DPM server I generally get a consistent 1-2GB/s transfer with the odd peak up to 4GB/s on really big files. So getting the data off the disk is my bottleneck.

    However when DPM is doing a replica creation I get distinct peaks to 1Gb/s with periods of nothing in between as can be seen in the image from task manager on DPM below

    I have read about using the WaitInMSPerRequestForClientRead regkey to reduce impact on the client but I want the opposite effect so I have set it to 0.

    I have also set the AV on the fileserver to only scan on writes not reads.

    I tried Jumbo frames on both NICs but found I lost connection often so went back to default there.

    Both machines have ample RAM so there is no paging going on.

    My initial replica creation was interrupted by patching and the subsequent consistency check that I started has so far been going 214hours and 30 minutes, checked approx 33Tb of which it transferred approx 10Tb of data. This averages to 0.14 of a Terrabyte per hour processed giving about 10.5 days to complete.

    What can I do to get the DPMRA to transfer data at a rate similar to robocopy?

    Tuesday, October 29, 2013 4:08 AM

Answers

  • Unfortunately I can't answer your question but I would suggest not to use such a large datasource with DPM. As you can see the CC jobs run for a very long time and if for some reason (and it happens more often than you would think) the datasource turns inconsistent then you have to go though the whole process of running a CC job again. This means that during the time the CC job is running, which seems to run an awfully long time on your system, there are no backups being done! Can you really risk going for more than a week without regular backups? I try to keep the filesever datasources below 2 TB each so in case one needs a CC check it finishes rather quickly. The DPM guys said that they will address the issue of long running CC jobs in the future, but I don't know if something was done in DPM 2012 R2 yet...
    • Edited by Marcus Krämer Tuesday, October 29, 2013 10:59 AM
    • Marked as answer by EdgeUQ Sunday, November 3, 2013 11:20 PM
    Tuesday, October 29, 2013 10:58 AM

All replies

  • Unfortunately I can't answer your question but I would suggest not to use such a large datasource with DPM. As you can see the CC jobs run for a very long time and if for some reason (and it happens more often than you would think) the datasource turns inconsistent then you have to go though the whole process of running a CC job again. This means that during the time the CC job is running, which seems to run an awfully long time on your system, there are no backups being done! Can you really risk going for more than a week without regular backups? I try to keep the filesever datasources below 2 TB each so in case one needs a CC check it finishes rather quickly. The DPM guys said that they will address the issue of long running CC jobs in the future, but I don't know if something was done in DPM 2012 R2 yet...
    • Edited by Marcus Krämer Tuesday, October 29, 2013 10:59 AM
    • Marked as answer by EdgeUQ Sunday, November 3, 2013 11:20 PM
    Tuesday, October 29, 2013 10:58 AM
  • What can I do to get the DPMRA to transfer data at a rate similar to robocopy?

    Why not just use Robocopy? That's how we do our initial replica creations all the time (but from USB as our DPM servers are across the WAN).  I don't know why DPM would be slow doing replica creation itself, but I bet normal DPM operation will be fine if you just do the initial creation using robocopy.

    I disagree somewhat with the other reply about volume sizes.  We protect volumes that are 10TB or more and don't have any issues.  YES, if a consistency check needs to run it may take some time, but ours usually complete in less than a day and that is over a WAN link.  And consist checks don't usually happen.

    Good luck


    • Edited by Rod Savard Friday, November 1, 2013 4:35 PM
    Friday, November 1, 2013 4:34 PM
  • Thanks for your perspective. Unfortunately 2Tb is insufficient for many of the projects that need to be stored on our filesystem. I can not reasonably ask users to split their data over multiple volumes for the one project.

    I know I am an unusual case with such large datasets. It seems that DPM is the wrong tool. I have been using a scheduled robocopy in parallel while I try to get DPM to work. I might just have to ditch DPM and use some powershell scripting to check on the Robocopy results....

    Sunday, November 3, 2013 11:17 PM
  • Thanks Rod.

    My CC has been running 117 hours and 30min so far.

    Sunday, November 3, 2013 11:19 PM
  • Thanks Rod.

    My CC has been running 117 hours and 30min so far.


    Is this after doing an initial replica creation using robocopy?
    Tuesday, November 5, 2013 9:43 PM
  • No I had the initial replica eventually finish over the network. That took so long that by the time the CC happened there was 4Tb of changes to replicate.
    Tuesday, November 5, 2013 11:02 PM