none
DPM 2016 MBS Performance downward spiral

    Question

  • Hi Guys!
    I´m so pissed, really i´m at a point were i would like to just uninstall and go home.
    Why? Since we upgrade to DPM 2016 and switched to MBS we have massive problems.
    The System
    We have 3 independent locations,
    On every location is one physical machine with Windows Server 2016 and DPM 2016 UR4.
    On 2 Location we have and Raid 6 Storage attachd with Fibrechannel.

    On 1 Location we use and Buffalo NAS attached via ISCSI and an 1 Gigabit connection.
    On all 3 Location we also use Tapes.

    The Problem:
    On all 3 locations the performance is gradually going down. If something took 15 minutes at the start it took 1 our and 50 minutes a few weeks later.
    The performance goes down until every single backup takes so long that new scheduled backups are in qeue. 
    So, after 2 – 3 Months my backups basicly stop working.

    The current "Workaround":

    1. Take the Storage offline.
    2. Violently delete it from DPM.
    3. Take the Storage online an reattach it to DPM.
    4. DPM now formats that with REFS.
    5. DPM Shell: dmpsync.exe –ReallocateReplica
    6. Get-DPMProtectionGroup -DPMServerName scdpm | Get-DPMDatasource | Start-DPMDatasourceConsistencyCheck
    7. Wait until everything is finishd.

    Now Backups will work fine for a few weeks, than they work for a few weeks, than they work, then they kind of work, then you start to pull your hair and then I start at 0 again.

    At the moment it´s Round number 6.


    I also had very long conversation with the storage manufacturer and i´m sure: It´s not the storage.
    The problem is REFS and MBS.

    So i´m not the first one with this problem, but please - is there anyone who found a real solution for this. Yes? Can you explain it to me as simple as possible. Not because I would not understand but im already at a point that I need easy words to be sure there is absolutely nothing I do wrong. ^^
    Oh it´s not the Windows Defender / Anti Virus problem.


    Thanks in advance. I also will provide an Screenshot were I write to tape from the storage. The first 3 entries are from before my last rebuild and the last one from today. I rebuild yesterday.
    I also did not wait as long as usually, so if you think 1:53 is not so much longer than 00:19… if I would wait longer it would be 5+ hours and than 8+ hours and than …

    https://imgur.com/ejf7bJm










    • Edited by Intirius Thursday, January 25, 2018 12:50 PM
    Thursday, January 25, 2018 12:42 PM

All replies

  • You are not alone. The problem in this case is mostly ReFS, the file system behind MBS. DPM is doing things with ReFS that fills up the servers RAM and also causes slower backups.

    Things that helped, but didn't solve the problem:

    1. Install the latest Cumulative Update for Windows Server 2016. Somewhere in there is an updated refs driver that helps a bit with the memory consumption. See: https://support.microsoft.com/en-us/help/4016173/fix-heavy-memory-usage-in-refs-on-windows-server-2016-and-windows-10

    2. There is also this KB about the server becoming unresponsive: https://support.microsoft.com/en-us/help/4035951/refs-volume-using-dpm-becomes-unresponsive-on-windows-server-2016

    You can also play around with the registry settings in the two KBs. With these articles i was able to get our server stable for the most part. Some backups are still super slow, but at least it doesn't completly freeze anymore.

    There is also a lengthy discussion in the Veeam Forums about this: https://forums.veeam.com/veeam-backup-replication-f2/refs-4k-horror-story-t40629-780.html

    According to the Veeam forums another update to the ReFS problems will be included in the February 2018 CU for Windows Server 2016. Let's hope this fixes this for good. I am tired of seeing a backup of a 6TB fileserver running for almost 22 hours with one thirds of the backups never completing.

    Thursday, January 25, 2018 2:13 PM
  • Hi,

    there is a known Bug with RefS on Server 2016, not DPM related.

    We have tested a private Fix, and the result has been very positive. From Microsoft we heard the RTM of this Fix should be released at the End of February.

    So only thing you can do, is still wait, sorry.


    Michael Seidl (MVP)

    SYSCTR Senior Consultant, Blogger, CEO

    Blog | Twitter | Facebook | LinkedIn | Xing | Youtube

    Note: Posts are provided "AS IS" without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Friday, January 26, 2018 9:06 AM
  • Another thing that helped, at least in the short term, is to put more RAM into your DPM server and reboot it regularly. Not a permanent fix, but for us some backups run a lot faster, at least for a couple of days.
    Friday, January 26, 2018 12:39 PM
  • Hey guys, Microsoft just released an optional patch that includes a new ReFS driver that reportably improves performance: https://support.microsoft.com/en-us/help/4074590/windows-10-update-kb4074590

    I'm installing it now and testing it out on my server. I'm going to do the "workaround" to start from scratch after I install it.

    Thank the guys over at https://forums.veeam.com/veeam-backup-replication-f2/refs-4k-horror-story-t40629-810.html I've been following that thread for awhile.

    Friday, February 23, 2018 3:56 PM
  • Brilliant - after playing around with those "tuneable parameters" I don't get "vhdmp" error messages anymore in System log but backup itself is as slow and buggy as before.

    Microsoft, fix this issue ASAP!

    Wednesday, February 28, 2018 12:48 PM
  • The fix is here: https://support.microsoft.com/en-us/help/4077525

    Michael Seidl (MVP)

    SYSCTR Senior Consultant, Blogger, CEO

    Blog | Twitter | Facebook | LinkedIn | Xing | Youtube

    Note: Posts are provided "AS IS" without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Wednesday, February 28, 2018 3:04 PM
  • Thanks@Michael - seems I can't find that update within SCCM/WSUS right now?
    Has it been pulled (I read about some errors caused by that update)?

    Thursday, March 01, 2018 10:17 AM
  • I have no info about that.

    Michael Seidl (MVP)

    SYSCTR Senior Consultant, Blogger, CEO

    Blog | Twitter | Facebook | LinkedIn | Xing | Youtube

    Note: Posts are provided "AS IS" without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Thursday, March 01, 2018 12:43 PM
  • Is anyone here using LTO7 tape (sas connection) and having slow issues as well?  Would 400 gb per hour with compression be a tad slow, even from a sas 12gbps raid 6 array ?  

    The specs on our library state min 2.2 TB to 5.4 TB / hour though.


    Tech, the Universe, Everything: http://tech-stew.com Just Plane Crazy http://flight-stew.com



    • Edited by techfun89 Thursday, March 01, 2018 4:37 PM
    Thursday, March 01, 2018 4:36 PM
  • It is an optional update.
    Thursday, March 01, 2018 9:03 PM
  • Totally weird stuff again - none of my 2016 servers finds this update, neither on SCCM/WSUS nor directly via Windows Update (and yes, I check for optional updates as well). By manually downloading the update from Microsoft Catalogue one can clearly see that the update is only available for Windows 10 systems.

    Nonetheless, I manually installed the update on my 2016 DPM server, at least that worked flawlessly. Now let's see how DPM and MBS behave...

    Friday, March 02, 2018 8:40 AM
  • Unfortunately, I still have problems with BMR's with MBS after this update.  After more than a year of waiting and many $$$ spent in premier support, it would be good to finally get these to work.
    Friday, March 02, 2018 4:35 PM
  • Installed the Update linked by Michael - absolutely no difference, Backups are slow and unreliable. I know we're a bit oldschool running our Disk2Disk-Storage on Synology NAS systems connected via ISCSI. But that worked flawlessly with DPM2012 - could have been faster (how ironic, compared to the situation now) but it simply worked.

    What I see now is DPM reading with around 10MB/s when doing Tape backups where it was reading with ten times the speed before on DPM2012. Write speed drops below 1MB/s - ridiculous.

    I played around a bit with the Synos, switching LUNs from block- to file-level, using RAID10 instead of RAID5 and so further and so on. No changes to be seen, absolutely no changes.

    I'm now gonna open a ticket with MS but from what I read in several other threads I might already know about their answer...

    Monday, March 05, 2018 12:23 PM
  • Good luck with that.  May be you'll get someone who actually cares about getting these issues resolved, but I doubt it.
    Monday, March 05, 2018 2:21 PM
  • Have you also made some Changes to the registry like described in the KB Article?

    Michael Seidl (MVP)

    SYSCTR Senior Consultant, Blogger, CEO

    Blog | Twitter | Facebook | LinkedIn | Xing | Youtube

    Note: Posts are provided "AS IS" without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Monday, March 05, 2018 7:38 PM
  • Only tried the timeout registry value without success.  Anyone have luck with the others?  Don't really have time to diagnose all of them, esp. if it turns out to be a waste of time.  Some guidance from MS would be helpful here.  I expect DPM to work without all of this effort as this is not my job.

    Monday, March 05, 2018 9:24 PM
  • Hi, we are having this issue on about 10 DPM servers in our environment, I've installed the february hotfix that is supposed to solve the performance issues with ReFS but I see no improvement whatsoever. I've configured one of those servers with the registry changes but no improvement either.

    Tape backups go below 1GB/min, where we should be able to achieve 10GB/min.

    However it looks like disk based backup and console responsiveness are better with the hotfix.

    Marc

    Tuesday, March 06, 2018 11:22 AM
  • Hello Guy's

    we also have Perfomance Problems since we use  DPM 2016.
    Has andybody a recommendation for the "tuneable REfs Settings" ?

    The truth about MBS is: 3x slower Backups with a (big) pinch of unrelability.

    regards
    Stefan

    Wednesday, March 07, 2018 9:13 AM
  • Hi,

    here are the recommended REG Settings from MS Support Case.


    Michael Seidl (MVP)

    SYSCTR Senior Consultant, Blogger, CEO

    Blog | Twitter | Facebook | LinkedIn | Xing | Youtube

    Note: Posts are provided "AS IS" without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Wednesday, March 07, 2018 1:12 PM
  • Michael:

    Did those recommended settings work?  I've had many recommendations for more than a year of various support cases from Microsoft and none have worked.  I'm highly skeptical about wasting more time.

    Thanks.


    • Edited by simdoc Wednesday, March 07, 2018 2:25 PM
    Wednesday, March 07, 2018 2:25 PM
  • At the moment we are not seeing any Troubles, but the related Customer to our Support Case, wasn't touched so far with the official Fix, cause we havent had time till now.

    Michael Seidl (MVP)

    SYSCTR Senior Consultant, Blogger, CEO

    Blog | Twitter | Facebook | LinkedIn | Xing | Youtube

    Note: Posts are provided "AS IS" without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Wednesday, March 07, 2018 4:28 PM
  • Hi,

    Good News Maybe

    I'm not sure if you had a fix but I experienced the same issues, worked fine for a while but then backups started to hang and finally to a point where I had to delete the storage and re-create, this put us at risk with our clients as we had agreements for 28 days retention. Logged with Microsoft, ReFS is the issue

    First suggestion, install KB4077525 - Didn't make any changes.

    Second suggestion, install reg keys for for optimal performance, these are: 

    Windows Registry Editor Version 5.00

     

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]

    "RefsEnableLargeWorkingSetTrim"=dword:00000001

    "RefsNumberOfChunksToTrim"=dword:00000032

    "RefsDisableCachedPins"=dword:00000001

    "RefsProcessedDeleteQueueEntryCountThreshold"=dword:00002048

    "RefsEnableInlineTrim"=dword:00000001

     

     

    Windows Registry Editor Version 5.00

     

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Configuration\DiskStorage]

    "DuplicateExtentBatchSizeinMB"=dword:00000100

     

     

    Windows Registry Editor Version 5.00

     

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Disk]

    "TimeOutValue"=dword:00000120

    This did improve the backup speed however again still not good enough, backups had gone from finishing at 09:00 to 07:00.

    Microsoft now stated that it was a known issue and that the Re-FS storage team are working on a fix.

    The good part now, well for me, going back to basics I went back to the start to check all config. I found that on the RAID card it was set to 'Write Through', changed this to 'Write Back'. Magic..a Backup which was taking 8-9 hours to complete now completes in 20-40 minutes.

    I made the change 2 days ago and working fine now, obviously could go back to issues but for now I'm happy.

    John


    Friday, March 09, 2018 9:16 AM
  • Hi JRH81,

    Did you only change the setting or did you recreate the volume as well? Some RAID Controllers change from Write Back to Write Through if they detect an issue with the battery.

    Our servers are already on write back, so not a solution for us.

    Marc

    Friday, March 09, 2018 9:29 AM
  • Hi Marc,

    Just changed the RAID setting, battery is fine, I think it was just missed during the config.

    Sorry to hear it hasn't helped, could be a temp solution and could be back to square one next week.

    John

    Friday, March 09, 2018 9:35 AM
  • Installed the mentioned updated and also used the regkeys but nothings helps. The back-ups just go every time slower and slower...The DPM server is only using 10% CPU and 20% memory so that is not the problem. Does anybody already tried DPM 1801 if that brings some improvements?

    Monday, March 12, 2018 11:32 AM
  • Hi,

    as this is related to RefS and not to DPM, 1801 will not change anything.


    Michael Seidl (MVP)

    SYSCTR Senior Consultant, Blogger, CEO

    Blog | Twitter | Facebook | LinkedIn | Xing | Youtube

    Note: Posts are provided "AS IS" without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Monday, March 12, 2018 1:00 PM
  • Ok thanks for the reply. Does anyone contact with Microsoft regarding this issue? Tomorrow the new patches will be released, is there a new update maybe for this? 
    Monday, March 12, 2018 1:51 PM
  • We were having an issue with 2 DPM2016 servers at 2 different sites where during a scheduled backup or online recovery point upload the jobs would hang, and when you tried to login to the server all you got was a black screen - resetting the machine was the only way to get it back.  Installed the February 2018 updates and it helped a little bit, but then eventually the same thing would happen after a few days.  Played with the registry settings a bit - not really alot of science here just increasing values.

    DPM server has 4 cores and 64gb of RAM - never shows any signs of being stressed at all, but it seems like ReFS must do something with the memory that doesn't show in normal tools.

    Right now I have these registry settings, and I've had successful runs (knock on wood) for the past week straight, which is the longest it's gone in a while without intervention:

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
    "RefsDisableLastAccessUpdate"=dword:00000001
    "RefsEnableInlineTrim"=dword:00000001
    "RefsDisableCachedPins"=dword:00000001
    "RefsProcessedDeleteQueueEntryCountThreshold"=dword:00010000
    "RefsNumberOfChunksToTrim"=dword:00000020
    "RefsEnableLargeWorkingSetTrim"=dword:00000001

    Hope this helps...

    Monday, March 12, 2018 2:09 PM
  • Hi,

    ReFS does not like many "cheap" RAID Controllers (including those in SAN Storages) and the way they are behaving.

    I would be really interested in the target storage you guys are using.

    With high end storage we do not experience those massive impacts.

    Furthermore DPM 2016 is even more picky on shared storage than 2012R2; so it does not like when you share the backend disks (not LUNs!) with other DPM-Server or serves.
    But that depends on the storage, of course.

    There is a difference between a 500k Netapp FAS and a Qnap or Synology

    regards

    /bkpfast


    My postings are provided "AS IS" with no warranties and confer no rights

    Tuesday, March 13, 2018 10:55 AM
  • Tried all the regkeys, but no luck.

    We are not using any RAID controllers. We have made a storage pool from the physical disks in Windows itself. It is a physical server so no shared storage. Disk cabinets are connected with SAS HBA's. 

    So still very poor performance. 



    • Edited by Lucas-076 Wednesday, March 14, 2018 12:37 PM
    Wednesday, March 14, 2018 12:20 PM
  • That's us too. No RAID but JBOD configured as Windows storage pools.
    Wednesday, March 14, 2018 2:44 PM
  • I have to revise my last entry, at least a little bit:
    Changing the LUNs on my Synologys from Block-Level to File-Level seems to do a bit of the trick, read/write speed has indeed increased. Following several advises from forums on the net, only File-Level LUNs do support caching whereas Block-Level LUNs don't.

    Did not make any difference on DPM2012 with NTFS-formatted storage but for DPM2016 using ReFS it obviously does. At least from what I've seen during the last two weeks while changing the LUNs (not completed yet), this seems to solve the issues with Disk2Disk-Backups.

    Nonetheless, read speed while doing Tape Backup is still slow as hell. Maybe this also get's solved once all LUNs are File-Based LUNs....

    Thursday, March 15, 2018 7:43 AM
  • Hi,

    First of all congratulations bkfast on your expensive storage.

    I don't read in the hardware requirements of DPM that we need an all-flash array to get decent performance. And it also does not explain the fact that if we destroy the filesystem and recreate all replica's the same system is up to 20-25 times faster and DOES perform at decent speed, that is 10GB/min tape backups to LTO6.

    Only after several weeks/months (or a certain number of iterations of the tape backup) we see a severe regression in tape backup speed. This is what the initial poster of this thread is reporting btw.

    For the record, we have a mix of Dell T620 and T630 servers running DPM. They all have a PERC RAID controller with 1GB cache.

    Marc

    Thursday, March 15, 2018 8:34 AM
  • hi all,

    pls exist solution for slow backup in the 2016 dmp ur4 ... i have instaled 2018-02 but not afect.

    OS: 2016

    DPM 2016 UR4

    backup storage is SAN dell sc2000.....

    is solution create ntfs and backup to this ?


    Falcon

    Sunday, March 18, 2018 1:25 PM
  • Hi all.

    I have the same problem with DPM 2016 on Refs system. Installed KB4077525 not help and registry settings too. DPM 2012 R2 hasn't performance problem in this server and storage. DPM storage is IBM Storwize v3700 and attached by SAS.

    I think start search other backup solution for our company.

    Tuesday, March 20, 2018 8:46 AM
  • HI Guy's

    i think we are seeing some improvements after applying kb4077525.
    The Registry Settings where already applied.
    Still the Backups are not 100% reliable.

    We are Using Direct Attatached 3,5' 7200rpm Disk  withing Raid 6 (HP Raid Controller).

    regards
    Stefan

    Tuesday, March 20, 2018 9:58 AM
  • what you use ? the storage is connect to storage pool and this add to dpm ? Or direct add volume with disk manager and connect to dpm ?

    Falcon

    Tuesday, March 20, 2018 1:33 PM
  • Hi,

    we have 3x disk enclouser (each enclouser with 12x SATA/Midline SAS, arragned as one Raid 6 Array) resulting into 3 Luns on the Raid Controller.

    I've added these 3 lun'S to one storage Pool and created on Big Vdisk.

    The BIG vdisk was added to dpm.

    regards

    Tuesday, March 20, 2018 1:50 PM
  • HI Guy's,

    today seems like a  little miracle, all backup finished with no erros and in a acceptable to fast time.
    10TB File Server Recovery Point in arround 30 Minutes.

    Could it be that the update does some secret optimization in the Background?

    regards

    Stefan

    Wednesday, March 21, 2018 6:58 AM
  • Applied the patch of last week (KB4088787) still no performance improvement. After 17hours only 28GB copied. 
    Thursday, March 22, 2018 7:02 AM
  • Applied the patch of last week (KB4088787) still no performance improvement. After 17hours only 28GB copied. 

    You are talking about Tape Peformance right?

    Thursday, March 22, 2018 7:10 AM
  • I'm sorry, no just a simple disk to disk recovery point creation
    Thursday, March 22, 2018 7:23 AM
  • We've been having exactly the same problem.

    Initially we see great performance.  The disks sit at around transfer 1.2GB/s speed during replica creation, and then slowly as the weeks go by performance gets worse and worse, until tape jobs that normally finish before we get into work Monday are finishing Thursday morning!

    I've been following the Veam thread and have all the patches and tried all the registry keys.  The memory issue has gone away, but ReFS performance still degrades over time.

    From what I can see it appears to be a fragmentation issue.  Initially most of the reading and writing is sequential, but as time goes on it gets more and more random which is why performance drops.  Blowing the disk away and recreating the replicas lays everything out nicely for sequential access again.  Allocate on write side affect or something..  


    • Edited by DJL Thursday, March 22, 2018 9:26 AM
    Thursday, March 22, 2018 9:24 AM
  • For us the latest Windows Server patches fixed most of the performance issues we had. DPM still uses a lot more memory than the system requirements suggest, but at least the backups are stable now. We had a fileserver protection point than run for ~36 hours, it's down to around 3 hours now.
    Thursday, March 22, 2018 2:32 PM
  • well, today i installed the March-Updates (KB4088787, KB4089510) and did a Reboot.

    Now the Problem is Back, 500 GB Recovery Point -> now over 3 hours (it was down to 30 Minutes).

    Now the question is: Did the update change something, or is refs getting faster with longer uptime?

    Will Monitor this again after the weekend.

    Regards

    Stefan

    Friday, March 30, 2018 1:12 PM
  • i mean that after restart is situation better, but after few days.....

    Falcon

    Monday, April 02, 2018 8:07 PM
  • After the March updates are also seeing slower backups after a couple of days. Not nearly as bad as before, but still noticable. My guess is that ReFS is still using too much memory on our really large backups (6 TB fileserver) and that eventually slows down things a bit. But for us it's managable, i am just happy that our server doesn't completly lock up anymore.
    Friday, April 06, 2018 1:13 PM
  • Months have passed, situation still unchanged. Installed May updates, DPM UR5, ... - nothing has changed, disk2disk backups take forever, tape backups will never finish, whole system gets blocked..

    Microsoft, get this issues sorted out IMMEDIATELY!

    Monday, May 14, 2018 6:37 AM
  • Hello Joerg,

    we also had a very big performance Problems with DPM and Mondern Backup Storage.

    But Since the refs driver was udpated (March Update, if i remember correct) the Perfomance now is pretty good.

    I don't know if this observation is correct but to me it seems that, after a reboot the performance is slower and after 1 oder 2 Days the performance is very satisfying for us.

    We are using DAS Storage (Local RAID6 Arrays with HP Raid Controller). So maybe thers something special to your iscsi setup. 

    Do you have the possibility to do a test Backup to a local disk? If the local Backup performs better, you know that you need to dig into your iscsi setup. 

    regards

    Stefan

     

    Monday, May 14, 2018 7:30 AM
  • Thanks Stefan - we see an opposite behaviour, like after a reboot system runs pretty well and gets slower and slower over days until it gets stuck completely. Had > 500 recovery points in queue this morning, where like >400 were in pending state and around 100 were "In Progress" but not doing anything (network usage on my DPM server <1Mbps).

    Nonetheless I will (again) try your suggestion and use a local disk as target....
    Can't be a "general ISCSI issue" as jobs work when being newly created and start getting slow after like 2 or three weeks. From my point of view there still is a big, big bug within the implementation of ReFS.

    Maybe you can gimme some details on your setup?
    DPM-Database on same machine? How much RAM is it allowed to use?
    How much RAM is installed on your system in total?
    Stuff like that... Thx. ;-)



    • Edited by Joerg Ott Monday, May 14, 2018 11:14 AM
    Monday, May 14, 2018 8:08 AM
  • Horrible performance for us too--much, much worse than 2012 R2.  We're not using iSCSI.  We are using JBOD configurations with SATA3 in Windows Storage Pools for MBS.  We have 12 physical cores (not using VM any more because performance was even worse) and 80 GB RAM in each box, much more hardware than 2012 R2 for much less performance. 

    While all types of backups are slower and more unreliable, it is almost impossible to get BMRs to work.

    Monday, May 14, 2018 1:31 PM
  • we have same trouble , DPM 1801 connected to SAN storage... in the system eventlog is 
    The description for Event ID 129 from source vhdmp cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
    If the event originated on another computer, the display information had to be saved with the event.
    The following information was included with the event:
    \Device\RaidPort4.....

    Falcon

    Monday, May 14, 2018 3:31 PM