none
Online Recovery Point taking long time

    Question

  • I am piloting DPM 2016 on Windows 2016 with Azure Backup.

    I have a Windows 2012 R2 file server with around 150gb of data.

    The when the disk based recovery point runs, a few hundred MB are transferred and it takes as few minutes.

    However, when the Online recovery point runs it takes several hours to complete.

    If i look at Resource Manager, I can see MsMpEng.exe and cbengine.exe reading reading every file in the replica.

    It seems almost like a what I assume a consistency check would look like.

    Tuesday, March 21, 2017 4:51 PM

All replies

  • Hi,

    Yes, that is the way that online backups work but incremental backups should only read files that have changed since the last backup based on entries in the NTFS USN Journal.  However, if a backup fails for some reason it may revert to checking all files again, but that should be rare.  You can try adding an exclusion in windows defender to not scan cbengine.exe process and see if that helps.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, March 21, 2017 8:06 PM
    Moderator
  • Thanks. I did add a windows defender exclusion for cbengine.exe and that helped somewhat.

    However, I tried the online backup with another file server and am experiencing the same type of file access by cbengine.exe. Is there a way to very that the back up is using the journal?


    Wednesday, March 22, 2017 3:07 PM
  • Hi,

    Yes, you can get a sense of how your backups are being performed using the below steps.

    1) Open an administrative command prompt and CD to the azure agent installations temp folder.

      CD C:\Program Files\Microsoft Azure Recovery Services Agent\Temp

    2) To search for all optimized backups using the USN run this: find /i "Using usn iterator for Ds" cbengine*.errlog

    3) To search for jobs that were not optimized to use the USN and take much longer run this: find /i "not using optimized DR" cbengine*.errlog

    Hopefully you will see the vast majority of the backup jobs used the usn iterator.

       


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, March 22, 2017 3:46 PM
    Moderator
  • There are 70 lines using USN and 126 not using.

    Any way to dig deeper?

    Thursday, March 23, 2017 5:48 PM
  • Hi,

    The most common cause if due to the previous backup failing.  I know there is some ongoing work to improve optimized backup resiliency even after a previous backup failed.  No eta on that change. 


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, March 24, 2017 4:53 PM
    Moderator
  • Hi,

    New agent was released that does help with unoptimized mode.  Please update the agent and see if that helps.

    KB4015082-Azure Backup Update for Microsoft Azure Recovery Services Agent March 2017

    Issues that are fixed     

                    
    • An issue that was causing failures while mounting VHDs for customers using Azure Backup on Windows Server 2016. It is strongly recommended that MARS Agent, SCDPM 2016 & Azure Backup Server customers upgrade to this version of MARS Agent to avoid experiencing this issue.
    • An issue that was causing backups to fail with errors pointing to failure of operations on specific files.
    • Issues that were causing incremental backups to run in unoptimized mode unexpectedly
    • Several stability and reliability issues pertaining to both Backup and Recovery


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, March 28, 2017 8:05 PM
    Moderator
  • I am installing the update today. I will let you know if it helps.
    Monday, April 17, 2017 3:58 PM
  • Hi,

    We recently discover a code defect that seems to only affects DPM 2016 running on Windows Server 2016 where all online backups may be performed un-optimized.

    The query below can be ran to see if this is occurring on your DPM Server.

       CD C:\Program Files\Microsoft Azure Recovery Services Agent\Temp
       Find /i "m_fUseUsnIterator" cbengine*.errlog 

    If the results show a Zero in m_fUseUsnIterator:0 - then you will need to wait for the new fix.

    <example snip>
    setting reg key for backup in progress:BackupInProgress_35186506057738; m_fUseUsnIterator:0; FileSpec cahnged:0; dwUsn:0
    >example snip<


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Monday, April 17, 2017 4:13 PM
    Moderator
  • 0F20    1ECC    04/19   19:22:21.008    70      replicationcontext.cpp(214)     [000000006A3BE450]      69C5AAC8-52EB-4293-882B-722B03DBA0E2            NORMAL  Hr: = [0x00000000] setting reg key for backup in progress:BackupInProgress_140739057176333; m_fUseUsnIterator:0; FileSpec cahnged:0; dwUsn:0

    I think I have the bug.

    Do you know when it with be fixed? Will be be in a hotfix or a UR?

    Thursday, April 20, 2017 12:13 PM
  • Hi,

    Yes indeed - this is your issue.  The fix will be included in the next MARS agent update but not sure on the exact release schedule.  We do have a private fix. If this is something you cannot wait for you can open a support case and ask for it.  Since it's a code defect you will not be charged.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, April 20, 2017 4:43 PM
    Moderator
  • I can wait depending on roughly how often the MARS agent gets updated.

    Are you thinking weeks or months?

    Thursday, April 20, 2017 8:22 PM
  • Hi,

    The agents typically get updated quarterly, sometime more frequently if they find critical problem or want to rollout a new feature.  If I had to guess, I would say probably June. 


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, April 21, 2017 5:49 PM
    Moderator
  • I guess I can wait as backups are finishing fine.

    If you have a chance could you look at my other question?

    I am trying to determine how space is used in Azure backup.

    https://social.technet.microsoft.com/Forums/en-US/4eaa8623-5452-4959-98b1-cc345560c0e1/getdpmclouddatasource?forum=dpmpowershell

    Friday, April 21, 2017 8:58 PM
  • I have a ticket [REG:117042115635458]. I think it may be related to USN as well. I am going to update to 2.0.9072.0
    Tuesday, April 25, 2017 8:48 PM
  • 2.09072.0 did not appear to help with bug m_fUseUsnIterator:0.
    Thursday, April 27, 2017 12:31 PM