none
DPM 2019: Error 40002 – The VHD containing the replica or one of its snapshots could not be mounted or unmounted RRS feed

  • Question

  • I'm running DPM 2019 v10.19.58.0. No updates available on Microsoft Update. Neither my backup host or client contain or are running on VHD's. The only fix is rebooting both servers and manually running a Sync. This bug was fixed in the latest DPM 2016 rollup but I can't find any acknowledgement of it with DPM 2019. How would I submit a bug report?
    Wednesday, May 22, 2019 1:48 PM

Answers

All replies

  • Hi,

    I believe this isn't fixed yet in DPM 2019, it was fixed in Update Rollup 7 for System Center 2016 Data Protection Manager, but I believe it will be fixed in Update Rollup 1 for DPM 2019.

    Update Rollup 1 for DPM 2019 is planned for Q3 in 2019.

    System Center roadmap updates for 2019
    https://thesystemcenterblog.com/2019/05/10/system-center-roadmap-updates-for-2019

    You can submit a bug report over here:

    https://feedback.azure.com/forums/914488-system-center-data-protection-manager


    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:


    • Edited by Leon Laude Wednesday, May 22, 2019 3:09 PM
    • Marked as answer by darkjedi213x Wednesday, May 22, 2019 5:17 PM
    Wednesday, May 22, 2019 3:08 PM
  • Thanks for the info. I have submitted a bug report here:

    https://feedback.azure.com/forums/914488-system-center-data-protection-manager/suggestions/37713133-dpm-2019-error-40002-the-vhd-containing-the-rep

    Please vote if you have the same issue.

    Wednesday, May 22, 2019 5:17 PM
  • Your're welcome, voted :-)

    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, May 22, 2019 5:19 PM
  • Adding my frustration to the list. This is affecting three of our DPM servers and it's really annoying.

    I tried the suggested fix in the announcement in the forum which didn't help at all.

    I have upvoted the feedback.

    Sunday, June 16, 2019 11:32 PM
  • Leon, tell me the problem is solving by the vendor or not?. The problem is still not solved
    Monday, July 15, 2019 1:07 PM
  • Update

    Update Rollup 1 for DPM 2019 is delayed and has been posponed to Q1 of 2020.

    https://thesystemcenterblog.com/2019/11/07/system-center-roadmap-update


    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, November 7, 2019 4:16 PM
  • For those that are running DPM 2019 on Windows Server 2019. Since installing the October 15 update for Windows 2019 (build 17763.832) I didn't have the issue appear.

    Fix:

    Addresses an issue that might cause error 0x1E, 0xA, or 0x50 to occur during a block cloning operation on an Resilient File System (ReFS) volume because of a race condition

    The same fix is also available in the October 15 update for Windows 2016 but I wasn't able to test that yet

    If not fixed it is at least improving on the situation.

    Marc

    Friday, November 8, 2019 9:17 AM
  • I'm found workaround for this issue. Not needed reboot SCDPM 2019 servers everyday.

    Add to task scheduler this script at once per hour:

    $DPMDisk='' #Put here your DPM Disk name

    $DPMServerName=${Env:ComputerName}
    Disconnect-DPMServer
    $JobsCount=(Get-DPMJob -DPMServerName $DPMServerName -Status InProgress).count
    if ($JobsCount)
    {
     exit 1
    }
    $DiskCount=Get-Disk -FriendlyName 'Msft Virtual Disk' -ErrorAction SilentlyContinue 
    if ($DiskCount)
    {
     exit 1
    }
    Get-Disk -FriendlyName $DPMDisk  | Set-Disk -IsOffline $True
    Get-Disk -FriendlyName $DPMDisk  | Set-Disk -IsOffline $False
    $Status=(Get-Disk -FriendlyName $DPMDisk).OperationalStatus
    if ($Status -like "*Online*")
    {
     Start-DPMDiskRescan -DPMServerName $DPMServerName
     $Alerts=Get-DPMAlert -DPMServerName $DPMServerName -IncludeAlerts AllActive
     if ($Alerts)
     {
      ForEach ($Alert in $Alerts)
      {
       $MountError=$Alert.DetailedErrorInfo.DlsErrorCode

       if ($MountError -eq 40002)
       {
         $Alert.TakeRecommendedAction($Alert.GetAdhocJobsContext())
       }
      }
     }
     exit 0
    }
    else 
    {
     exit 1
    }


    Tuesday, January 28, 2020 7:49 AM
  • This issue doesn't solved on CU1!



    Thursday, February 6, 2020 1:01 PM
  • Confirmed. Issues/errors persist.

    Works fine for a while, then fails on some random volume/replica. Both scheduled and manual consistency check fail repeatedly until the server is rebooted. Once rebooted, everything works fine for a day or two, then repeat the cycle.

    For the record, we never had this issue in 2016. It wasn't until we 'upgraded' to 2019 that the issues began.

    Thursday, February 6, 2020 5:51 PM
  • @Alexandr Klimenok,

    so basically your scripts sets the DPM storage to offline state and then back to online (when no DPM jobs are running), rescans the storage in DPM and then restarts the DPM jobs that ended in the VHD mount error. Am I correct?

    Also, DPM UR1 huge disappointment since it doesn't solve with the most glaring issues. And we've waited a year for this...

    Friday, February 7, 2020 3:07 PM
  • @MarkosP

    Yes, you are correct.

    I think SCDPM 2019 doesn't close some file handles after last mount vhdx.

    Monday, February 10, 2020 11:55 AM
  • My script doesn't help after install UR1, jobs crashes with id 40003. Needs reboot server after this issue.

    I think enough spend time for SCDPM, Microsoft doesn't help us. We will to migrate to Veeam and CommVault.

    Tuesday, February 18, 2020 1:02 PM
  • Although the workaround script does resolve the issue, unfortunately I have had no success in getting this script to run as a Scheduled Task. The task runs successfully when run manually, but fails with error 0x1 (Task Scheduler), error code 2147942401 from powershell.exe, every single time when it runs on a schedule.

    Is there some trick to get it to actually run on a schedule?

    Thursday, February 20, 2020 2:02 AM
  • @eforgas

    Please use this script for UR1:

    $DPMDisk='' #Put here your DPM Disk name

    $DPMServerName=${Env:ComputerName}
    Disconnect-DPMServer
    $JobsCount=(Get-DPMJob -DPMServerName $DPMServerName -Status InProgress).count
    if ($JobsCount)
    {
     exit 1
    }
    Get-Disk -FriendlyName $DPMDisk  | Set-Disk -IsOffline $True
    Get-Disk -FriendlyName $DPMDisk  | Set-Disk -IsOffline $False
    $Status=(Get-Disk -FriendlyName $DPMDisk).OperationalStatus
    if ($Status -like "*Online*")
    {
     Start-DPMDiskRescan -DPMServerName $DPMServerName
     $Alerts=Get-DPMAlert -DPMServerName $DPMServerName -IncludeAlerts AllActive
     if ($Alerts)
     {
      ForEach ($Alert in $Alerts)
      {
       $MountError=$Alert.DetailedErrorInfo.DlsErrorCode

       if ($MountError -eq 40002)
       {
         $Alert.TakeRecommendedAction($Alert.GetAdhocJobsContext())
       }
      }
     }
     exit 0
    }
    else 
    {
     exit 1
    }

    Friday, February 21, 2020 1:03 PM
  • Update March 10, 2020
    You can now try the registry changes mentioned by the product team below:

    The error 40002 could be reported in multiple scenarios of VHD mount and unmount failure.
    Here are more details -

    WMI Out of memory (Fixed with UR7 for DPM 2016 and already part of DPM 2019 RTM):
    One of the top scenario where we observed the 40002 error is due to WMI out of memory. In a scenario, where you see 40002 error in DPM Console and at the same time you may see WMI-Activity error event ID 5858 logged with Result code 0×80041006. This code translates to “not enough memory for the operation’. To address this issue, with DPM 2016 UR 7 we increased the default WMI memory limit from 20MB to 60MB by adding registry keys (details below). This change is already part of DPM 2019 RTM.

    If you still see the same failure, the work around could be to increase this limit a bit more. But if the issues persists, there could be more reasons why the calls to WMI might fail. It could be due to inadequate OS resources causing WMI to perform slow or the degraded performance of underlying storage. Our team is investigating further on how we can isolate these issues.

    Registry Key Details:
    Reg Key : HKLM\SOFTWARE\Microsoft\Wbem\CIMOM
    Value: High Threshold On Events (B)
    Data (String) : 60000000

    Reg Key : HKLM\SOFTWARE\Microsoft\Wbem\CIMOM
    High Threshold On Client Objects (B)
    Data (String) : 60000000

    Sparse file issue with Windows Server 2016:
    If this issue is specifically seen, when DPM 2019 /2016 is installed on Windows Server 2016, please check if event ID 27268 is logged in the Hyper-V-VMSS logs with the description as “Virtual hard disk files must be uncompressed and unencrypted and must not be sparse”.
    To resolve this issue , install the latest updates for Windows Server 2016.

    If you are still experiencing high frequency of this error, please reach out to our customer support team to troubleshoot further.

    https://feedback.azure.com/forums/914488-system-center-data-protection-manager/suggestions/37713133-dpm-2019-error-40002-the-vhd-containing-the-rep?tracking_code=06d168d80533d0a3f3b623c091d53a8c


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, March 10, 2020 7:11 AM
  • @Leon Laude

    These settings doesn't help for mount/unmount issue. I have opened case for DPM 2019 CU1 on Microsoft support. They can't help me with this issue. I think it a bug or architecture hole of SCDPM 2019.

    Friday, March 13, 2020 9:33 AM
  • 2) Increase the WMI MemoryPerHost setting.

    a. Start “wbemtest” as administrator.
    b. Connect to the “root” namespace (not “root\default”, just “root”)
    c. Click Open Instance. Specify “__ProviderHostQuotaConfiguration=@”
    d. Select Local Only for easier readability. You will see the threshold values.
    e. Increase the MemoryPerHost 1073741824
    f. Save Property
    g. Save Object
    h. Click Exit.

    3) Reboot the DPM / MABS Server.

    Regards
    Mike Jacquet


    I'v found this in DPM 2016 thread with the same problem. This settings helps me for now - 6 days with no VHD error. But i'm still testing.
    Monday, March 30, 2020 8:52 AM
  • This solution will not help. I still see error 40002 all registry settings have been added. I also installed
     UR1 and failure again.
    • Edited by Aleks__ Monday, March 30, 2020 7:28 PM
    Monday, March 30, 2020 7:28 PM
  • Registry settings does not help me too, but this solution (using wbemtest) looks like works - for now no 40002 errors for allmost two weeks. Before it was only 2 - 3 days to get an error
    Saturday, April 4, 2020 1:36 PM
  • The possible errors behind the error ID 40002 are quite many, it is sort of a "generic error".

    The DPM engineers identified the WMI memory limit to be "one" out of many possible issues, so it does not necessarily fix everything as a few of you already mentioned.

    It is therefore important that you reach out to our customer support team so they can troubleshoot further and hopefully identify the issue.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Saturday, April 4, 2020 3:42 PM