none
The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)

    Question

  • let me start with - DPM....GRRRRRRR!

    With that out of the way...

    I  am continually getting:

    "The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)"

    during a file server volume sync.  The PG is made up of ONE file server's two volumes E:\ and F:\.  One volume will sync just fine, the other will fail with this error.  I reboot the DPM server, it works for a sync or two, then back to this error.  I see NOTHING to indicate what the problem is or how to even approach this issue.

    Any help is appreciated!

    D


    DLovitt

    Wednesday, November 16, 2016 9:14 PM

All replies

  • I'm getting this, too.
    Thursday, December 1, 2016 9:37 PM
  • I've seen this when i reboot the DPM server while it's doing something. No big surprise though.
    Friday, December 2, 2016 2:48 PM
  • This recently started happening with me too. For me we upgraded to DPM 2016 and the workloads using modern storage experience this from time to time. A reboot of the DPM server resolves the issue for a while and then it comes back. I think that it's possible that backup is interfering with the data dedup that is running on the VHDX files for modern storage. The host server that is hosting the DPM VM is also holds the storage pool. We do not have a separate SMB file server.
    Monday, December 5, 2016 7:26 PM
  • @P_Brock: So I'm clear, you're deduping the target VHDX files (the files you're backing up to) and you think this is possibly interfering?  If I have this right, then as a point of comparison, that is not my situation.  I am not deduping my backup drives... yet. That said, I too have one physical server. DPM is a VM on that host and the host server also hosts the storage pool holding the VHDX target drives used by DPM.  

    DLovitt

    Monday, December 5, 2016 7:55 PM
  • I'm seeing some "errors detected during paging operation", "reset to device", "failed to mount the virtual disk", "filter manager failed to attach" and other disk-related errors in the Eventlog. The device names are hard to track down (like \Device\Harddisk7\DR1431 and \Device\HarddiskVolume1877), but I think these are virtual disks associated with Modern Storage...

    The end result is replicas are getting corrupted or something. Consistency check won't fix them. I have to delete and re-protect. I get one or two replicas corrupted every day.

    This could be some kind of hardware storage issue on my end. I'm looking into that. But I don't see any hardware errors and this is a brand new server.

    Monday, December 5, 2016 8:51 PM
  • I stand corrected, actually a reboot does allow me to run a consistency check on those replicas and get them back into working order without unprotect/delete/reprotect. That might kinda explain these errors:

    Filter Manager failed to attach to volume '\Device\HarddiskVolume3368'. This volume will be unavailable for filtering until a reboot.  The final status was 0xC03A001C.

    Monday, December 5, 2016 9:28 PM
  • Is there any word on a possible patch to fix this? This was not a known issue with DPM 2012 R2. A reboot does fix the problem but what causes this and why now?
    Tuesday, December 13, 2016 4:27 PM
  • Hi,

    DPM 2016 installed on Windows Server 2016 uses Modern Backup Storage whereby each protected data source is now backed up to a .vhdx file stored on the DPM Storage pool volume.  The .vhdx file is mounted when a new recovery point or consistency check is performed and dismounted when the job is complete.  DPM leverages Hyper-V infrastructure to perform the mounts / dismounts and it sounds like something is going wrong there.

    You can look in the msdpm*.errlog files to see if there are any log entries detailing the problem.  Good log entries would look like this.

    42F4    1104    12/13   14:04:04.088    11      VhdContainer.cs(220)                    NORMAL  MountStorage: Succeeded for RequestId: 81861e01-7068-493a-8803-7f2a78a94dec
    42F4    8698    12/13   14:04:21.034    11      VhdContainer.cs(272)                    NORMAL  UnmountStorage: Succeeded for RequestId: 81861e01-7068-493a-8803-7f2a78a94dec

    You can use this find command to see if you can locate a failure.

    C:\Program Files\Microsoft System Center 2016\DPM\DPM\Temp>find /i "vhdcontainer.cs(2" msdpm*.errlog

    I would also suggest checking your anti-virus software and exclude the volume(s) used for the storage pool and see if that helps.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, December 13, 2016 5:19 PM
    Moderator
  • We have Windows Defender on both the DPM VM and DPM disk server (Windows Server 2016 Built-in). During the scan time, there were no mount/unmounts issues. On a side note, this issue only happens to our file servers.

    Here is a snippit of what my log showed around the time the mounting error occurred. I noticed the top states that it can't mount but then at the end of the error it states that it had a successful unmounts. No where in between does it mention a successful mount. Are RequestId's different each time DPM wants to mount storage or are they the same per each recovery point, etc? I noticed after the reboot when I kicked off the failed sync job the request id was different.

    =============================================================

    WARNING           MountStorage: Failed for RequestId: 55cabbfd-c94e-4109-b8ac-c7d6a6a24b85 with exception: Microsoft.Internal.EnterpriseStorage.Dls.StorageManager.StorageManagerException:

    Failed to mount vhd: Microsoft.Internal.EnterpriseStorage.Dls.Wmi.NonFatalWmiException: Job \\DPMSERVER\root\virtualization\v2:Msvm_StorageJob.InstanceID=""B7D3118D-06F3-46E1-918A-1B3575BBAD89""

    failed with status Exception",,,,

    WARNING              at Microsoft.Internal.EnterpriseStorage.Dls.Wmi.ResilientWmiCommand.WaitForJobCompletion(ManagementScope managementScope", String jobPath, Boolean throwOnFail),,

    WARNING              at Microsoft.Internal.EnterpriseStorage.Dls.Wmi.ResilientWmiCommand.InvokeMethod(ManagementScope managementScope", ManagementObject mObj, String methodName, Object inParams, Boolean throwOnFail)

    WARNING              at Microsoft.Internal.EnterpriseStorage.Dls.StorageManager.VhdMountManager.MountVhd(String path", VhdMountMode mode),,,

    WARNING           *** Mojito error was: WMIFailedToExecuteQuery; 0; None",,,,

    WARNING              at Microsoft.Internal.EnterpriseStorage.Dls.StorageManager.VhdMountManager.MountVhd(String path", VhdMountMode mode),,,

    WARNING              at Microsoft.Internal.EnterpriseStorage.Dls.StorageManager.VhdMountManager.MountInternal(VhdMountInstance instance)",,,,

    WARNING           *** Mojito error was: SMVhdMountError; 0; None",,,,

    NORMAL             UnmountStorage: Succeeded for RequestId: 55cabbfd-c94e-4109-b8ac-c7d6a6a24b85",,,,




    • Edited by renegade34g Wednesday, December 14, 2016 9:27 PM
    Wednesday, December 14, 2016 9:25 PM
  • HI,

    Interesting, so this appears to be related to some wmi problem - but scratching my head why it only seems to effect the file server workloads you are protecting. I don't have anything on this error, so can't offer a solution.  If others are seeing the same errors then I guess someone should open a case to have it investigated. 

    The unmount was probably issued as a result to the mount failure as a precaution to ensure it is unmounted. Some calls report success because they didn't get an error, so for instance back to back unmounts would succeed even though only the first unmount actually performed the work.  Yes, the request ID's are unique each time.

    UPDATE1 12-16-16 - We have had a few customers report this now and we are actively investigating the issue. More to come as we learn more.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Thursday, December 15, 2016 12:05 AM
    Moderator
  • I'm the OP and haven't chimed in for a while. 

    As I stated in the initial post, it was only a problem on my file server too. And only for one backed up volume at a time. 

    I don't have access to my logs anymore because yesterday, I rebuilt my entire DPM solution.  I rebuilt the physical host server (Win2016), rebuilt the DPM VM (Win2016), reinstalled SQL for DPM (SQL2014) and reinstalled DPM 2016.  This time, when I created the new Protection Group for my large file server however, I used 2 different VHDX files/drives, one for each protected volume on the file server. So, the file server's "E:\" drive backs up to the E:\ drive on DPM and the file server's "F:\" drive backs up to the F:\ drive on DPM (it's purely a coincidence that the drive letters are the same- ;) FYI).  I'll see how this goes.  This somehow makes sense to me to be a possible solution - but likely wishful thinking. 

    I will say, if this continues to be a problem, among the MANY other problems and deficiencies I've see with DPM, then I for one am done with DPM and will move on to another solution.


    DLovitt

    Thursday, December 15, 2016 12:24 AM
  • Thanks for the info Mike. My team and I are going to look into some other things and post back anything that may help. And if we don't find anything or all out hair has been pulled out, we shall submit a ticket.
    Thursday, December 15, 2016 4:31 PM
  • I setup DPM 2016 (CU 1) on Server 2016 (local Volumes so new storage pool added 18TB Volume which DPM formatted ReFS) about 3 days ago and I've just had this on a couple of 2010 protected exchange mail databases... a restart doesn't fix it for me I have to un-protect then delete the inactive protection then re-protect which isn't a great solution... half the reason for DPM is so i can look back in time if i need to....
    Thursday, December 15, 2016 6:06 PM
  • I went through and added some synchronization offsets so that all my protection groups weren't hitting at the same time. I also suspect there's some default background filesystem maintenance/optimization processes or something using up additional I/O, but I haven't been able to nail that down. So at least in part I think this may be related to some I/O starvation hitting a timeout somewhere.

    Anyway, implementing the offsets has calmed down the instances of replicas going inconsistent and un-fixable until reboot. However, I still get a bunch of disk, VDS Basic Provider, vhdmp, FilterManager errors on a daily basis.

    Tuesday, December 20, 2016 4:51 PM
  • Hi.

    This is really a huge problem. We have these issues too with one of our customers DPM. We are protecting 16 SAP systems with 4 SQL DBs each and some filefolders. We have two PGs, one for SQL, one for Files. We sync both every 15 minutes.
    We are getting these different errors, all sporadically, no schema recognizable:

    The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)
    There is not enough space in DPM storage pool volume   (ID 33515)
    The VHD specified for mount or dismount operation cannot be found. (ID 40010)
    The specified storage is unusable as it is a system volume, has existing data, is in an unhealthy state or formatted incorrectly. (ID 40005)
    and of course the volume missing errors.

    What I found out:
    Volume missing can be fixed with a "Start-DPMDiskRescan" or the corresponding button "Rescan" in the DPM Managementconsole. Sometimes the volume missing errors will increase wehen you first do the rescan. Just wait around 30 seconds and then run the rescan again, then all volume missing errors will go away.
    The more jobs run at the same time, the more errors will occur. Especially the volume missing error will raise a lot, when the file syncs all start at the same time.

    I have spent some hours over the christmas days and developed some scripts to serialize the backups. With this method just one job will run at a time and the scripts will do some automatic fixes (e.g. trigger a CC or clean up the volume missing errors). Certainly not perfect, but they work for us. Of course, with the serialization the jobs do not succeed in 15 minutes, and the express full backups will run 2-3 hours, but it is better to have a sync or incremental every 30 minutes or every few hours than nothing.

    The scripts explained:
    Please test and use the scripts at your own risk!
    Create a folder C:\Scripts\DPM and extract the ZIP file from here https://dataonegmbh-my.sharepoint.com/personal/am_dataone_de/_layouts/15/guestaccess.aspx?docid=01e1e3f6642af4399adccd605d332698d&authkey=AXjIrLt7OwqHdp2clHc_IHc
    Create-ExpressFull-DB.cmd -> Schedule this one to run the full express backups for the SQL DBs, e.g. every day at 8pm
    Create-RecPoint-File.cmd -> Schedule this one to run the recoverypoints for the filefolders, e.g. every day at 6pm
    Create-IncSync.cmd -> Schedule this one to run the incremental backups of the SQL-DBs and the synchronization jobs for the filefolders, every day, every 15 minutes
    You have to modify the PGs to run the syncs/incrementals "just before a recoverypoint" and update the days the jobs run on a weekly basis.

    Even with the scripts, some jobs fail 4-5 times and the 6th try is sucessfull.
    I did not have to reboot the server, nor did I have to reprotect some datasources - until now.

    I hope Microsoft will fix this in a short term. We have a Call open, but no solution by now. I will keep you up to date.

    Best regards
    Andreas

    Monday, December 26, 2016 11:02 PM
  • I have seen this issue too. Workaround: Reboot DPM server, consistency check on the data source with the error, manual protection point. I've only seen this twice in 4 weeks since we switched to MBS so the workaround is good enough for me.
    Wednesday, December 28, 2016 10:25 AM
  • Hello Christian,

    I work in the PG team at DPM. We are trying to root cause this bug.

    Would it be possible for you to share info about the bug with us? If you, can you please connect with us at AskAzureBackupTeam@microsoft.com

    It would really help us out.

    Thanks and Regards,
    Aditi Gangwar.

    Thursday, December 29, 2016 2:47 PM
  • Hello MarxA,

    I work in the PG team at DPM. We are trying to root cause this bug.

    Would it be possible for you to share info about the bug with us? If you, can you please connect with us at AskAzureBackupTeam@microsoft.com

    It would really help us out.

    Thanks and Regards,
    Aditi Gangwar.

    Thursday, December 29, 2016 2:48 PM
  • Hello t3rmin,

    I work in the PG team at DPM. We are trying to root cause this bug.

    Would it be possible for you to share info about the bug with us? If you, can you please connect with us at AskAzureBackupTeam@microsoft.com

    It would really help us out.

    Thanks and Regards,
    Aditi Gangwar.

    Thursday, December 29, 2016 2:48 PM
  • Hello k-strider,

    I work in the PG team at DPM. We are trying to root cause this bug.

    Would it be possible for you to share info about the bug with us? If you, can you please connect with us at AskAzureBackupTeam@microsoft.com

    It would really help us out.

    Thanks and Regards,
    Aditi Gangwar.

    Thursday, December 29, 2016 2:49 PM
  • Hello renegade34g,

    I work in the PG team at DPM. We are trying to root cause this bug.

    Would it be possible for you to share info about the bug with us? If you, can you please connect with us at AskAzureBackupTeam@microsoft.com

    It would really help us out.

    Thanks and Regards,
    Aditi Gangwar.

    Thursday, December 29, 2016 2:49 PM
  • Hello Darius,

    I work in the PG team at DPM. We are trying to root cause this bug.

    Would it be possible for you to share info about the bug with us? If you, can you please connect with us at AskAzureBackupTeam@microsoft.com

    It would really help us out.

    Thanks and Regards,
    Aditi Gangwar.

    Thursday, December 29, 2016 2:49 PM
  • Hello P_Brock,

    I work in the PG team at DPM. We are trying to root cause this bug.

    Would it be possible for you to share info about the bug with us? If you, can you please connect with us at AskAzureBackupTeam@microsoft.com

    It would really help us out.

    Thanks and Regards,
    Aditi Gangwar.

    Thursday, December 29, 2016 2:50 PM
  • I sent an email to the address you specified above, but I thought I'd include the contents here as well:

    As I’d mentioned in my last response to the forum post, I’ve rebuilt my entire DPM solution.  Since the rebuild, I’ve had no issues with my file server replica’s vhd going offline or not being able to mount or unmount… until this morning.  Now, one of the file server volumes is not synchronizing with error “The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)”  The things I have to offer right now are:

    1.      My original build of the DPM 2016 solution was an upgrade from 2012 R2.  The 2012 R2 solution was very new and I was still in testing mode when I performed in-place upgrades to 2016, following most of MS’s guidance on such an upgrade.  It was a COMPLETE upgrade, meaning I upgraded the host server (H-V) from 2012 R2 to 2016, the guest DPM server from 2012 R2 to 2016, and the DPM install from 2012 R2 to 2016.  I don’t recall what version of SQL I had on that server but suffice it to say, it was local to the DPM guest and was compatible to both versions.  I did not perform the upgrade to accommodate transitioning the storage vhdx files from one version to the other, but rather created all new Modern Backup Storage vhdx files, so I likewise recreated all of my PGs as well. 

    2.      After completely rebuilding my DPM solution, I did break out my file server’s volume replications into two different disk storage targets on DPM.  This had the appearance (until this morning, at least) to have perhaps alleviated the problem to some degree.  That is obviously not the case anymore. 

    By the way, the grammar in the error message is incorrect.  It should read “or one of its snapshots” not “it’s”.  “It’s” means “it is”.  Yea, I know… grammar police… but I can’t help it - lol.


    DLovitt

    Thursday, December 29, 2016 4:18 PM
  • After a while i am seeing high memory usage on our DPM 2016 server running on Windows Server 2016. 15,8 GB of 16 GB used. With the sysinternal tool RamMap i can see that over 10 GB are used by something called Metafile. This issue has been described with Windows Server 2008/2008R2, for example here: https://prashantd.wordpress.com/2013/11/14/solved-high-memory-usage-by-metafile-on-windows-2008-r2/

    Rebooting the server or using RAMMap -> Empty Working Set fixes this for a while.

    I once managed to observe this when our fileserver was backed up. Around 5 TB of data, lots of small files (user profiles). Not shure on how and why this happens or if this is even a bad thing. 

    Maybe the problem described in this thread is only happening when the server is low on available memory?

    Are you guys seeing the same thing? 

    Tuesday, January 3, 2017 8:09 AM
  • Could you tell me if you have followed this guidance within your virtual DPM server for your DPM storage pool drive as outlined in this KB:-

    https://technet.microsoft.com/en-us/library/hh757836(v=sc.12).aspx

    I too am having this issue and noticed a lot of defrag (dynamic optimisation) events in the application log. I looked into that as a potential cause and found this article.

    Also can you tell me if your "Bytes per filerecord segments" are configured at 4096 as outlined in this article:-

    http://blog.budo.net.au/post/2016/04/19/the-danger-of-dedup-with-dpm

    Just interrested to see what your underlying NTFS / deduplicated volumes look like as this appears to be the current reference architecture and it's certainly how I have been deploying DPM 2016.

    Here is Microsoft's official recommendations for configuring de-duplicated underlying storage for a DPM virtual machine:-

    https://technet.microsoft.com/en-us/library/dn891438(v=sc.12).aspx

    Thanks.


    • Edited by V J Meyer Tuesday, January 24, 2017 2:49 PM Enhance Post
    Tuesday, January 3, 2017 3:28 PM
  • We have a physical Windows Server 2016 with local disks. These are a raid 6 volume (dedicated RAID controller) formated as ReFS and used as modern backup storage. I am also seeing a lot of Defrag events in Event Viewer -> Windows Logs -> Event Viewer:

    The storage optimizer successfully completed retrim on 33b8312214ac40b1b8df9b4177665c8e (C:\Program Files\Microsoft System Center 2016\DPM\DPM\Volumes\Replica\33b83122-14ac-40b1-b8df-9b4177665c8e)

    We are not running chkdsk or defragmentation manually, this happens automatically.

    We do not use deduplication on the dpm server or the underlying storage. We protect a server that uses dedup though, 300 GB VMM server with Hyper-V templates and ISOs.

    Wednesday, January 4, 2017 9:09 AM
  • Hi Christian,

    The events 258 from defrag that detail storage optimizer retrim against the DPM replica .vhdx file is normal.  This ensures any files or blocks freed inside the .vhdx file get freed on the underlying DPM storage pool volume.

    More on trim can be found in below article. 

    TRIM/UNMAP Support in Windows Server 2012 & Hyper-V/VHDX 


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, January 4, 2017 6:56 PM
    Moderator
  • After a while i am seeing high memory usage on our DPM 2016 server running on Windows Server 2016. 15,8 GB of 16 GB used. With the sysinternal tool RamMap i can see that over 10 GB are used by something called Metafile. This issue has been described with Windows Server 2008/2008R2, for example here: https://prashantd.wordpress.com/2013/11/14/solved-high-memory-usage-by-metafile-on-windows-2008-r2/

    Rebooting the server or using RAMMap -> Empty Working Set fixes this for a while.

    I once managed to observe this when our fileserver was backed up. Around 5 TB of data, lots of small files (user profiles). Not shure on how and why this happens or if this is even a bad thing. 

    Maybe the problem described in this thread is only happening when the server is low on available memory?

    Are you guys seeing the same thing? 

    I'm seeing this exact same issue, though "empty working set" doesn't fix all instances of it, it just seems to move the metafile from active to standby memory.  I've had this happen twice now and rebooting fixes the issue, but I'm not sure why the metafile is getting so large.

    Monday, January 9, 2017 3:00 PM
  • Hi,

    You can try adding this registry value and see if it helps with memory usage.

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
        REG_DWORD:RefsEnableLargeWorkingSetTrim

    Set value RefsEnableLargeWorkingSetTrim = 1

    Reboot the DPM Server when it's idol and monitor.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, January 10, 2017 11:54 PM
    Moderator
  • Same Problem here...
    Wednesday, January 11, 2017 8:01 AM
  • I also have the "Failed to mount vhd" error in my DPM logs with the same symptoms, i.e. file servers backup jobs are failing.

    Often rebooting DPM, running synchronisation from the DPM warning and then creating a recovery point from the DPM error resolves the issue, but usually only for a day, then the problem returns.

    I'm running DPM2016 UR1 on a physical server with plenty of RAM and disk space, using MBS.

    It's a production environment, so this is a major problem for us. We can't afford to remove protection as we will then lose recovery points which are essential for our business. The only thing I can think to do is go back to the old storage mechanism and not use MBS.

    I have enough storage volumes to free one up and do this, but the only way I can think to migrate back without losing recovery points is to backup the entire VM for the file servers so that I can keep the individual disk recovery points on the MBS volume until their retention period expires.

    Before I attempt this is there any further advise? ...or anything new from the DPM team?

    Wednesday, January 11, 2017 11:18 AM
  • If this is a problem for you, you should open a case with Microsoft. For us it is annoying, but not really a problem.

    I've noticed that backups of one of the volumes on our fileserver takes a long time. 15 minutes+ for synchronisation, up to 50 minutes for the synchronisation after midnight (00:xx).

    This is an archive volume. So old roaming profiles, old data. Lots of small files, some of them with NTFS compression. I am deleting some of the old data, disabling NTFS compression. I will also zip some folders with lots of small files, will see if that helps.

    Edit: Just noticed this is the same volume that sometimes has the ID 40002 VHD mount problem.
    Wednesday, January 11, 2017 1:45 PM
  • People from Microsoft, including Aditi Gangwar who is Program Manager on the DPM team have posted in this thread. I'm not sure how raising an additional case with Microsoft would help me...
    Wednesday, January 11, 2017 2:12 PM
  • Hi,

    You can try adding this registry value and see if it helps with memory usage.

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
        REG_DWORD:RefsEnableLargeWorkingSetTrim

    Set value RefsEnableLargeWorkingSetTrim = 1

    Reboot the DPM Server when it's idol and monitor.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Mike- I'm unclear.  Are you indicating that the high memory usage (which I too am seeing) is:

    a. indeed part of or even THE problem that this thread is about? 

    b. another issue all together?

    c. not an issue per se, but here is a registry hack, if you're interested?


    DLovitt

    Wednesday, January 11, 2017 11:35 PM
  • I agree. 

    Christian, it's great that you don't consider this to be much of an issue.  Unfollow and move on.  For the rest of us, we must figure this out.  This is a major problem for me, because my backup system DOESN'T ACTUALLY WORK.  Raising a case with MS is pointless, I believe, at this moment.  They have elicited feedback and logs from everyone on this thread.


    DLovitt

    Wednesday, January 11, 2017 11:39 PM
  •  I am also having this exact same error. In my particular case, DPM seems to work normal, but as soon as the communication is lost with the storage system, the issue arises. I can even reproduce it. For example, I am using an external iSCSI target device which if I physically disconnect it for a while of the network, then reconnect, the issue begins to happen. It also happens if I touch the iSCSI configs like to disconect/reconnect. A reboot seems to help, but not always. I had to re-do the Protection Groups for DPM to start working again. Kind of frustrating.

    Jose Angel Rivera

    Thursday, January 12, 2017 6:19 AM
  • I want to figure this out as well. I just won't be losing any sleep over a backup of a less important archive volume.

    I removed our problematic volume from backups. This volume had long backup times and was failing with ID 40002 every couple of days. Didn't help, memory use is up again.

    Thursday, January 12, 2017 6:12 PM
  • I have set the RefsEnableLargeWorkingSetTrim Registry entry on my DPM server and rebooted the server. We still get the ID 40002 error and the high memory use on our DPM 2016 server.
    Friday, January 13, 2017 12:40 PM
  • For info, here's an extract from my DPM logs showing the failure:

    1438 267C 01/11 00:05:01.887 11 VhdContainer.cs(224) WARNING MountStorage: Failed for RequestId: 2880ad56-7036-43df-8e26-60c3b538366c with exception: Microsoft.Internal.EnterpriseStorage.Dls.StorageManager.StorageManagerException: Failed to mount vhd: Microsoft.Internal.EnterpriseStorage.Dls.Wmi.NonFatalWmiException: Job \\UK-DPM04\root\virtualization\v2:Msvm_StorageJob.InstanceID="D321BB93-D7BE-4EDB-8CB6-0BFBA634462B" failed with status Exception
    1438 267C 01/11 00:05:01.887 11 VhdContainer.cs(224) WARNING   at Microsoft.Internal.EnterpriseStorage.Dls.Wmi.ResilientWmiCommand.WaitForJobCompletion(ManagementScope managementScope, String jobPath, Boolean throwOnFail)
    1438 267C 01/11 00:05:01.887 11 VhdContainer.cs(224) WARNING   at Microsoft.Internal.EnterpriseStorage.Dls.Wmi.ResilientWmiCommand.InvokeMethod(ManagementScope managementScope, ManagementObject mObj, String methodName, Object inParams, Boolean throwOnFail)
    1438 267C 01/11 00:05:01.887 11 VhdContainer.cs(224) WARNING   at Microsoft.Internal.EnterpriseStorage.Dls.StorageManager.VhdMountManager.MountVhd(String path, VhdMountMode mode)
    1438 267C 01/11 00:05:01.887 11 VhdContainer.cs(224) WARNING *** Mojito error was: WMIFailedToExecuteQuery; 0; None
    1438 267C 01/11 00:05:01.887 11 VhdContainer.cs(224) WARNING   at Microsoft.Internal.EnterpriseStorage.Dls.StorageManager.VhdMountManager.MountVhd(String path, VhdMountMode mode)
    1438 267C 01/11 00:05:01.887 11 VhdContainer.cs(224) WARNING   at Microsoft.Internal.EnterpriseStorage.Dls.StorageManager.VhdMountManager.MountInternal(VhdMountInstance instance)
    1438 267C 01/11 00:05:01.887 11 VhdContainer.cs(224) WARNING *** Mojito error was: SMVhdMountError; 0; None
    1438 1FF8 01/11 00:05:01.887 27 MountDismountStorageBlock.cs(181) 2880AD56-7036-43DF-8E26-60C3B538366C WARNING Failure: Mount Storage task for replicaId 590104ea-5e94-41dc-b1b4-a5a072711c68, pidId 00000000-0000-0000-0000-000000000000 with Error code = SMVhdMountError
    1438 1FF8 01/11 00:05:01.887 27 FsmBlock.cs(130) 2880AD56-7036-43DF-8E26-60C3B538366C WARNING Backup.MountStorageBlock : <-- Exited FSM block with FAILURE (errorCode = SMVhdMountError) ErrorInfo = SMVhdMountError 

    Monday, January 16, 2017 11:11 AM
  • HI,

    our DPM 2016 (on Server 2016) also experienced this issue today, a reboot seems to temporarily fix the problem.

    regards

    Stefan

    Friday, January 20, 2017 7:03 AM
  • Any update on this Microsoft? ...I need to decide whether to revert my disk array's to the old storage methed, so any response on the status of this issue would be appreciated...
    Tuesday, January 24, 2017 9:11 AM
  • ive got that debuging loggin on now though its not done this since.  but im getting something when recovering that might be similar related...  i often recover our shares to external storage for offsite backup and its always just worked but now since i setup 2016 (CU1) it now fails with this error:

    DPM encountered an error while reading from the recovery point used for recovery.  Either the recovery point no longer exists or if you selected a share for recovery the path to its contents is...

    you cant se any more of the error

    Tuesday, January 24, 2017 3:11 PM
  • Hi everyone.

    Today we had a quiet long session with several people from Microsoft on one of our DPM servers experiencing the issue. Reproduced the issue and collected bunch of logs. Hope they will find a solution now.

    Best regards
    Andreas

    Tuesday, January 24, 2017 7:05 PM
  • I'm noticing periods where all I/O to the MBS storage just pauses for up to several hours. Usually it's when DPM is working pretty hard. Often no errors are logged and 4 hours later, it continues. I don't believe it's a hardware issue. The disks have been tested, there are no hardware errors anywhere. And it's extremely performant. It feels more like some kind of software "clog". Too many things open at once exhausting some limit somewhere. Something like that.
    Tuesday, January 24, 2017 7:45 PM
  • If have the same problem.


    Gert MCITP SA, EA & VA, MCSE Private Cloud, MCSA Windows 2012

    Thursday, January 26, 2017 9:17 AM
  • Hi,

    After some internal investigations we have found some code defects that can lead to this condition/error and are currently testing some fixes. Based on test results we're hoping to include the fix in the next update rollup.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, January 26, 2017 5:23 PM
    Moderator
  • Please hurry! I'm fighting with DPM every single day. Nightmare!
    Thursday, January 26, 2017 6:14 PM
  •  Based on test results we're hoping to include the fix in the next update rollup.

    Hello Mike,

    so will there be no single Hotfix for the Problem? Or do i need to Contact Support to get the Hotfix?

    I have to reboot DPM every Second Day now, to get all Backup Jobs running again. No Fun :(

    regards

    Stefan

    Friday, January 27, 2017 7:38 AM
  • Mike,

    Will this fix address all the issues discussed in this thread, or just one? We've been suffering from high memory usage (we're now limiting the size of the disk cache), pauses to MBS disk IO, resulting hangs in the DPM console, vhdmp timouts/resets, Disk warnings for VHDs, FilterManager errrors, as well as less frequent issues mounting replicas. We've found that rescheduling our PGs so that BMR recovery points don't overlap with any other operations has helped a lot.

    Can you share a timescale for the next rollup?

    Regards,

    Tim

    Friday, January 27, 2017 12:15 PM
  • I'm having the exact same issue too. DPM is all hardware based, server 16 and dpm 16 with a 40TB RAID 10 dedicated to the modern storage volume. In addition to the constant protection group failures related to the VHD not being mountable, I have noticed that the processor use on my server is maxed out by the msdpm.exe process. This DPM server was rebuilt on the exact same hardware as dpm 12r2 on server 12r2 and 8 cores of processor was more than sufficient. I purchased the second 8 core processor for this server and wouldn't you know that it will floor all 16 processor cores to 100% while its backing up systems. Not a single PG on this DPM 16 has on-the-wire compression turned on either. I have no idea why transferring data from one server to another would require 16 processors at 3.5Ghz each but its using them.

    -Jason

    Friday, January 27, 2017 4:21 PM
  • Hi,

    UPDATE:  The product group is in the final testing phase for UR2 and feel it will be best to wait for the comprehensive update that address this and other issues related to modern backup storage.  Barring any severe issues that update should be released early part of February.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Friday, January 27, 2017 6:54 PM
    Moderator
  • Mike, can you confirm the private fix is available or still a work in process. I emailed as suggested and no response thus far.

    Thanks! K

    Tuesday, January 31, 2017 1:20 PM
  • I got no response to my email either.
    Tuesday, January 31, 2017 7:19 PM
  • I have installed the private "fix", and the original issue that I started this thread for (VHD could not be mounted) appears to not come up anymore.  That said, there are still SIGNIFICANT issues with this version of DPM (in my environment at least) that appear to center around this new Modern Backup Storage and ReFS.  Every day I must, at minimum, reboot the DPM server to get backup/syncs operational again.  Not to mention the continual other issues like seemingly random consistency checks on VMs, random recovery points that take ages, Syncs that suddenly go at a snails pace for no apparent reason, inability to navigate to recovery paths deeper than about 130 characters, etc...

    DO NOT MIGRATE TO THIS SOLUTION UNTIL UR2 IS AVAILABLE - AT THE VERY LEAST.  Perhaps wait even longer or, better yet, use a different product. I'm actually embarrassed for Microsoft to put out such a horrific program.  It's REALLY bad. Really REALLY bad. Worse than Michael Jackson, who, as you all know, was BAD.

     

    DLovitt



    Tuesday, January 31, 2017 8:01 PM

  • DO NOT MIGRATE TO THIS SOLUTION UNTIL UR2 IS AVAILABLE - AT THE VERY LEAST.  Perhaps wait even longer or, better yet, use a different product. I'm actually embarrassed for Microsoft to put out such a horrific program.  It's REALLY bad. Really REALLY bad. Worse than Michael Jackson, who, as you all know, was BAD.

    I totaly agree with that , i migrated dpm 2012 R2 to 2016 (Even on Server 2012 with no modern Backup).
    After that i moved Some VM'S to a new Hyper-V Host and added them to a new Backup. That led to a complete crash of DPM. There was a Private Hotfix to get DPM working again, but the Backups of the VM'S still where lost.

    So i ended up reinstalling on Server 2016 with modern Storage wich now seems like a big Mistake.

    The Only Problem is that i need Support for Server 2016.

    BTW: i also didn't recieve an Reply from the Azure Guys.

    regards

    Stefan


    • Edited by steven_85 Wednesday, February 1, 2017 6:53 AM
    Wednesday, February 1, 2017 6:51 AM
  • DLovitt did you ever work with Symantec Backup Exec? DPM 2016 in its current state is bad, but Backup Exec was like that all the time. At least with DPM we can hope for patches that will actually fix these problems and that the product will eventually become as rock solid as earlier versions of DPM.

    Wednesday, February 1, 2017 8:56 AM
  • DLovitt did you ever work with Symantec Backup Exec? DPM 2016 in its current state is bad, but Backup Exec was like that all the time. At least with DPM we can hope for patches that will actually fix these problems and that the product will eventually become as rock solid as earlier versions of DPM.

    I don't think I see your point.  Yes, I used BE for years. Yes, it was a miserable product.  There is a reason I kicked it out of my environments 5 years ago.  But just because it was deficient doesn't give a pass to DPM, or the DPM product managers for putting out a product that is equally bad if not significantly worse.  If this thread was about a comparison between DPM 2016 and BE, that would be one thing.  This is strictly about significant problems with DPM 2016 and the absolute FACT that at this point it is barely functional. 

    In my opinion, I'm doing the community a service by pointing that out.  You seem to want to apologize for Microsoft, which is your right, but I can not and will not let them off the hook so easily.  As a matter of fact, I can safely say that the product Microsoft is currently putting out across the board, is a more difficult to administer and operate than at any point in my history of working with MS products.   DPM 2016 is just the most recent example.


    DLovitt





    Wednesday, February 1, 2017 4:09 PM
  • Microsoft has completely shifted its focus to "The Cloud". In my Enterprise we operate our own cloud as everyone always has. On premise Exchange 2016, On prem Dynamics CRM 2016, DPM 2016, Office 2016, Skype FB 2015, Sharepoint 2016 and administering each one of these is a night and day difference as compared to a few years back. All the real admin functions moved out of the GUI and into powershell commands (interesting since the fundamental premise of Windows was GUI) and where updates and bug fixes were regular, now the cloud gets everything new and On prem is left with the occasional scrap. On prem enterprise software is clearly in their rearview mirror and I wouldn't be surprised if some of these packages go the way of TMG.

    -Jason

    Wednesday, February 1, 2017 4:39 PM
  • We're also having the same problems, frequent ID 40002 and constant 100% CPU usage on our primary and secondary servers. Adding vCPUs does nothing to fix the issue.

    UR2 can't come quickly enough.

    Wednesday, February 1, 2017 5:18 PM
  • now the cloud gets everything new and On prem is left with the occasional scrap. On prem enterprise software is clearly in their rearview mirror and I wouldn't be surprised if some of these packages go the way of TMG.

    Totally agree, except that with MS, "cloud" isn't all that great either.  We're all in with Office 365, and there are incredible, like smack your head, deficiencies in their offering.  I'll give them that it's "OK", and not having to admin Exchange on prem is very nice, but there are genuine, daily issues we deal with while living in the cloud, they we never had on prem.

    Sorry I let this thread get off topic earlier.  Enough of the bashing, I just couldn't resist based on some of the responses earlier.


    DLovitt

    Wednesday, February 1, 2017 9:08 PM

  • In my opinion, I'm doing the community a service by pointing that out.  You seem to want to apologize for Microsoft, which is your right, but I can not and will not let them off the hook so easily.  As a matter of fact, I can safely say that the product Microsoft is currently putting out across the board, is a more difficult to administer and operate than at any point in my history of working with MS products.   DPM 2016 is just the most recent example.

    I am not defending Microsoft, just reminding people what a terrible product Backup Exec was and probably still is.

    That said, there is no excuse for the quality of some of the 2016 series products.  DPM 2016, Windows Server 2016 and VMM 2016, to name a few, all have serious issues. All of them feel rushed and there are so many bugs in there that i have stopped counting. Earlier releases like DPM 2012R2 and Windows Server 2012R2 also had their problems too, but nothing so severe and concentrated. Basic functionality looks like it has never been tested and that worries me.

    This week i had to reboot our DPM server every day because some backup couldn't finish or the server used up over 40 GBs of RAM. Hopefully UR2 will fix most of these issues. I thought i was done babysitting our backup system when we migrated away from Backup Exec :(

    Friday, February 3, 2017 9:52 AM
  • UPDATE:

    The Azure team emailed me and suggested I wait for the official release of the patch to be released in a couple weeks as opposed to the private fix.  I assume this will be with UR2. In our environment, its not a big deal and we can get by with rebooting when the error occurs.

    My comment on the over arching question of "Is DPM 2016 ready for prime-time?" Probably not, but "we" all know any Microsoft products major new release is going to be troublesome until brave (or mentally deficient) admins like us deploy it, find the problems and have them resolved. They don't call it bleeding edge for nothing. Not defending M$ here, just the facts and I too very much dislike their "Cloud First" strategy but any public company is driven by stock holders and the board of directors and thus revenue growth and earnings per share is always king. And the Cloud is raking in baskets of cash for Microsoft. Not to mention for some the Cloud just makes sense. I for one will stay on-prem until they pry my servers from my cold dead hands. -K

    Friday, February 3, 2017 2:24 PM
  • Unfortunately we are experiencing the same MBS/ReFS/Disk/FilterManager issues with DPM 2016.  It's barely working as a backup solution for us at the moment.

    I agree with others - do not upgrade to DPM 2016, wait and see if UR2 is any better.




    • Edited by DJL Monday, February 6, 2017 2:20 PM
    Monday, February 6, 2017 2:18 PM
  • had to reboot again today (7-02-2017) for the second time this week!!!

    still no fix to this issue.


    katlego

    Tuesday, February 7, 2017 9:08 AM
  • Hi,

    DPM 2016 update rollup 2 (UR2) that contains a fix for this issue should be released this week.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    • Marked as answer by Darius Lovitt Wednesday, February 8, 2017 11:22 PM
    • Unmarked as answer by Darius Lovitt Monday, February 27, 2017 11:27 PM
    Tuesday, February 7, 2017 4:11 PM
    Moderator
  • That's good news. It can't come soon enough! I'm off to reboot my server again...
    Tuesday, February 7, 2017 7:10 PM
  • Hi, 

    https://support.microsoft.com/en-us/help/3209593/update-rollup-2-for-system-center-2016-data-protection-manager

    • Marked as answer by Darius Lovitt Wednesday, February 8, 2017 11:22 PM
    • Unmarked as answer by Darius Lovitt Monday, February 27, 2017 11:27 PM
    Tuesday, February 7, 2017 9:58 PM
  • Hi, 

    https://support.microsoft.com/en-us/help/3209593/update-rollup-2-for-system-center-2016-data-protection-manager

    Breathless! This is such great news!  oh, wait....it's a Microsoft Patch. It won't install, of course. 

    All I get is: • Update Rollup 2 for System Center 2016 - Data Protection Manager (KB3209593) - Error 0x80070643.

    SMH- f'n pathetic.

    Has anyone had success installing this yet?


    DLovitt


    Tuesday, February 7, 2017 10:49 PM
  • I did. Installed and then restarted. Updated all the agents and waiting to see the results.

    -Jason

    Tuesday, February 7, 2017 11:03 PM
  • Jason- anything "special" i need to know about getting this thing installed?  I've tried 3 times now WITH reboot between.

    Oh yea, and for some added fun, try to go get the WindowsUpdate.log file to figure out WHY a windows update won't install, as in my case.  You know, the one in the %windir%.  You're in for a real treat- lol


    DLovitt


    Tuesday, February 7, 2017 11:15 PM
  • Hi, 

    https://support.microsoft.com/en-us/help/3209593/update-rollup-2-for-system-center-2016-data-protection-manager

    Breathless! This is such great news!  oh, wait....it's a Microsoft Patch. It won't install, of course. 

    All I get is: • Update Rollup 2 for System Center 2016 - Data Protection Manager (KB3209593) - Error 0x80070643.

    SMH- f'n pathetic.

    Has anyone had success installing this yet?


    DLovitt


    Argh! Same here. Same error. :( :(
    Tuesday, February 7, 2017 11:19 PM
  • Mother oF mothra!  This is painful.  Typical, but painful.

    The only thing SLIGHTLT constructive I can add is this:

    https://social.technet.microsoft.com/Forums/itmanagement/en-US/0112bfbb-b28f-4193-8802-091c13026491/consistently-get-windows-update-error-80070643-when-installing-dpm-update-rollup-9-kb3112306?forum=dataprotectionmanager

    There appears to be a switch to run to get a previous problematic DPM rollup to install.  I have the full package of 2016 UR2 downloaded already, but there are several files in the package, so i don't know which to run manually with this switch. 


    DLovitt

    Tuesday, February 7, 2017 11:26 PM
  • Mother oF mothra!  This is painful.  Typical, but painful.

    The only thing SLIGHTLT constructive I can add is this:

    https://social.technet.microsoft.com/Forums/itmanagement/en-US/0112bfbb-b28f-4193-8802-091c13026491/consistently-get-windows-update-error-80070643-when-installing-dpm-update-rollup-9-kb3112306?forum=dataprotectionmanager

    There appears to be a switch to run to get a previous problematic DPM rollup to install.  I have the full package of 2016 UR2 downloaded already, but there are several files in the package, so i don't know which to run manually with this switch. 


    DLovitt

    Though the update (installed through Windows Updates) said "failed", after a reboot, I do show the new DPM version: 5.0.322.0. So I'm left wondering if it's half-installed....
    Tuesday, February 7, 2017 11:58 PM
  • Hi,

    If you are having problem installing the update, the log file should contain information about the error.

    C:\Program Files\Microsoft System Center 2016\DPM\DPMLogs\DataProtectionManager-KB3209593.log

    InstallExecute. Return value 3.   <----- Search for this

    -OR-

    hr=0x800  <----- Search for this


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, February 8, 2017 12:01 AM
    Moderator
  • Downloaded UR2 outside of Windows Update and ran it again. This time it completed without error.
    Wednesday, February 8, 2017 12:12 AM
  • Sorry, I never attempted to use Windows update.  I just downloaded the package straight from the catalog and ran the installer.  Had no issues. 

    -Jason

    Wednesday, February 8, 2017 12:37 AM
  • I download direct from the Windows Update Catalog and installed manually - didn't have any problems with the install.

    Instantly we've seen synchronisation times drop from an average 20-30 mins to 2 mins on one of our 6TB files data sources!

    Wednesday, February 8, 2017 9:19 AM
  • Instantly we've seen synchronisation times drop from an average 20-30 mins to 2 mins on one of our 6TB files data sources!

    Intressting, did you Upgrade the DPM-Agent on the Server ? Did you have to reboot the Server After Agent installation?

    First i tried intalling UR2 via Windows Updates -> that failed.
    Manual installation worked after a reboot.

    regards.

    Wednesday, February 8, 2017 10:24 AM
  • I installed UR2 and rebooted the DPM server as requested.  Agents were updated on the protected servers, but they weren't rebooted.

    Sync times haven't improved across the board, only on data sources where i've been having problems with syncs, consistency checks etc 

    Sadly i'm still getting FilterManager "failed to attached..." and Disk "An error was detected on device \Device\Harddisk13\DR100 during a paging operation." errors.

    Wednesday, February 8, 2017 10:44 AM
  • Installation over Windows Update somehow doesn't update the admin console. Manually downloading and installing the update (the big file out of the four) works.
    • Marked as answer by Darius Lovitt Wednesday, February 8, 2017 11:23 PM
    • Unmarked as answer by Darius Lovitt Monday, February 27, 2017 11:27 PM
    Wednesday, February 8, 2017 2:16 PM
  • I cannot get the update to run on my Windows 10 workstation for the remote admin console. The installer crashes. Anyone tried that yet?

    -Jason

    Wednesday, February 8, 2017 2:20 PM
  • Works for me. The installer in the UR2 is only a patch though. You need to install the Admin Console from the installation media first.
    Wednesday, February 8, 2017 4:18 PM
  • I was finally able to install UR2- downloaded from the catalog and manually ran the largest file.

    So far, it's night and day.  Time will tell, but at this point I'm.... happy?  Is that the feeling?  It's been so long...

    Syncs take SIGNIFICANTLY less time.  I'm seeing green success, rather than a litany of red fails. Memory usage is WAY down.  Recovery point creation is actually working within a reasonable amount of time.

    The only question I have is: is the admin console standalone now, or is it still tied to Orchestrator or OpMan, or whatever? 


    DLovitt

    Wednesday, February 8, 2017 5:42 PM
  • We've always used the admin console that comes with DPM. 
    Thursday, February 9, 2017 10:25 AM
  • I gotta say, everything has been really good since this patch. Thank you to the Microsoft staff who got this patch produced. Memory, processor, backup performance and durability of the jobs is equal to what I was familiar with on 2012 R2.

    -Jason

    Friday, February 10, 2017 4:38 PM
  • I have also installed the patch manually because it failed via windows updates but so far i haven't had any issues...yet

    katlego

    Monday, February 13, 2017 12:00 PM
  • This was the first weekend I've come in to the office without a litany of DPM errors in my Inbox. This patch finally made DPM 2016 usable. Thanks.
    Monday, February 13, 2017 6:10 PM
  • DPM 2016 is still not usable for us.

    We get many Issues. Our Call is opend since the 15 December 2016. Now we are at Troubleshooting with two PGs.

    Tuesday, February 14, 2017 8:20 AM
  • We've also experienced a fix of all issues with UR2.  The server has been up for 4 days with no slow down and no excessive RAM utilization.  Prior to the patch we would be lucky to make it an entire day before DPM started seizing during syncs.
    Tuesday, February 14, 2017 9:01 PM
  • The problem with RAM remained ((
    Thursday, February 16, 2017 6:57 AM
  • The problem with RAM is apparently due to the ReFS and not DPM. They ReFS engineering team is aware of the issue and will be releasing a patch for that this quarter (Q1).

    -Jason

    Thursday, February 16, 2017 11:35 AM
  • I've unmarked all the "answers".  After the last round of MS updates, I'm mostly right back to where I started. At least twice now, my file server sync has stopped working, i have to reboot, and then the next sync takes DAYS to finish again.  Whatever the case is, this is NOT fixed.

    DLovitt

    Monday, February 27, 2017 11:30 PM
  • Hi.

    With UR2 the backups run without problems using my scripts. With the normal schedule in DPM (multiple backupjobs triggerd at one time), the errors still remain the same. Our MS Call is still open, but no solution until now.

    Currently I do some further testings on my own.

    I would like to know, how you provision your DPM-Storage. Do you use drive letters or mount points? I would really appreciate your feedback on this.

    Best regards

    Andreas


    • Edited by MarxA Tuesday, February 28, 2017 11:08 PM
    Tuesday, February 28, 2017 11:07 PM
  • Hello, 

    Few weeks after update to UR2 backups was working great. Now the original problem occurred. Any suggestions ?

    Wednesday, March 8, 2017 11:53 AM
  • If you have installed dpm 2016 ur2 and still have memory consumption problem, then contact AskAzureBackupTeam@microsoft.com and they will provide you the patch. I asked them and got one. Very grateful and can sleep better now, since backups are working much better.
    Thursday, March 9, 2017 6:19 PM
  • We're running DPM 2016 UR2 and have these filter errors constantly.  Does MS know that they have not fixed the problem?
    Thursday, March 16, 2017 2:34 PM
  • This months Windows Updates seem to have helped the memory consumption issue.

    Still loads of FilterManager and Disk warnings/errors though:

    Thursday, March 16, 2017 9:14 PM
  • Hi,

    Just for clarity - the high memory usage issue is fixed on the March Windows update - please install this update:

       March 14, 2017—KB4013429 (OS Build 14393.953)

    Then, specifically, you may also need to tune some ReFs parameters to get the best overall results.  See the companion article below for details.

       FIX: Heavy memory usage in ReFS on Windows Server 2016 and Windows 10  


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, March 20, 2017 3:56 PM
    Moderator
  • Yes, I too have tons of:

    Disk event ID 51 "An error was detected on device \Device\Harddisk4\DR5156 during a paging operation."

    FilterManager event ID 3 "Filter Manager failed to attach to volume '\Device\Harddisk2\DR5173'.  This volume will be unavailable for filtering until a reboot.  The final status was 0xC03A001C."

    Disk event ID 158 "Disk 8 has the same disk identifiers as one or more disks connected to the system. Go to Microsoft's support website (http://support.microsoft.com) and search for KB2983588 to resolve the issue."

    .NET Runtime event ID 1025:

    Application: wmiprvse.exe
    Framework Version: v4.0.30319
    Description: The application requested process termination through System.Environment.FailFast(string message).
    Message: Unexpected exception thrown from the provider:
     System.Exception: Element not found. (Exception from HRESULT: 0x80070490)
       at Windows.ApplicationModel.Package.get_Dependencies()
       at Microsoft.ConfigurationManager.WinRTProvider.Windows8Application.get_DependencyApplicationNames()
    Stack:
       at System.Environment.FailFast(System.String)
       at WmiNative.WbemProvider.WmiNative.IWbemServices.CreateInstanceEnumAsync(System.String, Int32, WmiNative.IWbemContext, WmiNative.IWbemObjectSink)

    But the disk and tape backups and restore seems to be functioning okay.  The errors just make me feel uneasy about it.

    Monday, April 24, 2017 6:09 PM
  • Hello all together,

    we have the same issue with our DPM Server.

    OS is Windows Server 2016 with all updates, DPM 2016 UR2.

    Server was running fine with Windows Server 2012 and DPM 2016 UR (without Modern Backup Storage). Since we have migrated to Windows Server 2016 (clean new installation an restore of the DPM database) and Modern Backup Storage (recreated all protection groups) we have a lot of Filter Manager Errors (Filter Manager failed to attach to volume...) and Disk Warnings (An error was detected on device \.... during a paging operation.

    This leads to missing volumes Errors in DPM and this leads to inconsistent recovery points...

    We have a smaller installation of a DPM 2016 with Modern Backup Storage on a second location which is running without any issues.

    For my understanding UR2 helped a lot regarding this problem, but the problem is still not fixed ;-(
    Right now we tried to reduce the amount of jobs at the same time by creating more protection groups and playing with offsets but still haven't found an acceptable solution.

    Does anybody have an idea how to adjust the timeouts when a lot of vhdx files are mounted at the same time?

    @Microsoft are there any improvements in planning?

    Thanks

    Michael

    Monday, June 12, 2017 12:07 PM
  • Hi Michael.

    How did you set up MBS? Do you use drive letters or mountpoints to provision the DPM volumes? What kind of storage do you have? Is it a virtual or physical Setup?

    Best regards

    Andreas

    Monday, June 12, 2017 3:36 PM
  • We had the same problems with our setup.  Horrible disk performance and errors mounting the VHDs all over the place.  I ended up having to completely blow away the DPM install.  I rebuilt the MBS as a mirror rather than parity and it's way better now.  Disk IO looks like what we'd expect and no more problems where VHDs won't mount without constant reboots.  MBS actually does deliver on the promise of more efficient disk usage, so we can still fit the backups on the mirror set.
    Monday, June 12, 2017 3:56 PM
  • Hi Andreas,

    we use only one mountpoint (C:\DPMStorage).
    We have a local attached SAS storage (4x HP MSA 60 with 12x 2TB Disks) running a RAID 60 volume on a HP P812 SAS Controller.
    DPM is running on a physical server (HP DL385 G7 with 2x 8 core CPUs and 64GB memory).

    On the second location we also have a physical DPM server (HP DL385 G7 with a single 8 core CPU and 16GB memory) and a local attached SAS storage (1x HP MSA 60 with 12x 2TB Disks) running a RAID 6 volume on a HP P812 SAS Controller. Here we also use only one mountpoint (C:\DPMStorage). This DPM works like a clockwork.

    Both machines are based on exactly the same os image have identically hardware setup (just a different sizing) and setup exactly the same way (clean installation an restore of DPMDB).

    Best Regards

    Michael

    Monday, June 12, 2017 4:05 PM
  • Hi Michael.

    The mountpoint C:\DPMStorage is your problem. DON'T use a mountpoint on C-drive. Ether use a different drive to provision the mountpoint or use a drive letter. We had exactly this isse, even after we applied UR2, which really did a lot things better for us and others on this thread. But still we had lots of VHD mount errors. We had a call with Microsoft ongoing. Finally we found out, that (for what reason ever) we had these errors only when we used a mountpoint on drive C. We switched to a moutpoint on drive D and a driverletter, both did the trick. Finally we decided to use driveletters. You can change this on the fly in the Windows diskmanagement. DPM will recognize the new mountpoint or driveletter. All backups will continue to run and restores work fine.

    So change the mountpoint and have a nice evening ;-)

    Best regards

    Andreas



    • Edited by MarxA Monday, June 12, 2017 4:19 PM
    • Proposed as answer by Michael EC Tuesday, June 13, 2017 7:54 AM
    Monday, June 12, 2017 4:16 PM
  • Hi Andreas,

    I will try this directly, if it works, you will be my hero.

    I will just try the change from C: to D: on a DPM in the test environment and than directly on the productive DPM.

    Keep you informed.

    Thank You, have a nice evening, too ;-)

    Best Regards

    Michael

    Monday, June 12, 2017 4:50 PM
  • Hi Andreas,

    since yesterday evening we didn't saw any Volume Missing errors ;-)))
    In addition we could successfully run a consisty check of a data source which was always failling before changing from the mount point on C:\ drive to a drive letter (D:\).

    Really appreciate your help!

    @E-Rock-001 Thank You also for your information, I guess if the server would have more IOPS maybe we would just not run into the issue like on our smaller DPM on the second location. Anyway I now switched both DPMs to drive letters to make sure I won't run in trouble.
    Are you using mount points on the system drive?

    Best Regards

    Michael


    • Edited by Michael EC Tuesday, June 13, 2017 8:52 AM
    Tuesday, June 13, 2017 8:10 AM
  • Hi Michael.

    I'm glad I could help!

    Best regards

    Andreas

    Tuesday, June 13, 2017 7:14 PM
  • Hi Andreas,

    THANKS!!!!!!!!!!!!!!!!!!!!

    /Jan


    Wednesday, June 21, 2017 1:42 PM
  • We are running into this same issue even after Update Rollup 2 is applied (DPM 2016). A couple jobs a day will fail with "The specified storage is unusable as it is a system volume, has existing data, is in an unhealthy state or formatted incorrectly". Typically resume backups will work. Other times we will get a missing volume error, at which point a "rescan" operation on the disk storage will resolve the error. Event viewer is showing two different errors, "Filter Manager failed to attach to volume ______" and "An error was detected on device \Device\Harddisk\_____ during a paging operation". Storage is a 14 drive RAID 6 set (one drive as hot spare) presented to DPM as a (roughly) 80TB ReFS volume, drive letter D. Everything looks good/healthy from the RAID software (this is a brand new server).

    Is there anything else to look for, or should I hope for an eventual update rollup 3?

    Wednesday, June 21, 2017 7:47 PM
  • Hi Andreas,

    Could you please provide me more details how to change the default moutpoint location from drive C to another drive?
    I've tried using Windows disk management but every time a recovery point is initiated I can see two mount points for the virtual disk - one for the new location and one pointing to "c:\program files..."

    Regards,
    Petya

    Wednesday, August 30, 2017 10:12 AM
  • Hi Petya.

    What you see works as designed. The only mount point(s) you can change from C to another drive are the ones for the ReFS-Volumes you provision for MBS. The mount points you see under the DPM installation folder (e.g. "C:\Program files ...") cannot be changed unless you would install DPM on another drive. These mount point are for legacy compatibilities and for example when you use Azure backup for Long term backups. For all other read/write operations DPM does not use these mount points and accesses the VHDXs directly by GUIDs.

    So when you changed the mount points for the MBS volumes to another drive then C, you're done. I did not try to install DPM to another drive than C. Neither me nor the Microsoft engineers think, that this would help.

    Did you get better results after changing the mount points? On my systems, that really helped a lot. Not all problems have gone, but it is indeed a lot better.

    Best regards
    Andreas

    Tuesday, September 19, 2017 7:39 AM
  • Hi,

    Please see new forum announcement about this - It had proven to solve the majority of these errors.

    DPM error The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, March 27, 2019 7:42 PM
    Moderator
  • Hi,

    Please see new forum announcement about this - It had proven to solve the majority of these errors.

    DPM error The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)

    Having the same problem on a brand new DPM 2019 install.  40002 every hour.  I switched to modern storage versus our previous DPM 2016 install.  Since a reboot fixes this for a while, it'll be interesting to see if these proposed changes actually fix it.

    EDIT:
    A week later and the problem went away.  Hopefully it stays away.

    • Edited by JEmlay Monday, April 15, 2019 4:22 PM
    Wednesday, April 10, 2019 7:35 PM
  • We've implemented to WMI changes and registry keys and are still seeing VHD mounting issues.The other thread mentions changing the "mount point" for MBS away from the system drive.

    The mountpoint C:\DPMStorage is your problem. DON'T use a mountpoint on C-drive.

    Ether use a different drive to provision the mountpoint or use a drive letter.

    We have a drive F: exposed for MBS.

    Or is this something else they are talking about?

    Sunday, April 14, 2019 6:37 AM
  • I feel your pain.  I have just spun up two SCDPM 2019 (different locations) boxes and now have this same issue on both.

    The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID: 40002)

    One would think MS would have solved this by now since it has been an issue for several years.  My 2012R2 boxes were never an issue.  They just worked but when I upgraded my infrastructure to Server 2019 I had no choice but to go with SCDPM 2019.  I am using a QNAP SAN mounted as an E: drive and formatted as ReFS.  Rebooting the server does not help.

    MS - Are you aware of this issue?  I need to either get it resolved or move to a different backup solution.


    Wednesday, May 1, 2019 2:29 PM
  • I have exactly the same problem, dpm 2019 is installed out of the box, and I received the same error as in dpm 2016. The problem with the refs was not solved. All backups slow down a lot. And this problem has been going on for a couple of years.

    • Edited by Aleks__ Tuesday, May 7, 2019 11:18 AM
    Tuesday, May 7, 2019 8:06 AM
  • Have you experienced a recurrence of the issue since you made the change?

    I made the WMI change a couple of weeks ago and initially thought it resolved, but it turned out to be only a temporary respite. Since then I've been continually tweaking the parameters upwards but its getting me nowhere - again this morning I have several instances of the VHD mount error. Frustrating.

    Steve

    Thursday, May 16, 2019 10:44 AM
  • Hi Stephen, 

    Can you tell me the number of protected hosts and the MemoryPerHost value you are using? 

    I am currently protecting +/- 225 Hyper-V VMs, roughly 40TB total on a DPM 2019  server with 40GB RAM at te moment. We are running Server 2019, and experiencing the same issues as described in this topic. 


    Bas

    Thursday, May 16, 2019 8:28 PM
  • Have you all seen the description of the DPM 1807 (2016-and-a-bit) update? 

    Update Rollup 7 for System Center 2016 Data Protection Manager

    This update fixes the following issues:

    • Backup operation fails with ‘40002 – The VHD containing the replica or one of its snapshots could not be mounted or unmounted’ error.
    • Data Protection Manager Recovery agent service crashes during Hyper-V Virtual Machine backup operation.
    • Data Protection Manager service intermittently crashes during backup operations.  
    • When a Sharepoint farm with more than 100 databases is protected to Azure, recovery fails during re-catalog operation.
    • Recovery of Hyper-V VMs as files in ‘Disk to Tape (D-T) backup scenario fails.
    • Storage requirement for a Sharepoint farm on a secondary DPM server is not calculated accurately when DPM server uses Modern Backup Storage.
    • Recovery of a Sharepoint farm from Azure fails.  


    Thursday, May 16, 2019 8:34 PM
  • Hi Bas,

    I've got 42 SQL servers for a total of 490 DBs protected consuming roughly 58TB of storage currently.

    Its running on a physical Server 2019 box with 20 cores and 32GB.

    Current memory settings are as follows:

    High threshold: 256000000

    Memory all hosts: 4294967296

    Memory per host: 2147483648

    Cheers

    Steve

    Thursday, May 16, 2019 8:56 PM