Data Protection Manager - General announcement
-
Link
We have seen DPM 2016 or later servers using Modern Backup Storage (MBS) experience the below error.
The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)
This can be caused by a real failure to mount / dismount a replica or snapshot .vhdx file – or most likely the mount / dismount is working but WMI call is failing due to WMI out of memory error.
NOTE: We have seen some customers experience this problem when using volume mountpoints for modern backup storage (MBS) so it suggested to assign a drive letter to the MBS volume(s) , then refresh the DPM Storage pool so it picks up the drive letter, then remove the mount point.
To mitigate the WMI memory problem you can implement the below changes.
1) Add these registry settings. Copy / paste into notepad and save as WMI.REG (no .txt extension) - then right-click the wmi.reg and click merge.
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Wbem\CIMOM]
"High Threshold On Client Objects (B)"="60000000"
"High Threshold On Events (B)"="60000000"2) Follow the below blog and change memory from 512mb to 1GB.
Memory and Handle Quotas in the WMI Provider Service
https://techcommunity.microsoft.com/t5/Ask-The-Performance-Team/Memory-and-Handle-Quotas-in-the-WMI-Provider-Service/ba-p/3733191.Start “wbemtest” as administrator.
2.Connect to the “root” namespace (not “root\default”, just “root”)
3.Click Open Instance. Specify “__ProviderHostQuotaConfiguration=@”
4.Select Local Only for easier readability. You will see the threshold values.
5.Increase the MemoryPerHost value to something greater. (For example, double it to 1GB 1073741824)
6.Save Property
7.Save Object
8.Click Exit.3) Reboot the DPM Server.
-
Link
On March 7, 2019, we shared that System Center 2019 would be coming soon. As of March 14, 2019, we are pleased to let you know that System Center 2019 is generally available. Customers with a valid license of System Center 2019 can download media from the Volume Licensing Service Center (VLSC). We will also have the System Center 2019 evaluation available on the Microsoft Evaluation Center.
Faster backups with Data Protection Manager 2019
Data Protection Manager (DPM) 2019 will provide backups optimized in time (faster) and space (consumes less storage).
- DPM improves performance of your backups with a 75 percent increase in speed and enables monitoring experience for key backup parameters via Log Analytics.
- DPM further supports backup of VMware VMs to tape. In addition to Windows Server 2019, DPM now provides backups for new workloads such as SharePoint 2019 and Exchange 2019.
-
Link
ISSUE: BMR backups fail
Issue occurs on Data Protection Manager\Microsoft Azure Backup Server 2016 using Modern Backup Storage. This occurs because windows backup server writers directly to the DPM\MABS share during backup and there could be IO latency greater than the standard SMB timeout of 45-60 seconds. There are also some general items that should be configured for BMR backups please see Pre-requisites below.
Pre-Requisites:
Latest UR patch for Data Protection Manager or Microsoft Azure Backup Server
Ensure Replica volume is sized correctly – https://blogs.technet.microsoft.com/dpm/2017/04/14/how-to-increase-dpm-2016-replica-when-using-modern-backup-storage-mbs/
Ensure July CU update or above is applied for REFS - https://support.microsoft.com/en-us/help/4345418/windows-10-update-kb4345418
Symptoms:
On DPM server:
DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code:
0x6B48C3A0). (ID 30229 Details: Internal error code: 0x80990ED0)
OR
You may also encounter an error “The storage involving the current operation could not be read from or written to. (ID 40003)”
Log Name: DPM Alerts
Source: DPM-EM
Date: 7/7/2017 11:11:36 AM
Event ID: 3106
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: <servername>
Description:
The replica of Non VSS Datasource Writer on <servername> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. (ID: 3106)
The storage involving the current operation could not be read from or written to. (ID: 40003)
On Protected server:
Log Name: Microsoft-Windows-Backup
Source: Microsoft-Windows-Backup
Date: 5/8/2018 10:43:42 AM
Event ID: 5
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: <Servername>
Description:
The backup operation that started at '2018-05-08T15:17:32.312386000Z' has failed with following error code '0x8078015B' (Windows Backup encountered an error when accessing the remote shared folder. Please retry the operation after making sure that the remote shared folder is available and accessible.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
And
Log Name: Microsoft-Windows-SmbClient/Connectivity
Source: Microsoft-Windows-SMBClient
Date: 5/21/2018 4:23:49 PM
Event ID: 30809
Task Category: None
Level: Error
Keywords: (64)
User: N/A
Computer: <servername>
Description:
A request timed out because there was no response from the server.
And
Server name: <Servername>
Session ID:0x17801D0000031
Tree ID:0x1
Message ID:0x4000015
Command: Flush
Guidance:
The server is responding over TCP but not over SMB. Ensure the Server service is running and responsive, and the disks do not have high per-IO latency, which makes the disks appear unresponsive to SMB. Also, ensure the server is responsive overall and not paused; for instance, make sure you can log on to it.
Log Name: Microsoft-Windows-SMBServer/Operational
Source: Microsoft-Windows-SMBServer
Date: 5/21/2018 4:25:37 PM
Event ID: 1020
Task Category: (1020)
Level: Warning
Keywords: (8)
User: SYSTEM
Computer: <ServerName>
Description:
File system operation has taken longer than expected.
Client Name: <IP>
Client Address: <IP:Port>
User Name: <Username>
Session ID: 0x17801D0000031
Share Name: <sharename>
File Name: WINDOWSIMAGEBACKUP\<Servername>\<Backupdate>\<GUID>.VHDX
Command: 7
Duration (in milliseconds): 353151
Warning Threshold (in milliseconds): 120000
Guidance:
The underlying file system has taken too long to respond to an operation. This typically indicates a problem with the storage and not SMB
Cause:
SMB timeout is too short for IO latency for BMR on REFS
Resolution\Workaround:
Option 1:
Set registry and reboot protected server
- Add the registry key below on the protected server
reg add HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters /v SessTimeout /t REG_DWORD /d 1200
Option 2: If you cannot reboot server (Note: Ensure you check the list of dependencies for the Lanmanworkstation service as those services will need to be started again too)
- Add the registry key below on the protected server
reg add HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters /v SessTimeout /t REG_DWORD /d 1200
Note: The 1200 second setting is a 20 minute timeout. This is not a rigid number but has been tested with success. Depending on latency you can adjust as needed. To get suspected latency please review REFS event logs on DPM\MABS server.
- Stop/Start Lanmanworkstation service
Note: There are some other services that depend on Lanmanworkstation, just start those back.
-
Link
There is one optimization change you can make in DPM 2016 using Modern Backup Storage that “may” help reduce the backup times. The issue is that DPM tries to get the size of the recovery point by asking ReFs to count the blocks associated with that recovery point so it can update the DPMUI “storage used” in the UI.
To eliminate that additional calculation that can take a long time and allow the next backup to start you can disable that calculation using the below DPM powershell command.
Manage-DPMDSStorageSizeUpdate.ps1 -ManageStorageInfo StopSizeAutoUpdate
As a result, the “storage consumed” for each data source will not be displayed after a new backup is completed.
The above is covered in the release notes: https://docs.microsoft.com/en-us/system-center/dpm/dpm-release-notes
Section titled: Recovery Points not being pruned, leading to an accumulation of Recovery Points.
If you need to know the amount used for a data source you can use the associated UpdateSizeForDS command.
-
Link
There may be circumstances where the replica volume for a protected data source is under allocated or a very large increase of protected data results in synchronization or recovery point job failures due to inadequate space on the replica.
Type: Recovery point
Status: Failed
Description: DPM is out of disk space for the replica. (ID 58 Details: There is not enough space on the disk (0x80070070))As in previous versions, DPM 2016 has replica auto-grow feature for modern backup storage (MBS) to help increase the replica sizes as the data sources grows. Under normal use conditions this feature works perfectly fine. If a job fails due to out of disk space, it will automatically grow the replica volume and schedule a new synchronization or recovery point job to run one hour in the future. However, if the amount of new data is much larger than the size of the replica after auto-grow, the jobs will continue to fail even as the auto-grow continues to increase the replica size. After two failures, auto-rerun is no longer triggered by default. In DPM 2012 R2 you had the ability to manually grow the replica volume as large as you wanted by using the “modify disk allocation” wizard, however in the DPM 2016 you do not have the ability to manually grow the replica volume using the same wizard. This can cause delays in getting good backups again to meet your SLA. If you know the new size of the protected data or want to grow the replica volume for future data growth in order to prevent future backup failures, you can grow the replica manually.
The below BLOG contains a DPM PowerShell script to make it easy to manually increase a replica volume to the new desired size.
How to increase DPM 2016 replica when using Modern Backup Storage (MBS)
https://techcommunity.microsoft.com/t5/System-Center-Blog/How-to-increase-DPM-2016-replica-when-using-Modern-Backup/ba-p/351818 -
Link
When using DPM 2016 to protect SystemState / BMR workloads the storage used for those backups will be higher when using DPM Modern Backup Storage (MBS) as compared to using legacy Logical Disk Manager (LDM) based storage used in DPM 2012 R2.
When DPM initiates a SystemState or BMR backup, Windows Server Backup is used and overwrites the existing WindowsImagebackup folder with the new backup data. For SystemState that folder is on the protected server and later transferred to the DPM Server replica. For BMR, Windows Server Backup overwrites the folder on the Replica volume directly.
When the replica is located on a LDM based volume, Volume Shadow Copy (VSS) snapshots are used for recovery points. When the files are overwritten on the replica volume, VSS performs a read for the original file blocks – compares them to what is about to be written - then either copies the original blocks to the shadow copy (recovery point volume) if different or just lets the write thru if the blocks are identical. This is called Copy On Write (COW). Between backups a lot of the blocks are the same between the files so the shadow copies were relatively small.
Modern Backup Storage (MBS) uses ReFs clones for recovery points. There is no copy on write operation like there are when using VSS, all writes are performed but instead of overwriting the original files the new blocks are written to different locations on the MBS volume. This is called allocate on write, so even blocks that are the same will use different free space on the Refs volume. That results in more storage consumption as each backup will consume the amount of space as a complete SystemState or BMR backup. IE: If the initial replica size shows 24GB used, then all subsequent backups will come very close to consuming the same amount of space. If the backups are taken frequently or have a long retention period the storage usage can be significant.
To mitigate this behavior I would suggest either take fewer BMR / SystemState recovery points, use a reduced retention period, or just plan on needing more storage for SystemState / BMR backups.
-
Link
System Center Data Protection Manager 2012 R2 UR11 added support for VMware host level backup. If you attempt to perform an Item Level Recovery (ILR) from the latest recovery point for a Protected Windows VM, you receive an ID: 958 popup error message with the below details.
IMPORTANT NOTE: This problem is not present in Microsoft Azure Backup Server (MABS) update rollup 1 (UR1) that also added support for VMWare host level backups.
WORKAROUND(S)
===========1) Select an earlier recovery point if the file(s) (version) you want to recover was included in an earlier recovery point.
2) Restore the entire VM to an alternate location and copy the desired file(s) out of that restored VM.
3) Create a new AD-HOC recovery point manually. That will enable ILR to work for all recovery points including the latest one you made manually.UPDATE: This issue is now fixed in DPM 2012 R2 UR13. However, you need to consider the following for the fix to work.
1. For Protection Groups created after applying UR13, Item Level Recovery will work fine.
2. For older Protection Groups, you need to modify the PGs by adding/removing a small VM or making some configuration changes to the PG for the fix to work. - Link
-
Link
SC DPM 2016 brings in features that delivers improvements in multiple key areas of backup efficiency, performance, flexibility and security. With Modern Backup Storage (MBS), DPM 2016 is turning around the way backups are stored by leveraging modern technologies like ReFS Block Cloning, ReFS Allocate On Write and VHDX. This to 3X faster backups and 50% reduction in storage consumption thus reducing overall backup TCO.
https://blogs.technet.microsoft.com/dpm/2016/10/19/announcing-sc-dpm-2016-with-modern-backup-storage/ -
Link
DPM 2016 comes with Modern Backup Storage which delivers 50% storage savings, 3x faster backups, and more efficient storage with Workload-Aware storage. Modern Backup Storage can be used by setting up DPM 2016 on Windows Server 2016.
Add Storage to DPM 2016 -
Link
Very happy to report the release SC DPM 2012 R2 UR11 that adds protection for VMware host level backup of VMs. Here are major features of VMware VM Backup. Here are the few highlights of VMware VM backup.
- Agentless backup
- Backup at vCenter or ESXi level
- Backup VMware VM folders
- Ability to do ILR
- Protect large data center using DPM Scale Out
- Backup to Disk/Cloud
Here are the blogs and articles that are published.
BLOG - Implementing Microsoft DPM host level protection of VMware VMs
BLOG-Announcing VMware VM Backup using System Center Data Protection Manager
-
Link
We have made a bunch of improvements in this update, related to reboot requirements, Enterprise file server backup, online backups and more.
-
Link
After you install DPM 2012 R2 UR7 update (KB3065246), DPM cannot delete expired recovery points. This will lead to extra recovery point volume usage.
Please download and install the fix for this issue.
Fix for a known issue with Update Rollup 7 for System Center 2012 R2 Data Protection Manager
MORE INFORMATION
==============DPM 2012 R2 UR7 introduced a new switch to the powershell cmdlet Get-RecoveryPoint, it now requires a -Datasource switch as seen below:
Get-RecoveryPoint -Datasource $ds
If you have any custom powershell scripts that use that cmdlet, please modify them to use the new -Datasource switch.
-
Link
Below are some handy links for the DPM Console troubleshooter which can help you resolve console crashes and three DPMDB maintenance blogs.
Guided Walkthrough for Troubleshooting Data Protection Manager console issues
DPMDB Maintenance Part 1: Database consistency check and your DPMDB
DPMDB Maintenance Part 2: Identifying and dealing with fragmentation in your DPMDB
DPMDB Maintenance Part 3: Dealing with a large DPMDB -
Link
Hello - Please find the following BLOG on the new streamlined method of registering a DPM 2012 SP1 or later DPM server to backup to Azure. There is no longer a requirement to upload a certificate when registering a DPM server for online backups.
Below are two Blogs on the subject.
How to register a DPM 2012 SP1 or later server to backup to Microsoft Azure
Simplified workflow to configure backup to Azure from DPM -
Link
Please find the following whitepaper detailing how to dedup DPM storage pool to help reduce overall disk storage requirements for disk based backups. This solution requires Windows 2012 R2 and System Center Data Protection Manager 2012 R2
-
Link
For DPM Console crash issues, please review this troubleshooter before posting a new question.
Guided Walkthrough for Troubleshooting Data Protection Manager console issues
-
Link
updated 6-5-13
Hello Everyone,
System Center DPM 2012 is not supported running on Windows Server 2012. We have had reports of DPM 2012 console crashes when navigating to the recovery tab if DPM 2012 is installed on Windows server 2012. This issue is fixed in System Center 2012 SP1 which supports being installed on Windows 2012 operating system.
If you are already in this condition, you have two options:
A) You can upgrade to System Center 2012 Sp1.
B) You can backup the DPMDB, rebuild the DPM server using Windows 2008 R2 install DPM 2012, then restore your current database. The process is explained in the below technet article.
Recovering DPM servers
http://technet.microsoft.com/en-us/library/jj244606.aspxWe also do not support protecting Windows Server 2012 workloads using SCS DPM 2012, you must upgrade to DPM 2012 Sp1. Please do not expect any support for DPM 2012 running on Windows Server 2012, or issues while protecting data sources in Windows Server 2012.
-
0 VotesThe agent operation failed because DPM was unable to identify the computer account Error ID 318
Hi, Please help regarding the following issue. Once in a week or two we get error below.. The DPM protection agent on ServerName.Domain.Local ...Unanswered | 0 Replies | 789 Views | Created by Rajeev_Bhandari - Tuesday, December 1, 2020 11:22 AM
-
0 Votes
backup issue
Production Servers -> Agent Status -> OK , but in I going to Recovery I do see last couple days my backups for all my jobs I try manual Recovery point it's works but ...Answered | 12 Replies | 688 Views | Created by Mooner09 - Thursday, November 26, 2020 2:07 PM | Last reply by Mooner09 - Monday, November 30, 2020 2:19 PM -
0 Votes
System Center Configuration Manager 2019 Deployments Issue
Dear Teams, I want deploy SCCM on windows server 2019 environment. Please advice and share Microsoft download link with me, SCCM version and SQL Server ...Unanswered | 1 Replies | 518 Views | Created by Parvez BD - Sunday, November 29, 2020 3:46 PM | Last reply by Leon Laude - Monday, November 30, 2020 6:53 AM -
0 Votes
DPM 2019 Installtin error
I am installting DPM 2019 but it is throwing error: My environment : Remote SQL server installed 2017 SQL ...Answered | 4 Replies | 2485 Views | Created by purushottam151 - Tuesday, September 3, 2019 3:36 PM | Last reply by M.faqahat - Monday, November 30, 2020 6:31 AM -
0 Votes
How to best "Keep an Offline Backup" when a ransomware attack occurs.
How to best "Keep an Offline Backup" when a ransomware attack occurs. We use SCDPM att our backup tool, and how can we protect or DPM servers from a ransomware ...Unanswered | 1 Replies | 792 Views | Created by SaiTech - Thursday, November 26, 2020 9:35 PM | Last reply by Leon Laude - Thursday, November 26, 2020 10:08 PM -
1 Votes
DPM 2019 Update Rollup 2 - SQL errors during update
Hi all, We recently updated our dpm server from 2016 to version 2019. Now we wanted to patch the update rollup 2 but are getting the following ...Proposed | 2 Replies | 1263 Views | Created by Luke46 - Wednesday, November 11, 2020 10:18 AM | Last reply by AntoineCardon - Thursday, November 26, 2020 1:25 PM -
29 Votes
DPM 2016 MBS Performance downward spiral
Hi Guys! I´m so pissed, really i´m at a point were i would like to just uninstall and go home. Why? Since we upgrade to DPM 2016 and switched ...Unanswered | 230 Replies | 36442 Views | Created by Intirius - Thursday, January 25, 2018 12:42 PM | Last reply by TJ Cornish - Sunday, November 22, 2020 7:47 PM -
0 Votes
Finding the Mount Point Path
Management>Online shows Online subscriptions "Status: Active". The Microsoft Azure Recovery Services Agent is running and updated to version 2.0.9197.0. I only have ... -
12 Votes
DPM Error ID: 36 - DPM Cannot access the path, becasue part of the path has been deleted or renamed
I have 6 dpm 2012 sp1 servers backing up a 2012 sp1 8 node hyper-v cluster. all backups ran perfect all week. We do not run backups on saturday night. Backups ...Answered | 17 Replies | 14907 Views | Created by Anncex - Monday, February 24, 2014 9:47 PM | Last reply by ecureuil38 - Thursday, November 19, 2020 7:09 PM -
0 Votes
SCCM - Third Party Updates (Open Manage Agent) Drivers
Hi all! I've recently enabled third party software updates (enabled client settings, set IIS service account etc) and have the Dell metafiles syncing. I ...Proposed | 1 Replies | 534 Views | Created by sircoolcat89 - Thursday, November 19, 2020 11:22 AM | Last reply by Leon Laude - Thursday, November 19, 2020 12:45 PM -
0 Votes
Temporarily exclude individual users from custom ATP policy
No idea if I'm in the right forum for this, but we have created a custom ATP policy for preventing the passing credit card information. However, once a year, our financial team needs to share ...Proposed | 1 Replies | 532 Views | Created by GS597 - Sunday, November 15, 2020 5:13 PM | Last reply by Leon Laude - Sunday, November 15, 2020 6:09 PM -
2 Votes
Backup Office 365
Hi there, Can someone answer me if I can backup my office 365 data with Data Protecion Manager? And if yes, point me to where the documentation is? Tried ...Unanswered | 18 Replies | 4102 Views | Created by evolvtyon - Tuesday, February 21, 2017 9:42 PM | Last reply by Rajat2816 - Thursday, November 12, 2020 11:12 AM -
0 Votes
INTUNE installation/Add Unable to Login
We are having an issue deploying new equipment to our AzureDomain via Intune. We get locked out of the new hardware 50%+ of the time and have to begin the process all over again. By ...Answered | 2 Replies | 1011 Views | Created by 1-SE - Monday, November 9, 2020 6:27 PM | Last reply by 1-SE - Monday, November 9, 2020 8:06 PM -
0 Votes
System Center Configuration Manager
I received SQL Server TCP Port error. The suggestion to check SQL Server Network Configuration TCP/IP properties for TCP Dynamic Port ...Proposed | 1 Replies | 544 Views | Created by Tadejobi - Thursday, November 5, 2020 4:22 PM | Last reply by Leon Laude - Thursday, November 5, 2020 5:54 PM -
0 Votes
DPM cannot connect to drive after partition changed the GUID
Hi All, The C: drive on a windows 10 client was running low on space and after all other attempts to free up space, I used a 3rd party (EASEUS) partition management ...Unanswered | 3 Replies | 838 Views | Created by MandoesaM - Monday, October 26, 2020 6:27 AM | Last reply by Leon Laude - Thursday, November 5, 2020 2:10 PM -
0 Votes
sccm 2006 - Application deployment doesnt copy a batch file to the client cache - no idea why.
Hi, Thanks for reading: I am in a sccm 2006 environment - deploying some software to a Windows 7 vm. The app has many files, all of which copy to the ...Proposed | 1 Replies | 662 Views | Created by Kim.Cross - Tuesday, November 3, 2020 10:54 AM | Last reply by Leon Laude - Wednesday, November 4, 2020 3:50 PM -
1 Votes
SCOM 2016 to 2019 migration and what we should test after migration (Test cases) ?
We have planned migrate SCOM server from 2016 (1807) to SCOM 2019, so I would like to ask test cases that we should performed after migration, if anyone has please please share.Answered | 2 Replies | 726 Views | Created by Kosala Randika - Tuesday, November 3, 2020 7:54 AM | Last reply by Kosala Randika - Tuesday, November 3, 2020 9:20 AM -
0 Votes
Not able to shutdown or restart & CPU utilization is 100 %
Every time when I trying to shutdown or restart PC it says like some program is preventing you from shutdown or restart,when I choose shutdown/restart anyway it keeps loading as shuting down ...Proposed | 1 Replies | 547 Views | Created by antonylineesh - Sunday, October 25, 2020 3:12 PM | Last reply by Leon Laude - Sunday, October 25, 2020 7:20 PM -
0 Votes
DPM 2019 RU2 Fails to install
Morning, On two different DPM 2019 RU1 servers RU2 fails to install. "There were problems installing some updates..." UR2 SC2019 DPIN ...Proposed | 34 Replies | 3773 Views | Created by Lord Melch - Thursday, August 13, 2020 7:17 AM | Last reply by Lord Melch - Wednesday, October 21, 2020 12:36 PM -
6 Votes
DPM 2012 S1 UR8 - event 999 service crash
Hello, I've tried to add a new computer to a protection group and the console crashed. Since the DPM Service crash and I cannot open the managemnet ...Answered | 8 Replies | 3404 Views | Created by Manu_75009 - Wednesday, December 17, 2014 1:24 PM | Last reply by Renke Holert - Wednesday, October 21, 2020 11:36 AM
Data Protection Manager - General announcement
-
Link
We have seen DPM 2016 or later servers using Modern Backup Storage (MBS) experience the below error.
The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)
This can be caused by a real failure to mount / dismount a replica or snapshot .vhdx file – or most likely the mount / dismount is working but WMI call is failing due to WMI out of memory error.
NOTE: We have seen some customers experience this problem when using volume mountpoints for modern backup storage (MBS) so it suggested to assign a drive letter to the MBS volume(s) , then refresh the DPM Storage pool so it picks up the drive letter, then remove the mount point.
To mitigate the WMI memory problem you can implement the below changes.
1) Add these registry settings. Copy / paste into notepad and save as WMI.REG (no .txt extension) - then right-click the wmi.reg and click merge.
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Wbem\CIMOM]
"High Threshold On Client Objects (B)"="60000000"
"High Threshold On Events (B)"="60000000"2) Follow the below blog and change memory from 512mb to 1GB.
Memory and Handle Quotas in the WMI Provider Service
https://techcommunity.microsoft.com/t5/Ask-The-Performance-Team/Memory-and-Handle-Quotas-in-the-WMI-Provider-Service/ba-p/3733191.Start “wbemtest” as administrator.
2.Connect to the “root” namespace (not “root\default”, just “root”)
3.Click Open Instance. Specify “__ProviderHostQuotaConfiguration=@”
4.Select Local Only for easier readability. You will see the threshold values.
5.Increase the MemoryPerHost value to something greater. (For example, double it to 1GB 1073741824)
6.Save Property
7.Save Object
8.Click Exit.3) Reboot the DPM Server.
-
Link
On March 7, 2019, we shared that System Center 2019 would be coming soon. As of March 14, 2019, we are pleased to let you know that System Center 2019 is generally available. Customers with a valid license of System Center 2019 can download media from the Volume Licensing Service Center (VLSC). We will also have the System Center 2019 evaluation available on the Microsoft Evaluation Center.
Faster backups with Data Protection Manager 2019
Data Protection Manager (DPM) 2019 will provide backups optimized in time (faster) and space (consumes less storage).
- DPM improves performance of your backups with a 75 percent increase in speed and enables monitoring experience for key backup parameters via Log Analytics.
- DPM further supports backup of VMware VMs to tape. In addition to Windows Server 2019, DPM now provides backups for new workloads such as SharePoint 2019 and Exchange 2019.
-
Link
ISSUE: BMR backups fail
Issue occurs on Data Protection Manager\Microsoft Azure Backup Server 2016 using Modern Backup Storage. This occurs because windows backup server writers directly to the DPM\MABS share during backup and there could be IO latency greater than the standard SMB timeout of 45-60 seconds. There are also some general items that should be configured for BMR backups please see Pre-requisites below.
Pre-Requisites:
Latest UR patch for Data Protection Manager or Microsoft Azure Backup Server
Ensure Replica volume is sized correctly – https://blogs.technet.microsoft.com/dpm/2017/04/14/how-to-increase-dpm-2016-replica-when-using-modern-backup-storage-mbs/
Ensure July CU update or above is applied for REFS - https://support.microsoft.com/en-us/help/4345418/windows-10-update-kb4345418
Symptoms:
On DPM server:
DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code:
0x6B48C3A0). (ID 30229 Details: Internal error code: 0x80990ED0)
OR
You may also encounter an error “The storage involving the current operation could not be read from or written to. (ID 40003)”
Log Name: DPM Alerts
Source: DPM-EM
Date: 7/7/2017 11:11:36 AM
Event ID: 3106
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: <servername>
Description:
The replica of Non VSS Datasource Writer on <servername> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. (ID: 3106)
The storage involving the current operation could not be read from or written to. (ID: 40003)
On Protected server:
Log Name: Microsoft-Windows-Backup
Source: Microsoft-Windows-Backup
Date: 5/8/2018 10:43:42 AM
Event ID: 5
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: <Servername>
Description:
The backup operation that started at '2018-05-08T15:17:32.312386000Z' has failed with following error code '0x8078015B' (Windows Backup encountered an error when accessing the remote shared folder. Please retry the operation after making sure that the remote shared folder is available and accessible.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
And
Log Name: Microsoft-Windows-SmbClient/Connectivity
Source: Microsoft-Windows-SMBClient
Date: 5/21/2018 4:23:49 PM
Event ID: 30809
Task Category: None
Level: Error
Keywords: (64)
User: N/A
Computer: <servername>
Description:
A request timed out because there was no response from the server.
And
Server name: <Servername>
Session ID:0x17801D0000031
Tree ID:0x1
Message ID:0x4000015
Command: Flush
Guidance:
The server is responding over TCP but not over SMB. Ensure the Server service is running and responsive, and the disks do not have high per-IO latency, which makes the disks appear unresponsive to SMB. Also, ensure the server is responsive overall and not paused; for instance, make sure you can log on to it.
Log Name: Microsoft-Windows-SMBServer/Operational
Source: Microsoft-Windows-SMBServer
Date: 5/21/2018 4:25:37 PM
Event ID: 1020
Task Category: (1020)
Level: Warning
Keywords: (8)
User: SYSTEM
Computer: <ServerName>
Description:
File system operation has taken longer than expected.
Client Name: <IP>
Client Address: <IP:Port>
User Name: <Username>
Session ID: 0x17801D0000031
Share Name: <sharename>
File Name: WINDOWSIMAGEBACKUP\<Servername>\<Backupdate>\<GUID>.VHDX
Command: 7
Duration (in milliseconds): 353151
Warning Threshold (in milliseconds): 120000
Guidance:
The underlying file system has taken too long to respond to an operation. This typically indicates a problem with the storage and not SMB
Cause:
SMB timeout is too short for IO latency for BMR on REFS
Resolution\Workaround:
Option 1:
Set registry and reboot protected server
- Add the registry key below on the protected server
reg add HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters /v SessTimeout /t REG_DWORD /d 1200
Option 2: If you cannot reboot server (Note: Ensure you check the list of dependencies for the Lanmanworkstation service as those services will need to be started again too)
- Add the registry key below on the protected server
reg add HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters /v SessTimeout /t REG_DWORD /d 1200
Note: The 1200 second setting is a 20 minute timeout. This is not a rigid number but has been tested with success. Depending on latency you can adjust as needed. To get suspected latency please review REFS event logs on DPM\MABS server.
- Stop/Start Lanmanworkstation service
Note: There are some other services that depend on Lanmanworkstation, just start those back.
-
Link
There is one optimization change you can make in DPM 2016 using Modern Backup Storage that “may” help reduce the backup times. The issue is that DPM tries to get the size of the recovery point by asking ReFs to count the blocks associated with that recovery point so it can update the DPMUI “storage used” in the UI.
To eliminate that additional calculation that can take a long time and allow the next backup to start you can disable that calculation using the below DPM powershell command.
Manage-DPMDSStorageSizeUpdate.ps1 -ManageStorageInfo StopSizeAutoUpdate
As a result, the “storage consumed” for each data source will not be displayed after a new backup is completed.
The above is covered in the release notes: https://docs.microsoft.com/en-us/system-center/dpm/dpm-release-notes
Section titled: Recovery Points not being pruned, leading to an accumulation of Recovery Points.
If you need to know the amount used for a data source you can use the associated UpdateSizeForDS command.
-
Link
There may be circumstances where the replica volume for a protected data source is under allocated or a very large increase of protected data results in synchronization or recovery point job failures due to inadequate space on the replica.
Type: Recovery point
Status: Failed
Description: DPM is out of disk space for the replica. (ID 58 Details: There is not enough space on the disk (0x80070070))As in previous versions, DPM 2016 has replica auto-grow feature for modern backup storage (MBS) to help increase the replica sizes as the data sources grows. Under normal use conditions this feature works perfectly fine. If a job fails due to out of disk space, it will automatically grow the replica volume and schedule a new synchronization or recovery point job to run one hour in the future. However, if the amount of new data is much larger than the size of the replica after auto-grow, the jobs will continue to fail even as the auto-grow continues to increase the replica size. After two failures, auto-rerun is no longer triggered by default. In DPM 2012 R2 you had the ability to manually grow the replica volume as large as you wanted by using the “modify disk allocation” wizard, however in the DPM 2016 you do not have the ability to manually grow the replica volume using the same wizard. This can cause delays in getting good backups again to meet your SLA. If you know the new size of the protected data or want to grow the replica volume for future data growth in order to prevent future backup failures, you can grow the replica manually.
The below BLOG contains a DPM PowerShell script to make it easy to manually increase a replica volume to the new desired size.
How to increase DPM 2016 replica when using Modern Backup Storage (MBS)
https://techcommunity.microsoft.com/t5/System-Center-Blog/How-to-increase-DPM-2016-replica-when-using-Modern-Backup/ba-p/351818 -
Link
When using DPM 2016 to protect SystemState / BMR workloads the storage used for those backups will be higher when using DPM Modern Backup Storage (MBS) as compared to using legacy Logical Disk Manager (LDM) based storage used in DPM 2012 R2.
When DPM initiates a SystemState or BMR backup, Windows Server Backup is used and overwrites the existing WindowsImagebackup folder with the new backup data. For SystemState that folder is on the protected server and later transferred to the DPM Server replica. For BMR, Windows Server Backup overwrites the folder on the Replica volume directly.
When the replica is located on a LDM based volume, Volume Shadow Copy (VSS) snapshots are used for recovery points. When the files are overwritten on the replica volume, VSS performs a read for the original file blocks – compares them to what is about to be written - then either copies the original blocks to the shadow copy (recovery point volume) if different or just lets the write thru if the blocks are identical. This is called Copy On Write (COW). Between backups a lot of the blocks are the same between the files so the shadow copies were relatively small.
Modern Backup Storage (MBS) uses ReFs clones for recovery points. There is no copy on write operation like there are when using VSS, all writes are performed but instead of overwriting the original files the new blocks are written to different locations on the MBS volume. This is called allocate on write, so even blocks that are the same will use different free space on the Refs volume. That results in more storage consumption as each backup will consume the amount of space as a complete SystemState or BMR backup. IE: If the initial replica size shows 24GB used, then all subsequent backups will come very close to consuming the same amount of space. If the backups are taken frequently or have a long retention period the storage usage can be significant.
To mitigate this behavior I would suggest either take fewer BMR / SystemState recovery points, use a reduced retention period, or just plan on needing more storage for SystemState / BMR backups.
-
Link
System Center Data Protection Manager 2012 R2 UR11 added support for VMware host level backup. If you attempt to perform an Item Level Recovery (ILR) from the latest recovery point for a Protected Windows VM, you receive an ID: 958 popup error message with the below details.
IMPORTANT NOTE: This problem is not present in Microsoft Azure Backup Server (MABS) update rollup 1 (UR1) that also added support for VMWare host level backups.
WORKAROUND(S)
===========1) Select an earlier recovery point if the file(s) (version) you want to recover was included in an earlier recovery point.
2) Restore the entire VM to an alternate location and copy the desired file(s) out of that restored VM.
3) Create a new AD-HOC recovery point manually. That will enable ILR to work for all recovery points including the latest one you made manually.UPDATE: This issue is now fixed in DPM 2012 R2 UR13. However, you need to consider the following for the fix to work.
1. For Protection Groups created after applying UR13, Item Level Recovery will work fine.
2. For older Protection Groups, you need to modify the PGs by adding/removing a small VM or making some configuration changes to the PG for the fix to work. - Link
-
Link
SC DPM 2016 brings in features that delivers improvements in multiple key areas of backup efficiency, performance, flexibility and security. With Modern Backup Storage (MBS), DPM 2016 is turning around the way backups are stored by leveraging modern technologies like ReFS Block Cloning, ReFS Allocate On Write and VHDX. This to 3X faster backups and 50% reduction in storage consumption thus reducing overall backup TCO.
https://blogs.technet.microsoft.com/dpm/2016/10/19/announcing-sc-dpm-2016-with-modern-backup-storage/ -
Link
DPM 2016 comes with Modern Backup Storage which delivers 50% storage savings, 3x faster backups, and more efficient storage with Workload-Aware storage. Modern Backup Storage can be used by setting up DPM 2016 on Windows Server 2016.
Add Storage to DPM 2016 -
Link
Very happy to report the release SC DPM 2012 R2 UR11 that adds protection for VMware host level backup of VMs. Here are major features of VMware VM Backup. Here are the few highlights of VMware VM backup.
- Agentless backup
- Backup at vCenter or ESXi level
- Backup VMware VM folders
- Ability to do ILR
- Protect large data center using DPM Scale Out
- Backup to Disk/Cloud
Here are the blogs and articles that are published.
BLOG - Implementing Microsoft DPM host level protection of VMware VMs
BLOG-Announcing VMware VM Backup using System Center Data Protection Manager
-
Link
We have made a bunch of improvements in this update, related to reboot requirements, Enterprise file server backup, online backups and more.
-
Link
After you install DPM 2012 R2 UR7 update (KB3065246), DPM cannot delete expired recovery points. This will lead to extra recovery point volume usage.
Please download and install the fix for this issue.
Fix for a known issue with Update Rollup 7 for System Center 2012 R2 Data Protection Manager
MORE INFORMATION
==============DPM 2012 R2 UR7 introduced a new switch to the powershell cmdlet Get-RecoveryPoint, it now requires a -Datasource switch as seen below:
Get-RecoveryPoint -Datasource $ds
If you have any custom powershell scripts that use that cmdlet, please modify them to use the new -Datasource switch.
-
Link
Below are some handy links for the DPM Console troubleshooter which can help you resolve console crashes and three DPMDB maintenance blogs.
Guided Walkthrough for Troubleshooting Data Protection Manager console issues
DPMDB Maintenance Part 1: Database consistency check and your DPMDB
DPMDB Maintenance Part 2: Identifying and dealing with fragmentation in your DPMDB
DPMDB Maintenance Part 3: Dealing with a large DPMDB -
Link
Hello - Please find the following BLOG on the new streamlined method of registering a DPM 2012 SP1 or later DPM server to backup to Azure. There is no longer a requirement to upload a certificate when registering a DPM server for online backups.
Below are two Blogs on the subject.
How to register a DPM 2012 SP1 or later server to backup to Microsoft Azure
Simplified workflow to configure backup to Azure from DPM -
Link
Please find the following whitepaper detailing how to dedup DPM storage pool to help reduce overall disk storage requirements for disk based backups. This solution requires Windows 2012 R2 and System Center Data Protection Manager 2012 R2
-
Link
For DPM Console crash issues, please review this troubleshooter before posting a new question.
Guided Walkthrough for Troubleshooting Data Protection Manager console issues
-
Link
updated 6-5-13
Hello Everyone,
System Center DPM 2012 is not supported running on Windows Server 2012. We have had reports of DPM 2012 console crashes when navigating to the recovery tab if DPM 2012 is installed on Windows server 2012. This issue is fixed in System Center 2012 SP1 which supports being installed on Windows 2012 operating system.
If you are already in this condition, you have two options:
A) You can upgrade to System Center 2012 Sp1.
B) You can backup the DPMDB, rebuild the DPM server using Windows 2008 R2 install DPM 2012, then restore your current database. The process is explained in the below technet article.
Recovering DPM servers
http://technet.microsoft.com/en-us/library/jj244606.aspxWe also do not support protecting Windows Server 2012 workloads using SCS DPM 2012, you must upgrade to DPM 2012 Sp1. Please do not expect any support for DPM 2012 running on Windows Server 2012, or issues while protecting data sources in Windows Server 2012.
