none
DPM showing negative values for used space RRS feed

  • Question

  • DPM... can anybody explain and/or tell me how to fix DPM displaying NEGATIVE values for allocated space?

    Let me give you an example: in the Protection tab, for some datasources, DPM shows 0kB storage consumed despite having couple recovery points, all of which are browsable and the data are there. When queried by Powershell, you see DPM shows negative values for ShadowCopyUsedSpace and ContainerUsedSpace properties:

    PS C:\Users\admin\Desktop> $ds | ? name -eq 'rct\<redacted_vm_name>' | select *space*
    AdditionalSpaceRequired      : 0
    ReplicaUsedSpace             : 41127190528
    ShadowCopyUsedSpace          : -60821958656
    ContainerUsedSpace           : -60821811200
    CanDisplayContainerUsedSpace : True

    DPM 2019, the state of the art backup system.

    Monday, October 14, 2019 3:15 PM

All replies

  • Hello Markos,

    Can you post the full PowerShell command you use?

    The DPM console showing "0 KB" is very strange, haven't encountered this before, have you tried restarting the console or the DPM server?

    Does it show "0 KB" for all protected workloads?

    What does the DPM reports tell about the disk capacity?

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, October 14, 2019 3:54 PM
  • Hi Leon,

    it's just basic commands: $DS = Get-DPMProtectionGroup | %{Get-DPMDatasource -ProtectionGroup $_} and then I just picked the one with negative values.

    The zero (negative) values are displayed for couple data sources. not too many, but still...

    I even used your reporting script from technet gallery (btw you have a mistake there - missing Read-Host on the first line that should ask you for DPM server name).

    What do you mean about the DPM reports - which particular one? Disk allocation report shows negative values for the same datasources - see attached screenshot for an excerpt


    • Edited by MarkosP Tuesday, October 15, 2019 2:48 PM
    Tuesday, October 15, 2019 9:32 AM
  • Thanks for correcting that :-)

    This is very interesting and not normal at all, what does the following command give you as a result?

    Get-DPMDataSource | Where {$_.Name -eq "RCT\VMname"} | Select ProductionServerName, Name, DiskAllocation | Sort-Object ProductionServerName, Name | Fl


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, October 15, 2019 2:43 PM
  • Well I'd hope that this is not normal at all, but I can't be surprised by DPM anymore :)

    Here's the output (redacted):

    ProductionServerName : <hyper-v-host-name>
    Name                 : RCT\<vm-name>
    DiskAllocation       : {Storage consumed:, 0 KB}



    • Edited by MarkosP Tuesday, October 15, 2019 5:35 PM
    Tuesday, October 15, 2019 2:51 PM
  • Any ideas Leon?

    Btw the reason why I started to dig into this is that I wanted to make a storage usage report. The problem is the data doesn't add up. When I for example use your script and sum both columns (either with the negative values or when making them positive), it doesn't add up to the total storage used (as reported in the Management tab).

    Thursday, October 17, 2019 11:12 AM
  • Sorry for the late reply, I'm not sure why it's showing negative values or "0 KB", my guess is that this could possibly be a bug.

    According to your given screenshot it appears to be a Hyper-V virtual machine? 

    Could you give us some more details about your environment?

    • DPM version and build.
    • Hyper-V host OS version.
    • Hyper-V guest VM OS version.
    • Hyper-V guest VM configuration version.

    Also give a list of all worklaods that are showing negative / 0KB values for the storage consumed.



    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, October 22, 2019 8:47 PM
  • Yes, 95% of the datasources are Hyper-V VMs.

    - DPM 2019, since there's no UR out there yet, it's build 10.19.58.0

    - Hyper-V Server 2016

    - guest VMs are mostly WS2016, some WS2019

    - all VMs config version 8

    What list of workloads? I can't post the real VM names here, do you mean how many exhibit this behavior?

    Tuesday, October 22, 2019 9:35 PM
  • So this issue only occurs on Hyper-V virtual machine workloads? For example RCT\VMname ?

    And are the VMs all backed up on host-level?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, October 22, 2019 10:15 PM
  • Currently yes, I didn't exactly track this. At this moment, it's showing negative values for 20 VMs, spread on 3 different Hyper-V hosts.

    All VMs that are backed up are backed up on the host level using RCT.

    There's couple volume/share backups and 2 BMR/System state backups, the rest is HV RCT (~200)

    Tuesday, October 22, 2019 10:29 PM
  • Any noticeable differences between the VMs that are reporting the storage consumed correctly and those VMs that are not?

    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, October 23, 2019 5:25 AM
  • What kind of diferences? There are none that I can think of.

    For reference, here's the current screenshot of the output of your script:


    • Edited by MarkosP Wednesday, October 23, 2019 9:20 AM
    Wednesday, October 23, 2019 9:19 AM
  • I haven't really used the "RecoveryPointVolumeSizeInGb" for Hyper-V workloads so I cannot say how it should look like. Comparing the Hyper-V workload sizes with other workload's sizes it doesn't make much sense that the values are negative, which makes this look like a bug.

    The "Storage consumed" shouldn't show 0 KB though, this indicates that something is wrong, and honestly most functions in DPM are pretty basic, so there's not much to it really.

    I would suggest you create a ticket directly to Microsoft about this, if this is indeed a bug it will be free of charge.

    https://support.microsoft.com/en-us/hub/4343728/support-for-business


    Blog: https://thesystemcenterblog.com LinkedIn:

    Sunday, October 27, 2019 11:24 PM
  • We're a MS partner, so that's not an issue, but the hopes of fixing it are super low. I'll wait for UR1 (which is overdue anyway).
    Tuesday, October 29, 2019 9:56 AM
  • Leon, I've read rumors that UR1 was pushed to 2020, do you know anything about it?

    Also, I can't figure out what values to use to determine space used by the backups. I have another DPM server which doesn't show any negative/0 values for backups, but when I list the sizes using your (or other) script, the sums don't make sense. For example:

    Sum of 'ReplicaSizeInGB': 31596

    Sum of 'RecoveryPointVolumeSizeInGB': 37086

    Total used space reported in the Management - Disk Storage section: 37826

    Any idea what values should be used for different backup types to get a correct sum?

    Thursday, November 14, 2019 8:36 AM
  • Yes the Update Rollup 1 for DPM 2019 has been postponed to Q1 of 2020.

    https://thesystemcenterblog.com/2019/11/07/system-center-roadmap-update/

    It might be my script that doesn't work too well with Modern Backup Storage (MBS), I will have to check on it when I have time.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, November 14, 2019 1:46 PM
  • Hi,

    Just checking to see if you have any update on your issue?

    If your issue was resolved, may I ask you to mark all the answers that helped you? This way it will also help others in the future who face the same challenge. Many thanks in advance!


    Best regards,
    Leon

    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, December 11, 2019 9:35 PM
  • As a side effect of our DR plan, this problem was (temporarily) mitigated (we have 2 DPM servers, only using 1 at a time and when the one in use slows down over time (due to the various problems like ReFS fragmentation, VHD mount bugs etc) we recreate all backups on the other DPM server and delete backups on the original one).

    So as of now, there are no backups with negative/zero values, but that doesn't mean the problem is resolved. I'll leave this thread open.

    Still, the totals don't make up for the storage used, so there's some other problem there.

    Totals:

    ReplicaSizeInGB = 33153

    ReplicaVolumeSizeInGB = 44950

    Storage size [GB] = 50518

    Storage free [GB] = 681

    So it seems there are 16684 (50518-681-33153) GB missing (occupied by something, but not reported by DPM as used...

    Thursday, December 12, 2019 7:56 AM
  • When you create a protection group in DPM it pre-allocates space on the disks for recovery points, based on the amount of recovery points you specify and the retention time, therefore the space needed really relies on the delta change in data not the actual volume of data.

    I suspect the "pre-allocated" storage size is not counted in those values.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, December 12, 2019 9:10 AM
  • I know, but the pre-allocated size should be the 'ReplicaVolumeInGB' value or not?
    Thursday, December 12, 2019 9:12 AM
  • It "should", but I'm not sure if it does.

    Do you get the same replica size by running the script over here?

    https://techcommunity.microsoft.com/t5/System-Center-Blog/How-to-increase-DPM-2016-replica-when-using-Modern-Backup/ba-p/351818


    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, December 12, 2019 9:30 AM
  • Nope, that totals to 63413 GB :)
    Thursday, December 12, 2019 10:38 AM
  • The problem is all of these scripts work with different values and it seems noone know what the values mean.

    Here's an example datasource - I've pulled some size-related properties:

    ReplicaSize                             : 6264706826240
    RequiredReplicaSize                : 6264706826240
    ShadowCopyAreaSize              : 0
    RequiredShadowCopyAreaSize : 0
    ReplicaUsedSpace                   : 2941629292544
    ShadowCopyUsedSpace           : 3037837189120
    ContainerUsedSpace                : 3037837332480
    DiskAllocation                         : {Storage consumed:, 2 829,20 GB}

    What DPM shows as 'Storage consumed' in the GUI is the DiskAllocation size, which corresponds to the ShadowCopyUsedSpace or ContainerUsedSpace properties (hard to tell which, since the DiskAllocation is rounded down and both these values are very similar).

    I assume ReplicaSize/RequiredReplicaSize is the maximum size of the dynamic VHDX?

    It would be nice if someone could provide an answer to the meaning of these values and how to relate them to the total storage consumed on the storage as shown by the GUI.

    It's embarassing that it's such a problem to get a storage report from DPM ....

    Thursday, December 12, 2019 10:52 AM
  • I assume ReplicaSize/RequiredReplicaSize is the maximum size of the dynamic VHDX?

    >> Yes.

    Retrieving the actual data consumption from backup software has always been somewhat of a challenge.

    The only thing that is showing correctly is the Used space of the whole storage pool, and checking the actual disk space usage in Windows.

    Also if you have data deduplication enabled, the values might differ a lot.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, December 12, 2019 11:00 AM
  • I've done a comparison of the DiskAllocation - sum over all datasources (thanks to the hilarious formatting (spaces, various "smart" sizes (GB/MB)) it's overly complicated and manual work) and it +- corresponds to the total storage consumed reported in the GUI.

    The problem with that is sometimes the datasource DiskAllocation is way too big as if DPM doesn't free up the space - for example I can manually delete all but one RP for a DS, but this size will be much larger than if I delete the whole datasource and make a completely new backup. But that is another problem and I don't want to sidetrack this thread further...

    Thursday, December 12, 2019 11:11 AM
  • Hi Mark,

    Just for clarity ReplicaSize = (replica.vhdx logical size), ReplicaUsedSpace = (datasource data size), ContainerUsedSpace = (Sum of ReplicaUsedSpace + all RPs)

    After every recovery point (or synchronization) job, DPM asks ReFs to get the size of the all the unique blocks written to the replica for that job and DPM adds the sum to the ContainerUsedSpace which is what you see in the DPMUI for "storage consumed".

    Should it get out of whack for a few data sources you can run a DPMSYNC -ReconcileStorageInfo DatasourceID

    You can also run for all data sources, that may take a long time. DPMSYNC -ReconcileStorageInfo

    To get the datasourceID - Open SQL management studio and run the below query and copy the data source ID for the affected data source.

    Select DPMDB ---change to be your dpmdb name if different
    select ag.NetbiosName as servername, ds.DataSourceName,lr.DataSourceId
    from tbl_IM_DataSource as ds
    join tbl_PRM_LogicalReplica as lr
    on ds.DataSourceId=lr.DataSourceId
    join tbl_AM_Server as ag
    on ds.ServerId=ag.ServerId
    --where datasourcename = '%name%'  --optionally put in your datasource name
    order by ag.NetbiosName

    Then run from an administrative command prompt: DPMSYNC -ReconcileStorageInfo DatasourceID

    After it completes, close the DPM Console and re-open and see if that corrects the "Storage Consumed".

    revert back and let us know if that helped


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Thursday, December 12, 2019 10:19 PM
    Moderator
  • Thanks for chiming in and the clarification Mike, I'll test dpmsync for single datasource when/if it starts showing incorrect values.

    However I'm bit reluctant to use dpmsync again, because last time we did (per MS support engineer's instructions), our DPM became locked for many hours (roughly 2 days) and the result was all datasources were inconsistent and we had to run CC jobs which took days...

    Thursday, December 12, 2019 10:33 PM
  • Hi Mark, Yes- DPMSYNC -Sync command can run for a long time and will mark all replicas as inconsistent however DPMSYNC -ReconcileStorageInfo DatasourceID Is a totally different function and should be quick and will not affect the replica state. Anyhow if it happens again give it a whirl. Regards Mike Jacquet

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, December 13, 2019 1:29 AM
    Moderator