locked
Server 2012 Dedup not processing anymore? RRS feed

  • Question

  • I have configured the dedup to 2 days but I have several files that look the same and are over 2 days that are not getting deduped.

    Everytime I start a optimization job manually, the process indicator just stand at 0% and doesn't seem to move. I can see that the system is taking up memory resources.

    Is there someway I can see which file the dedup is currently procressing or get more detailed info about the dedup progress?

    I have checked the event logs for Dedup service but they don't contain anything helpful.


    Monday, May 27, 2013 7:00 AM

All replies

  • Let deduplication process complete...

    StarWind iSCSI SAN & NAS

    Monday, May 27, 2013 9:21 AM
  • Is there someway you can see when it will complete? I can only see that it is showing 0% after many many hours.

    • Edited by ALX- Monday, May 27, 2013 9:27 AM
    Monday, May 27, 2013 9:27 AM
  • Is there someway you can see when it will complete? I can only see that it is showing 0% after many many hours.

    Use referenced PowerShell scripts (Get-DedupJob and Get-DedupStatus) to query extended information about what's there. See:

    http://technet.microsoft.com/en-us/library/hh831434.aspx

    Please post info here. Thanks!


    StarWind iSCSI SAN & NAS

    Monday, May 27, 2013 4:42 PM

  • The starttime was yesterday 21:54 and the time now is 19:43 so it has been running for 22 hours and still 0% progress.

    There is no information about estimated completion time. Not even if I add | fl in the command.

    PS C:\windows\system32> get-dedupjob

    Type               ScheduleType       StartTime              Progress   State                  Volume
    ----               ------------       ---------              --------   -----                  ------
    Optimization       Scheduled                                 0 %        Queued                 E:
    Optimization       Scheduled                                 0 %        Queued                 E:
    Optimization       Scheduled          21:54                  0 %        Running                E:


    PS C:\windows\system32> get-dedupstatus

    FreeSpace    SavedSpace   OptimizedFiles     InPolicyFiles      Volume
    ---------    ----------   --------------     -------------      ------
    5.64 TB      3.31 TB      474                474                E:


    • Edited by ALX- Monday, May 27, 2013 5:45 PM
    Monday, May 27, 2013 5:45 PM
  • Hi.

    How big is the volume if you saved 3.31 TB?


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Oscar Virot

    Monday, May 27, 2013 7:24 PM
  • Hi.

    I found the following passage at Technet - Plan to Deploy Data Deduplication

    Deduplication throughput

    The deduplication feature can process roughly 2 TB of data per volume in a 24-hour period (100 GB per hour) when running a throughput optimization job on a single volume. Multiple volumes can run in parallel if CPU core processors and memory are available. The actual speed of deduplication is dependent on CPU, disk I/O, memory and available server resources at the time that deduplication is running. Deduplication is designed to pause if the server starts to run out of resources so it does not impact the server’s regular workload. The busier the server is, the longer it will take to process the data. If you have a data volume that is very busy, you might consider scheduling a regular throughput optimization job during off-peak hours. For more information, see Enable data deduplication on the volume.

    So not knowing how big the volume is a couple of days is not unexpected time.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Oscar Virot

    Monday, May 27, 2013 7:36 PM
  • The volume is 16,3 TB. Do you mean that it will take roughly 8 days for it to complete a dedup ? Or is it just the amount of files in size that matters?

    Monday, May 27, 2013 7:53 PM
  • I haven't been able to play with dedup with large volumes yet so I'm not really sure. But according to that page I would guess 2TB of new files per day. So if you have added 4 TB please calculate 2 days. If you interrupt the scan im not sure if it needs to restart from the beginning with 8 days.

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Oscar Virot

    Monday, May 27, 2013 8:04 PM
  • If dedup can't resume and it stops at max 24 hours it would mean dedup is not working on > 2TB files?

    Can't find any more info on this. Anyone else have a clue? And can you see more detailed which file is being processed?

    Monday, May 27, 2013 8:24 PM
  • Hi.

    Have you proved the Update-DeDupStatus cmdlet?


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Oscar Virot

    Monday, May 27, 2013 10:09 PM
  • Hi,

    As you said that the max value of Duration of deduplication process is 24 hours. So if the dedupliation process cannot be finished in 24 hours, it will cause a problem as the job cannot be finished in time.

    I believe this is not the purpose of setting the 24 hours limitation and I will try to confirm with development team about this.


    TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tnmff@microsoft.com.

    Wednesday, May 29, 2013 8:14 AM
  • The GUI only lets you configure for 24 hours, but I have tried using powershell to set the value to 0 and it changes the task scheduling job to run for 3 days.

    Wednesday, May 29, 2013 8:29 AM
  • Hi.

    I dont have an lab with that amount of data to test, but with the schedule you can set durationHours.

    Set-DedupSchedule

    -DurationHours<UInt32>

    Specifies the number of hours that the server runs the task before canceling it. The value 0 indicates that the server runs the job to completion. This cmdlet safely stops a data deduplication job and does not affect the files that the server is processing when it cancels the job.



    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Oscar Virot

    Wednesday, May 29, 2013 8:37 AM
  • Hi,

    I got the following reply from development group:

    The number applies to how much can be deduplicated on a single volume.
    If the machine has multiple cores and sufficient memory then Dedup will schedule jobs to run in parallel, one per volume, so the overall throughout of the machine could be 4x, 8x or even 16x than the number you quoted.
    If you have a choice when you provision a machine intended for Dedup is to provision with multiple large volumes rather than one huge volume (e.g. create 16 4TB volumes each rather than a single 64TB volume), Dedup does not have at the moment the ability to run multiple jobs in parallel on the same volume.

    So it seems that it is recommanded to have mutilple volume with lower size to get volumes deduplicated in 24 hours, instead of running dedup job for several days on a large volume.


    TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tnmff@microsoft.com.

    Thursday, May 30, 2013 1:05 AM
  • Thanks for the reply, but I don't know what to make of the recommendations.

    I have several full backup file that is 3,5 TB in size. If I make several 4 TB volumes as the developement group suggested, would dedup savings still work across all the volumes?

    For example, if I put each of the 3,5 TB file in each of the 4TB volumes, will it dedup the 3,5 TB files in each volume and save me storage?

    Or is it simply that my files are not a good candidate to be deduped?

    Then the other part is to make the backup software change backup patch each time because the volumes can be max 4 TB. Doesn't sound to convenient?


    • Edited by ALX- Thursday, May 30, 2013 8:52 PM
    Thursday, May 30, 2013 8:50 PM