locked
Use BranchCache for deduplication replication? RRS feed

  • Question

  • I have a deduplicated volume on Server A, and I would like to replicate the contents on a daily basis to Server B. I want the replication to be transmitted using the content chunks and index, not as raw files.  I've seen hints that BranchCache is the way to do this, but I have not seen an actual discussion of the process.

    Assuming BrnachCahe hosting is set up on the volume on Server A, how should I set up Ser ve B with branchCache? As a clinet, or as a hosted client? Do I need to precache the content using PowerShell? Do I do that on Server A, or Server B? Will I end up with a replicated volume on Server B? How do I acces the replicated files on Server B, directly from the server B volume, or do I have to access the files using Server A, and BranchCache automagically translates that a reference to the local Server B?

    Thanks,

    Brad Jensen

     
    Tuesday, November 13, 2012 8:23 PM

Answers

  • Hi Brad,

    It seems that DFSR is a better option for replicating files from Server A to Server B and keep them sync.

    DFS Step-by-Step Guide for Windows Server 2008

    http://technet.microsoft.com/en-us/library/cc732863(v=ws.10).aspx

    And for BranchCache configuration, I found a third party website which proivded detailed steps:

    http://www.windowsnetworking.com/articles_tutorials/Deploying-Hosted-Mode-BranchCache-Server-Part3.html

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    And please see the BranchCache Deployment Guide or other documentation for additional steps. Links to all of the BranchCache documents are at "BranchCache for Windows Server 2008 R2" at http://technet.microsoft.com/en-us/library/dd996634(WS.10).aspx


    TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tnmff@microsoft.com.

    Thursday, November 15, 2012 9:10 AM
  • I have a deduplicated volume on Server A, and I would like to replicate the contents on a daily basis to Server B. I want the replication to be transmitted using the content chunks and index, not as raw files.  I've seen hints that BranchCache is the way to do this, but I have not seen an actual discussion of the process.

    Assuming BrnachCahe hosting is set up on the volume on Server A, how should I set up Ser ve B with branchCache? As a clinet, or as a hosted client? Do I need to precache the content using PowerShell? Do I do that on Server A, or Server B? Will I end up with a replicated volume on Server B? How do I acces the replicated files on Server B, directly from the server B volume, or do I have to access the files using Server A, and BranchCache automagically translates that a reference to the local Server B?

    Thanks,

    Brad Jensen

     
    Brad, where did you find BrachCache (or anything within MS operating system built-in) does move de-hydrated content between places? I've been looking for such a solution quite for a while w/o much luck: seems like everything involved does re-hydrate content first and the only data reduction technique being in use is compression (also not very good one as turining it OFF and using third-party tools like Silver Peak and NexEx WAN accelerators does at least twice better job, makes A LOT of sense if you have something like T1 between data centres). 

    StarWind iSCSI SAN & NAS

    Wednesday, November 28, 2012 8:07 PM

All replies

  • Hi Brad,

    It seems that DFSR is a better option for replicating files from Server A to Server B and keep them sync.

    DFS Step-by-Step Guide for Windows Server 2008

    http://technet.microsoft.com/en-us/library/cc732863(v=ws.10).aspx

    And for BranchCache configuration, I found a third party website which proivded detailed steps:

    http://www.windowsnetworking.com/articles_tutorials/Deploying-Hosted-Mode-BranchCache-Server-Part3.html

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    And please see the BranchCache Deployment Guide or other documentation for additional steps. Links to all of the BranchCache documents are at "BranchCache for Windows Server 2008 R2" at http://technet.microsoft.com/en-us/library/dd996634(WS.10).aspx


    TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tnmff@microsoft.com.

    Thursday, November 15, 2012 9:10 AM
  • Just FYI, there is a new Microsoft guide for deploying BranchCache in hosted cache mode, at http://technet.microsoft.com/en-us/library/jj862369.aspx

    Thanks -


    James McIllece


    Thursday, November 15, 2012 7:34 PM
  • hanks, Shaon, but DFSR does not transfer the content chunks. Instead it expands the fileto raw size internally, then transfer the raw chunks using its own compression algorithm. We tested it, and the transmission is typically 1/3 of the raw file size, where the dedupliation chunks are 1/50 of the raw file.  Since each new daily backup file is 50 GB to 4 TB, DFSR is just too slow.

    It would be nice if MS would fix the replication to work with the deduplicated chunks, like other deduplication providers do.

    Friday, November 16, 2012 11:01 PM
  • I have a deduplicated volume on Server A, and I would like to replicate the contents on a daily basis to Server B. I want the replication to be transmitted using the content chunks and index, not as raw files.  I've seen hints that BranchCache is the way to do this, but I have not seen an actual discussion of the process.

    Assuming BrnachCahe hosting is set up on the volume on Server A, how should I set up Ser ve B with branchCache? As a clinet, or as a hosted client? Do I need to precache the content using PowerShell? Do I do that on Server A, or Server B? Will I end up with a replicated volume on Server B? How do I acces the replicated files on Server B, directly from the server B volume, or do I have to access the files using Server A, and BranchCache automagically translates that a reference to the local Server B?

    Thanks,

    Brad Jensen

     
    Brad, where did you find BrachCache (or anything within MS operating system built-in) does move de-hydrated content between places? I've been looking for such a solution quite for a while w/o much luck: seems like everything involved does re-hydrate content first and the only data reduction technique being in use is compression (also not very good one as turining it OFF and using third-party tools like Silver Peak and NexEx WAN accelerators does at least twice better job, makes A LOT of sense if you have something like T1 between data centres). 

    StarWind iSCSI SAN & NAS

    Wednesday, November 28, 2012 8:07 PM
  • Just FYI, there is a new Microsoft guide for deploying BranchCache in hosted cache mode, at http://technet.microsoft.com/en-us/library/jj862369.aspx.

    Thanks -


    James McIllece

    Please fix the link as you have extra "." @ the end so it gives 404 error unless "." removed :) Thanks!

    StarWind iSCSI SAN & NAS

    Wednesday, November 28, 2012 8:08 PM
  • Wednesday, November 28, 2012 8:50 PM
  • http://technet.microsoft.com/en-us/library/jj862369.aspx

    :-)

    Thanks -


    James McIllece

    I mean - edit your own post above for other guys :) I could remove "." myself to navigate :)

    StarWind iSCSI SAN & NAS

    Wednesday, November 28, 2012 10:32 PM
  • So basically the impression I am getting is that there is no way to replicate a deduplicated volume in Windows 2012 server.

    I wonder if MS plans to upgrade DFS-R to be deduplication replication capable.  A simple starting point would be to have it transfer dedupliation chunks to the remote volume, reinflating them in the process, and then run the dedupe optimize on the remote deduplication-enabled volume.

    The real bottleneck in this process for larger replication events (100GB and up) is the network bandwidth, not the cpu or disk on either end.

    I would write it for them, but then I would have to sell it, and then MS would fire up the magic gruel kettle and put 100 gnomes to work reproducing it to give away for free.

    I wish they would just go ahead and do that now.

    All the guys who just got done doing the Windows 8 upgrade (codename: Windows LSD) are probably looking for new fun projects.

    Come on, that's funny, I don't care who you are.

    Thursday, November 29, 2012 9:01 PM
  • I know this is a 2 year old post, but I have the same issue and still no useful documentation exists as usual with MS. I am struggling with bandwidth issues syncing a 2TB volume to a remote location. Our previous backup solution used source-side dedup, hasing all contents, sending them over and only sending unique blocks, dramaticaly saving bandwidth as well as time. Unfortunately our new backup solution doesn't have that functionality. I need to sync or copy the contents from our datacenter to the remote location. After reading this part:

    [quote]

    6) BranchCache™: Another benefit for Windows is that the sub-file chunking and indexing engine is shared with the BranchCache feature. When a Windows Server at the home office is running deduplication the data chunks are already indexed and are ready to be quickly sent over the WAN if needed. This saves a ton of WAN traffic to a branch office. 

    [/qoute]

    I was hoping to be able to use Branchcache together with dedup to save bandwidth, hasing all content and sending only unique blocks. I've been reading for days now, and testing a lot but haven't been able to pull this.

    Brad, have you been able to work this out?


    Monday, December 29, 2014 1:20 PM
  • Robert

    I never found anything to replicate Windows Deduplication volumes, so...we wrote our own replication software for Windows Deduplication. We are releasing it now as Replacador, but I can send you a a free copy as a professional courtesy.

    Basically it takes a snapshot, runs the deduplication job, replicates it to one or more identical deduplication copies, then removes the snapshot.It doesn't reflate anything, it sends only the reparse points and chunk store changes.

    I've also done quite a bit of test runs of Windows Deduplication, and I've come up with some interesting results I am putting on windowsdeduplication.com


    Wednesday, February 18, 2015 6:42 AM
  • Hi Brad,

    I'd certainly appreciate that. I've been working on this for a while but it seems I am unable to achieve this with Branchcache or any other Microsoft product I am aware of. Actually I only want to copy the last 10 or so backup files over, not the entire main-site-backup-repository. The remote site only holds the copies we need to get back running when the main site burns down or something like that. Having backups of the last 2 months is not evident at the moment I actually need that remote backup. However, if it would be only possible to replicate the whole volume, I think that's ok with me as well :)

    I will test the product and report any issues or bugs I might encounter in order to improve it.

    Thanks!

    Wednesday, March 18, 2015 3:34 PM
  • To test Windows Deduplication replication ith the 3rd party Replacador application, go to 

    replication-for-windows-2012-r2-deduplication-with-replacador

    there is a link o download the software, a second for the manual, and a third to request and evaluation ky


    Wednesday, March 18, 2015 4:22 PM
  • Ah, the evaluation version I could ofcourse always get, although that's not what you implied :) Will try with the evaluation though.
    Wednesday, March 18, 2015 8:07 PM