locked
Question about best way to distribute data RRS feed

  • Question

  • We are currently looking for a way to make data available in 2 separate locations, but as though they are actually all in 1 location as far as the user or the client computer is concerned.

    I am a developer by trade, so I know a decent amount about server technologies, but I am no pro by any stretch. Most of my work is on the client side of things.

    So I am really just fishing for ideas on how I would be able to do this.

    1) Currently there is no network setup between the 2 locations, and currently no servers are deployed at all. Just client machines at each location
    2) Because of #1, there is obviously no domain controller or anything like that
    3) I was looking at DFS because it looked promising, but it looked as though it is not designed to handle very large amounts of data?


    My ultimate goal is to make it so each client machine at both locations have a central repository to store large amounts of data. I know there are some web based solutions out there, but I want to keep everything in the physical locations of this network I want to create.

    So I want the client machines to be able to (for example) have a mapped network drive (S). This network drive would point to a server location that would contain both the data from a server at location A, and also the data on a server at location B, as if they were both in the same location. Most of this is to minimize redundancy in the data (the bad kind of redundancy). We would plan to at least RAID 0/10 the disks in each server for fault tolerance on the hardware side of things.

    So like I said, I am just looking for ideas, perhaps I am going in the totally wrong direction, which is why I figured I would post here.

    If I have not been clear, or you need more info to give me an accurate suggestion, please let me know.
    Matt Kleinwaks - MSMVP MSDN Forums Moderator - www.zerosandtheone.com - visit my website for a chance to win an MSDN subscription.
    Tuesday, September 16, 2008 6:40 PM

Answers

  • Hi Matt,

     

    I think the DFS solution of Windows Server 2008 may meet the needs of your demand. By using domain-based DFS, you may store files on distributed servers makes files available to the users and decreases latency and bandwidth use when the servers are located near users.

     

    Suggestion:

     

    1.      Location A: Deploy an Active Directory by promoting a Windows Server 2008 based-domain controller, and add DFS service role to make it hold the DFS namespace.

    2.      Location B: Deploy another Windows Server 2008 based member server, and also add DFS service role to hold the DFS namespace.

    3.      Make the two server become the DFS member servers, both of them hold the target folder both in Location A and Location B.

    4.      Configure DFS replication group to replicate the folder between location A and location B.

    5.      After you create the DFS share, you may make all the clients map a network drive to \\domainname\namespace\sharefolder.

     

    For your convenience, I have list some step-by-step guides as follow, you may refer to them

     

    AD DS Installation and Removal Step-by-Step Guide

    http://technet.microsoft.com/en-us/library/cc755258.aspx

     

    DFS Step-by-Step Guide for Windows Server 2008

    http://technet.microsoft.com/en-us/library/cc732863.aspx

     

    Overview of the Distributed File System Solution in Microsoft Windows Server 2003 R2

    http://technet.microsoft.com/en-us/library/cc787066.aspx

     

    Distributed File System Replication: Frequently Asked Questions

    http://technet2.microsoft.com/windowsserver2008/pt-BR/library/f9b98a0f-c1ae-4a9f-9724-80c679596e6b1046.mspx?mfr=true

     

    Hope it helps.


    David Shen - MSFT
    • Proposed as answer by Edwin vMierlo Friday, September 19, 2008 9:29 AM
    • Marked as answer by kleinma Friday, September 19, 2008 1:55 PM
    Friday, September 19, 2008 9:26 AM

All replies

  • Hi Matt,

     

    I think the DFS solution of Windows Server 2008 may meet the needs of your demand. By using domain-based DFS, you may store files on distributed servers makes files available to the users and decreases latency and bandwidth use when the servers are located near users.

     

    Suggestion:

     

    1.      Location A: Deploy an Active Directory by promoting a Windows Server 2008 based-domain controller, and add DFS service role to make it hold the DFS namespace.

    2.      Location B: Deploy another Windows Server 2008 based member server, and also add DFS service role to hold the DFS namespace.

    3.      Make the two server become the DFS member servers, both of them hold the target folder both in Location A and Location B.

    4.      Configure DFS replication group to replicate the folder between location A and location B.

    5.      After you create the DFS share, you may make all the clients map a network drive to \\domainname\namespace\sharefolder.

     

    For your convenience, I have list some step-by-step guides as follow, you may refer to them

     

    AD DS Installation and Removal Step-by-Step Guide

    http://technet.microsoft.com/en-us/library/cc755258.aspx

     

    DFS Step-by-Step Guide for Windows Server 2008

    http://technet.microsoft.com/en-us/library/cc732863.aspx

     

    Overview of the Distributed File System Solution in Microsoft Windows Server 2003 R2

    http://technet.microsoft.com/en-us/library/cc787066.aspx

     

    Distributed File System Replication: Frequently Asked Questions

    http://technet2.microsoft.com/windowsserver2008/pt-BR/library/f9b98a0f-c1ae-4a9f-9724-80c679596e6b1046.mspx?mfr=true

     

    Hope it helps.


    David Shen - MSFT
    • Proposed as answer by Edwin vMierlo Friday, September 19, 2008 9:29 AM
    • Marked as answer by kleinma Friday, September 19, 2008 1:55 PM
    Friday, September 19, 2008 9:26 AM
  • Well Dave,

    I don't think I will get a more thorough answer than that so I have marked as the answer.


    We don't have 08 servers at either location, but I don't think that will be an issue if it will meet our needs.

    One further question if you happen to revisit this post (I plan to try to find out myself too) is what are the ceiling limits with regards to size?

    Are there limits on the number of files, size of the shares, etc.. that we might hit at some point? I just need to be able to plan for that if there is any sort of limit, as these locations deal with large amounts of files and data back and forth, anywhere from files sizes of a few K upwards of a gig. Actual number of files will grow over time, but the initial support would need to be able to handle upwards of 500,000 files.

    Maybe these are not even limitations I need to worry about, I just want to be sure before I make a proposal.
    Matt Kleinwaks - MSMVP MSDN Forums Moderator - www.zerosandtheone.com
    Friday, September 19, 2008 2:01 PM
  • Hi,

     

    Thanks for the reply.

     

    As you said, you don't have Windows 2008 servers at either location. I think the Windows Server 2003 R2 is also OK. You can also enable and DFS namespace and DFS replication on Windows Server 2003 R2 based-server.

     

    DFS-R has no real limit on the size of files that can be replicated. The DFS-R Jet database has a theoretical limit of 32TB, with roughly 8GB available for file ID records. A volume can contain up to 8 million replicated files, and a server can contain up to 1 terabyte of replicated files. Actually, it is more a file system limitation and has nothing really to do with DFS-R but DFS-R will suffer from it. The real limit for now is the maximum NTFS volume.

     

    As these are large amounts of files in both the locations, I would like to suggest that you perform prestage data before DSFR synchronization to prevent large WAN network traffic with the initial synchronization.

     

    For more information, please refer to:

     

    How to use the Backup program to prestage data before DSFR synchronization in Windows Server 2003 R2

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;947726

    (This should be also applied to Windows Server 2008)

     

    Hope it helps.


    David Shen - MSFT
    Monday, September 22, 2008 6:24 AM