none
Very slow DFSR on Windows Server 2008

    Question

  • We need to have distributed file system between two location.
    I evaluating possibility to use DFS and two Windows Server 2008 to do it.

    I set up two computer with clean installation of system.
    The test is on two local subnets, so speed should not be an issue.
    Each server is defined in separate logical site.

    The DFSR folder I create on one server has 50 GB / 40,000 files.
    The initial replication still is not finished after couple of days.
    The transfer speed is only about 5kB/s and 3 files / min.
    I increase staging quote to 100GB, so it should not be a limit.

    I also make simple test of speed between both servers.
    I create shared folder on both servers and manually copy files.
    The transfer speed is reasonable 5 MB/s (1000 time faster that DFSR).

    There is no processor load on servers (0% CPU).
    This is test before real use so server do almost nothing else.
    The event log shows sometimes 5014 but followed immediately with 5004.

    Does anybody have an idea what is wrong?
    Thursday, May 28, 2009 4:02 PM

Answers

  • Hi Michal88,

    Based on the experience, TCP off loading on the network level may be causing the issue.

    Please add these following registry values to all DFSR replications partners to see it can be helpful.

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

    Value =DisableTaskOffload
    Type = DWORD
    Data = 1

    Value =EnableTCPChimney
    Type = DWORD
    Data = 0

    Value =EnableTCPA
    Type = DWORD
    Data = 0

    Value =EnableRSS
    Type = DWORD
    Data = 0

    Please note: You will need to reboot the server for these changes to take into effect.

    After the reboot monitor replication for the ammount of time to see if the issue can be resolved.

    Hope it helps.

    This posting is provided "AS IS" with no warranties, and confers no rights.
    • Proposed as answer by Ryan Capp Friday, May 29, 2009 8:55 PM
    • Marked as answer by David Shen Saturday, May 30, 2009 8:05 AM
    Friday, May 29, 2009 10:00 AM
  • Hi David,

    You are right, the problem was with TCP off loading.
    Replication now work perfectly.
    Thank you for help.

    • Marked as answer by David Shen Saturday, May 30, 2009 8:06 AM
    Friday, May 29, 2009 5:23 PM

All replies

  • Hi,

    Could it be a problem with your Antivirus software? I have seen that a lot of times.
    Try disabeling AV scan in both ends for a limited time.
    Jens Ole Kragh MCITP, MCTS, MCT http://jensolekragh.spaces.live.com/
    Thursday, May 28, 2009 5:23 PM
  • Thank you for reply.
    This is a evaluation setup, therefore I do not install any other software then OS.
    There is no AV software and the firewall is Windows Firewall.



    Thursday, May 28, 2009 5:34 PM
  • I don't know if you've tried all these tips listed here, but I'll include them anyway:

    1. Missing Windows Server 2003 Network QFE Hotfixes or Service Pack 2 (Not applicable in your case)
    2. Missing DFSR Service’s latest binary
    3. Out-of-date Network Card and Storage drivers
    4. DFSR Staging directory is too small for the amount of data being modified
    5. Bandwidth Throttling or Schedule windows are too aggressive
    6. Large amounts of sharing violations
    7. RDC has been disabled over a WAN link.
    (Not applicable again given your scenario)
    8. Incompatible Anti-Virus software or other file system filter drivers
    9. File Server Resource Manager (FSRM) configured with quotas/screens that block replication.
    10. Un-staged or improperly pre-staged data leading to slow initial replication.


    http://blogs.technet.com/askds/archive/2007/10/05/top-10-common-causes-of-slow-replication-with-dfsr.aspx

    Also check link-negotiation settings on the NICs/switches and make sure they're consistent; full-duplex or auto.
    • Edited by Ryan Capp Thursday, May 28, 2009 9:26 PM
    Thursday, May 28, 2009 9:23 PM
  • Yes, I've looked at this list before I post here.

    ad 1) N/A
    ad 2) The Windows Server 2008 Standard is with SP1 and have all latest updates installed.
    ad 3) I installed both servers two weeks ago and allow to download all available updates.
             Network cards is Broadcom NetXtreme 57xx Gigabit, chipset Intel 925X
    ad 4) I increase staging limit to 100GB (what is more than whole folder I replicate)
    ad 5) No throttling or schedule (set to run all the time with full bandwidth)
    ad 6) It is test so folder is rarely used at all, even as I set couple access accounts
    ad 7) N/A
    ad 8) Not installed
    ad 9) Not installed
    ad 10) I do not pre-stage data to second server.
    ad NIC) all are auto

    regarding 3 & NIC)
    I assume that problems there will effect all communication not only DFSR and file coping test run well.

    Thursday, May 28, 2009 10:23 PM
  • By the way, since you are on gigabit, do you have RDC enabled (as per #7)?

    Thursday, May 28, 2009 10:46 PM
  • The RDC is on for both connections (default value after installation).
    I find some suggestion that it can be turn off to reduce CPU load for LAN, but CPU load is near 0%.
    LAN is also only for test, later connection will be WAN.

    Both computers are on 1Gb LAN, but subnets are connected with 100Mb router.
    This is consistent with copy test 5MB -> ~50Mb (50% of bandwidth).

    Thursday, May 28, 2009 11:31 PM
    • Proposed as answer by kspartner Wednesday, October 01, 2014 8:18 PM
    Thursday, May 28, 2009 11:40 PM
  • Hi Michal88,

    Based on the experience, TCP off loading on the network level may be causing the issue.

    Please add these following registry values to all DFSR replications partners to see it can be helpful.

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

    Value =DisableTaskOffload
    Type = DWORD
    Data = 1

    Value =EnableTCPChimney
    Type = DWORD
    Data = 0

    Value =EnableTCPA
    Type = DWORD
    Data = 0

    Value =EnableRSS
    Type = DWORD
    Data = 0

    Please note: You will need to reboot the server for these changes to take into effect.

    After the reboot monitor replication for the ammount of time to see if the issue can be resolved.

    Hope it helps.

    This posting is provided "AS IS" with no warranties, and confers no rights.
    • Proposed as answer by Ryan Capp Friday, May 29, 2009 8:55 PM
    • Marked as answer by David Shen Saturday, May 30, 2009 8:05 AM
    Friday, May 29, 2009 10:00 AM
  • Thank you for your help.

    The replication process just stopped.
    This morning target folder size is exactly same as yesterday.
    Previously it at least slowly increase.

    I looked at those log files, but I still need to understand them.
    The log level is default (means 4) and there is many entries there.
    Anyway I attached examples, while searching what it their meaning.

    The source server:

    Many warning looking like this:
    20090529 08:30:34.122 2772 SRTR  2344 [WARN] InitializeFileTransferAsyncState::ProcessIoCompletion Failed connId:{} rdc:0 uid:{}-v74162 gsvn:{}-v74162 Error:  [Error:170(0xaa) InitializeFileTransferAsyncState::ProcessIoCompletion servertransport.cpp:2235 2772 W The requested resource is in use.] completion:0 ptr:09B27F40

    From time to time errors sequence:
    20090529 08:33:33.086 2608 DOWN  3991 [ERROR] DownstreamTransport::EstablishSession Failed on connId:{} csId:{} rgName:Common Volume Error:
    +    [Error:9027(0x2343) DownstreamTransport::EstablishSession downstreamtransport.cpp:3984 2608 C A failure was reported by the remote partner]
    +    [Error:9051(0x235b) DownstreamTransport::EstablishSession downstreamtransport.cpp:3984 2608 C The content set is not ready]
    20090529 08:33:33.086 2608 INCO  3566 InConnection::RestartSession Retrying establish contentset session. connId:{} csId:{} csName:common
    20090529 08:33:33.086 2608 INCO   774 [WARN] SessionTask::Step (Ignored) Failed, should have already been processed. Error:
    +    [Error:9027(0x2343) InConnection::EstablishSession inconnection.cpp:3657 2608 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) DownstreamTransport::EstablishSession downstreamtransport.cpp:4005 2608 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) DownstreamTransport::EstablishSession downstreamtransport.cpp:3984 2608 C A failure was reported by the remote partner]
    +    [Error:9051(0x235b) DownstreamTransport::EstablishSession downstreamtransport.cpp:3984 2608 C The content set is not ready]

    The target server:

    20090529 08:31:33.054 3196 SRTR   784 [WARN] SERVER_EstablishSession Failed on connId:{} csId:{} Error:
    +    [Error:9051(0x235b) UpstreamTransport::EstablishSession upstreamtransport.cpp:707 3196 C The content set is not ready]
    +    [Error:9051(0x235b) OutConnection::EstablishSession outconnection.cpp:2623 3196 C The content set is not ready]
    20090529 08:31:33.974 1664 DOWN  2723 AsyncRpcHandler::ProcessReceive Completion. connId:{} csId:{} reqType:AsyncPollRequest reqState:Completed status:9026 ptr:056820B0
    20090529 08:31:33.974 1664 DOWN  2748 [ERROR] AsyncRpcHandler::ProcessReceive Failed on connId:{} csId:{} reqType:AsyncPollRequest reqState:Completed status:9026 Error:
    +    [Error:9027(0x2343) AsyncRpcHandler::ReceiveAsyncPoll downstreamtransport.cpp:2131 1664 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) AsyncRpcHandler::ReceiveAsyncPoll downstreamtransport.cpp:2073 1664 C A failure was reported by the remote partner]
    +    [Error:9026(0x2342) AsyncRpcHandler::ReceiveAsyncPoll downstreamtransport.cpp:2073 1664 C The connection is invalid]
    20090529 08:31:33.974 1664 INCO  2543 InConnection::ReConnect Ignoring reconnect. state:4 connId:{}
    20090529 08:31:33.974 1664 RPCN   663 [ERROR] IoPortManager::ThreadEntryProc (Ignored) Failed IoCompletionProcessVvUp. Error:
    +    [Error:9027(0x2343) AsyncRpcHandler::ProcessReceive downstreamtransport.cpp:2809 1664 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) AsyncRpcHandler::ReceiveAsyncPoll downstreamtransport.cpp:2131 1664 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) AsyncRpcHandler::ReceiveAsyncPoll downstreamtransport.cpp:2073 1664 C A failure was reported by the remote partner]
    +    [Error:9026(0x2342) AsyncRpcHandler::ReceiveAsyncPoll downstreamtransport.cpp:2073 1664 C The connection is invalid]

    20090529 08:30:34.118 2588 INCO  2705 InConnection::ProcessErrorStatus (Ignored) Remote error connId:{} Error:
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4768 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) RpcFinalizeContext downstreamtransport.cpp:1096 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4711 2588 C A failure was reported by the remote partner]
    +    [Error:170(0xaa) DownstreamTransport::RawGet downstreamtransport.cpp:4711 2588 W The requested resource is in use.]
    20090529 08:30:34.118 2588 MEET  2032 Meet::Download Download Succeeded : false updateName:Tutorial_V2_Gerotor_Surfaces_html_6de9f06.png uid:{}-v74168 gvsn:{}-v74168 connId:{} csName:common csId:{}
    20090529 08:30:34.118 1920 INCO  5599 InConnection::LogTransferActivity Failed to receive RAWGET uid:{}-v74169 gvsn:{}-v74169 fileName:Tutorial_V2_Gerotor_Surfaces_html_7153b1c3.png connId:{} csId:{} stagedSize:0 Error:
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4768 1920 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) RpcFinalizeContext downstreamtransport.cpp:1096 1920 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4711 1920 C A failure was reported by the remote partner]
    +    [Error:170(0xaa) DownstreamTransport::RawGet downstreamtransport.cpp:4711 1920 W The requested resource is in use.]
    20090529 08:30:34.118 1920 INCO  2705 InConnection::ProcessErrorStatus (Ignored) Remote error connId:{} Error:
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4768 1920 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) RpcFinalizeContext downstreamtransport.cpp:1096 1920 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4711 1920 C A failure was reported by the remote partner]
    +    [Error:170(0xaa) DownstreamTransport::RawGet downstreamtransport.cpp:4711 1920 W The requested resource is in use.]
    20090529 08:30:34.118 2588 INCO  2705 InConnection::ProcessErrorStatus (Ignored) Remote error connId:{} Error:
    +    [Error:9027(0x2343) Meet::InstallStep meet.cpp:1657 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) Meet::Download meet.cpp:2052 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) InConnection::RawGet inconnection.cpp:5688 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4768 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) RpcFinalizeContext downstreamtransport.cpp:1096 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4711 2588 C A failure was reported by the remote partner]
    +    [Error:170(0xaa) DownstreamTransport::RawGet downstreamtransport.cpp:4711 2588 W The requested resource is in use.]
    20090529 08:30:34.118 1920 MEET  2032 Meet::Download Download Succeeded : false updateName:Tutorial_V2_Gerotor_Surfaces_html_7153b1c3.png uid:{}-v74169 gvsn:{}-v74169 connId:{} csName:common csId:{}
    20090529 08:30:34.118 2588 MEET  1263 Meet::Install -> WAIT Error processing update. updateName:Tutorial_V2_Gerotor_Surfaces_html_6de9f06.png uid:{}-v74168 gvsn:{}-v74168 connId:{} csName:common csId:{} code:170 Error:
    +    [Error:9027(0x2343) Meet::InstallStep meet.cpp:1657 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) Meet::Download meet.cpp:2052 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) InConnection::RawGet inconnection.cpp:5688 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4768 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) RpcFinalizeContext downstreamtransport.cpp:1096 2588 C A failure was reported by the remote partner]
    +    [Error:9027(0x2343) DownstreamTransport::RawGet downstreamtransport.cpp:4711 2588 C A failure was reported by the remote partner]
    +    [Error:170(0xaa) DownstreamTransport::RawGet downstreamtransport.cpp:4711 2588 W The requested resource is in use.]
    Friday, May 29, 2009 2:13 PM
  • Thank you for support.I applied those changes to registry and will monitor replication to see if it help.

    Friday, May 29, 2009 3:32 PM
  • Hi David,

    You are right, the problem was with TCP off loading.
    Replication now work perfectly.
    Thank you for help.

    • Marked as answer by David Shen Saturday, May 30, 2009 8:06 AM
    Friday, May 29, 2009 5:23 PM
  • I try to find better driver for network adapter, than one that came with Windows.
    Dell does not have one (both computers are Dell Dimension), but I find one directly at Broadcom.
    The updated driver from Broadcom works fine also with TCP off loading.

    Thank you to all who answer for help.
    Michal
    Friday, May 29, 2009 5:36 PM
  • HI Michal88

    I do the same and the result was on older Broadcom NIC Bluescreen after connect to a share. Newer Card ar running fine with the Broadcom Driver. So please be carefull with this Drivers on Dell Server.

    Old Server was Dell Blade Server PE1955

    New Server is a PE R410

    OS is Server 2008 R2

     

    Regards

    René

     

     

     

     

     


    Roendi
    Tuesday, March 23, 2010 10:07 AM
  • Hello David,

    Are the registry entries the same if on a 64 bit server?  Thank you.

    Wednesday, June 23, 2010 10:25 PM
  • Hi Michal88:

    Copying files from server 2008 would take hours what normaly takes seconds or minutes.

    Your post fixed my problem. You are a genius.

    Thank you so much.

    Thursday, April 28, 2011 3:40 PM
  • Thank you! You saved my day! :)
    Diogo Rodrigues
    Friday, May 27, 2011 9:32 PM
  • You sir hit the nail on the head.  Thank you so much for posting this resolution.
    Friday, April 13, 2012 4:44 PM
  • is there any NetSH command that can do similar thing ?

    /* Server Support Specialist */

    Monday, July 07, 2014 3:08 PM
  • Does this methods also applicable to the Windows Server 2008 Enterprise SP2 as Virtual Machine ?

    /* Server Support Specialist */

    Monday, July 07, 2014 3:10 PM