none
Poor File Copying Speeds

    Question

  • A department within our organization frequently has to copy large amounts of data (via robocopy & robocopy gui) from a network share onto a local machine or directly onto a USB device (thumb drive or external hard drive).  Usually it goes from Network Share to USB Drive to avoid a second hop.

    In the last few months they've complained more & more about file copy speeds as its gotten progressively worse.  It'll start off super quick, copying 1GB mere moments & ultimately slowing down to 1GB per day.  Obviously this not ideal when dealing with several hundred GB's (usually in the 300-750 range) to a few TB's (usually not more than 3).  The files themselves are relatively small averaging about 75KB in size: Some are 13KB or smaller, others are as much as 1 or 2MB.  Rebooting doesn't always fix the issue and after reimaging the machine it seems to work fine for a few days (sometimes more, sometimes less) then over time, copy speeds get slower & slower until its like 1KB/sec and its permanently in this odd broken state.

    The desktops in question all have gigabit Ethernet ports as are the Cisco switches they're directly connected to.  Our networking team has apparently ruled out any network foul play so they're looking at the workstations.  I'm open to the possibility that its a driver issue of some sort but where:

    • NIC?
    • BUS or Chipset?
    • USB Device?

    I'm looking for an intelligent way to troubleshoot this; preferably relying on a utility to test read/write & throughput speeds for the NIC, BUS & USB Device.

    What's the ideal way to handle something like this?


    • Edited by JuliusPIV Friday, March 16, 2012 2:41 PM
    Tuesday, March 13, 2012 1:38 PM

All replies

  • There's a hotfix for that.

    I/O throughput is low when large files are read sequentially in Windows 7 or in Windows Server 2008 R2

    http://support.microsoft.com/kb/2564236

    Wednesday, March 14, 2012 2:36 PM
  • Hey JS2010

    Thanks for taking the time to review & research this with me. I'm running through a series of tests now to get a baseline then will explore applying that patch.

    What's interesting is that nearly all of the files are under 1MB in size.  On average, the files are about 75KB; some smaller (around 13KB) and some larger (a few hundred KB to a few MB's).  It takes a terrible amount of time just copying 6.1GB via robocopy (all default settings + logging).

    Wednesday, March 14, 2012 6:21 PM
  • Hi,

    Please try to copy the files to local computer instead of USB drive. If the issue persists, it may be caused by your network environment. Please contact Cisco.

    Best Regards,

    Kim Zhou


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Thursday, March 15, 2012 9:08 AM
  • Again, thanks to all for responding so far.

    I'm trying to test this as intelligently and methodically as possible, so I wanted to start off with building a baseline:

    1. Throughput between my local machine & the network share
    2. Read/write baseline for the network share (from my local machine)
    3. Read/write baseline for my local disk (of course, from my local machine)
    4. Read/write baseline for the USB hard drive (again, from my local machine)
    5. Files copied from the network to the local disk
    6. Files copied from the local disk to the USB hard drive
    7. Files copied from the network to the USB hard drive

    I won't bore you with all the calculations but I might save it to my skydrive and post a link here in case someone is interested.  At the moment I still haven't finished running through the tests, but already I can see where the bottle neck appears to be.  I'm using a static data set of 6.1GB (101,101 files roughly 75KB on average, although some are considerably larger & smaller) for all my tests and I'm copying it 20 times (two sets of 10) recording the results after each set.  Here's what I have so far:

    • Takes just under 20 minutes to copy from a network share to my local machine at a rate of 382MB/min.
    • Takes just over 54 minutes to copy from my local machine to a USB hard drive at a rate of 44MB/min.

    The next step are as follows:

    1. Perform the 'network share to USB hard drive copy test' (20 times of course)
    2. Drop in a USB 3.0 card, use a USB 3.0 hard drive & retest local disk to USB and network share to USB
    3. Apply the patch JS2010 suggested, KB2564236, although it doesn't necessarily apply here since these are all large quantities of small files & retest network share to local disk.  If there's a significant difference in results, I'll then do network share to USB and network share to USB 3.0.

    I've used a variety of tools such as LAN SpeedTestCrystalDiskMark, and my personal favorite robocopy (console of course, defaults + logging).  I'm open to using other tools as well, so drop your recommendations.


    • Edited by JuliusPIV Friday, March 16, 2012 2:37 PM
    Friday, March 16, 2012 11:34 AM
  • Hi Kim - thanks for the reply!

    I've done that, and am still doing this as part of building a baseline.  However, having two hops (network -> PC, then PC -> USB) is not efficient nor an effective long term solution.  (read: so that can't be 'the way' unless it somehow proves to be more efficient)

    Friday, March 16, 2012 11:37 AM
  • I started a new copy loop sometime on the 17th to copy the same set of static data 10 times from my local disk to the USB hard drive.  The first two iterations were 'snappy', completing in about 53 minutes (2.24 MBytes/sec or 134MBytes/min).  Everything beyond the second iteration took progressively longer to complete the file copy:

    • 3rd Iteration: 1h17m
    • 4th Iteration: 3h21m
    • 5th Iteration: 3h55m
    • 6th Iteration: 6h16m
    • 7th Iteration: 10h42m

    When I arrived this morning it was still on the 8th iteration (so it never finished).  I stopped the copy process & restarted it & its still just as slow.

    I copied a gigantic file (8.8GB) from the network to my local PC then copied that same file from the local PC to the USB drive and that was pretty quick too.  Both transfers were several hundred MBytes/min per robocopy so nice and quick.  Seeing good results, I restarted the copy process of the static data set from the local PC to the USB drive and it was super slow again. 

    At this point I'm convinced the network isn't the cause or otherwise contributing to the slowness, so I won't be testing that moving forward.

    So here we are: It appears the latency is between the local PC and the USB hard drive.

    Monday, March 19, 2012 4:16 PM
  • I rebooted the machine and have tested copying 1.59MB of data (46 files) once onto 5 different USB devices.

    1. 60GB 7200RPM ATA Seagate Barracuda*: 51 secs @ 32539 Bytes/sec (1.861 MBytes/min)
    2. 320GB Western Digital My Book Premium: 42 secs @ 39907 Bytes/sec (2.283 MBytes/min)
    3. 80GB 5400RPM SATA Hitachi*: 27 secs @ 62272 Bytes/sec (3.563 MBytes/min)
    4. 500GB Western Digital My Passport Essential: 21 secs @ 78450 Bytes/sec (4.488 MBytes/min)
    5. 16GB Kingston Data Traveler thumb drive: 57 secs @ 29206 Bytes/sec (1.671 MBytes/min)

    * - These were internal hard drives connected to an IDE/SATA to USB adapter then connected to the PC.

    Comparatively, running the same copy on a machine that doesn't have this slowness issue takes almost no time (less than 1 second) with a transfer rate of around 4776300 Bytes/sec (273 MBytes/min).

    Monday, March 19, 2012 6:11 PM
  • Hi,

    Based on my experience and knowledge, the possible causes for this issue would be:

    - Driver issue

    - Performance issue

    - Other issue

    So firstly, please check if there are any disk controller driver and RAID driver update from your hardware vendor, please update them to the latest.

    Best Regards,

    Ruby Cheng


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Tuesday, March 20, 2012 9:23 AM
  • Ruby Cheng: Thanks for the reply.  In general I agree with your three points, that's usually the case for any problem.  ;)  In my humble opinion, I struggled with determining the best course of action for troubleshooting & measuring results in situations like this.  This was further complicated with the introduction of different machine models that have the same problem.  Nevertheless, I checked the Dell website for updated drivers (chipset & storage) as well as a BIOS update.  Their storage drivers were from 2008 (yikes!) so initially I used only the chipset drivers from mid-2011 (although I can't say with certainty I didn't already have these drivers loaded) and updated the BIOS.  These didn't bring me closer to a solution so I decided to try the older storage drivers but they didn't like the OS.

    When I initially began testing this, I hit up forums & search engines looking for evidence of a potential widespread issue as well as evidence of misconfiguration or some feature that may have been enabled or disabled that could influence this.  This included things like:

    I didn't want to dive into that for fear of muddying the waters so I took more basic troubleshooting approach.  Having completed several rounds of copying, I felt comfortable applying the hotfix mentioned above.  However the copy results haven't improved enough (if any) so I'm going down other avenues.


    Any advice is greatly appreciated.

    Tuesday, March 20, 2012 12:14 PM
  • Hi,

    To troubleshoot this slow copy issue, we need to know the following information:

    1. Does this problem can always be reproduced or just happen randomly?

    2. Which kind of files that you would like to copy to the new file server? Please try to copy a txt file less than 20KB and check if the issue persists.

    3. Please let me know the whole robocopy command that you use.

    Best Regards,

    Ruby Cheng


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Wednesday, March 21, 2012 9:58 AM
  • Hey Ruby Cheng thanks again for the follow up!

    The department that is reaching out to me has reproduced this on several machines down there a number of times.  I've been able to reproduce it once, mainly because I've only tried to reproduce it once.  I've wiped my test machine and will start over again today.  It'll probably take a few days to reproduce just because it only seems to happen after a certain threshold has been reached/exceeded.

    The files I'm copying are mostly .TIF files, but there's a handful of .TXT files in the mix as well, and they're all pretty small; There's just a large number of files (over 100K).  This image below is a screenshot from one of the directories that has 46 files and is only 1.6MB in size. (The one I used for testing a few posts above.)

    Just a reminder: The file copy slowness happens when:

    1. Copying files from the local machine to a USB hard drive
    2. Copying files from a network share to the same USB hard drive

    The robocopy command is very simple:

    • robocopy /e "source" "destination" /LOG+:"path\to\log.file" /eta /TEE /NP

    Wednesday, March 21, 2012 1:32 PM
  • Hi,

    I understand it is a randomly occur issue and it takes a few days to reproduce "because it seems to happen after a certain threshold has been reached/exceeded." Generally, we can get a robocoy output log to check what's going on during the copy.

    Action Plan

    =======
    1. Reduce the retry limit and/or retry time, so that Robocopy won’t retry forever.  
     
    E.g. /R:6 /W:20 will stop Robocopy retrying after a couple of minutes. To make the chose values the default for Robocopy jobs, run a small job with /REG to store these chosen values in the registry.
     
    E.g. ROBOCOPY . . /R:6 /W:20 /REG
     
    This will cause Robocopy jobs to fail after a period of time, instead of retrying forever.
     

    2.  Redirect Robocopy output to a log file by adding /LOG:<logFilePath> to the command line. This may give us an easier way of seeing if Robocopy is getting errors, and which ones, by examining the log file.

    Best Regards,

    Ruby Cheng


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Thursday, March 22, 2012 4:16 PM
  • I copied 6 small text files all under 20KB, the largest being 19348 bytes.  It took 1 full second to copy all 6 files at a rate of 21628 Bytes/sec or 1.237 MBytes/min.  That should have taken less than 1 second - normally a copy that small is instantaneous and the transfer rate is super high.

    I want to point out that the robocopy process isn't failing!  At no point is it ever stuck reading or writing one or more files.  So it doesn't matter if I set /R:0 /W:0, it still takes a long time.

    Also I am logging to a separate log file each time I run a copy so I can review the output to make sure its not running into some crazy error.  Again: Even if I copy 46 files (about 1.6MB) in size, it takes way longer than it should.  If you want to see a log file, I'll make one available.

    Friday, March 23, 2012 5:45 PM
  • Hi,

    After clarifying this issue, we can rule out the possibility of driver issue, network issue or robocopy issue.

    Your question falls into the paid support category which requires a more in-depth level of support. To troubleshooting the performance issue, we may need to collect a Performance monitor log for analysis. Please visit the below link to see the various paid support options that are available to better meet your needs. http://support.microsoft.com/default.aspx?id=fh;en-us;offerprophone

    Best Regards,

    Ruby Cheng


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Monday, March 26, 2012 6:56 AM
  • I've opened an incident with premier support - we'll see what they say.
    Tuesday, March 27, 2012 3:32 PM