locked
Robocopy Unbuffered I/O RRS feed

  • Question

  • Greetings,

    I have 1,000 files totalling 200 GB. It used to take Robocopy 20 minutes to copy the files, but over the last several months, it has been taking longer. It now takes about five hours.

    I thought I read using Robocopy's multi-threaded copies (/mt) doesn't work well with large files. I was thinking about using unbuffered I/O (/j), but I can't find any documentation about it. Does anyone know how it works? Is there any risk of data loss or corruption using it?

    Thanks,
    Shane.


    Windows 10 Home 64-bit


    • Edited by cShane Tuesday, November 14, 2017 11:14 AM fixed link
    Tuesday, November 14, 2017 11:09 AM

Answers

  • Hi,

    Unbuffered I/O is a simple file copy from a source location to a destination location. Buffered I/O augments the simple copy to optimize for future reads of (and writes to) the same file by copying the file into the filesystem cache, which is a region of virtual memory. Buffered I/O incurs a performance penalty the first time the file is accessed because it has to copy the file into memory; however, because memory access is faster than disk access, subsequent file access should be faster. The operating system takes care of synchronizing file writes back to disk, and reads can be pulled directly from memory.

    The usage note mentions large files vis-à-vis buffered I/O because:

    1.The up-front cost is expensive. The performance penalty with buffered I/O is substantially worse for large files.

    2.You get little in return. Large file blocks don't tend to stay in the cache for very long anyway, unless you have a ton of memory relative to the file size.

    3.It may not avoid disk I/O. Reads and write of large file data blocks increase the probability of requiring disk I/O.

    4.You probably don't need to buffer anyway. Large files tend to be less frequently accessed in practice than smaller files.

    You could get more information in the link below about How to: Copy very large files across a slow or unreliable network.

    https://blogs.msdn.microsoft.com/granth/2010/05/10/how-to-copy-very-large-files-across-a-slow-or-unreliable-network/

    Also see Microsoft's Ask The Performance Team Blog for more.

    https://blogs.technet.microsoft.com/askperf/2007/05/08/slow-large-file-copy-issues/

    Hope it will be helpful to you


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, November 15, 2017 7:00 AM

All replies

  • Hi,

    Unbuffered I/O is a simple file copy from a source location to a destination location. Buffered I/O augments the simple copy to optimize for future reads of (and writes to) the same file by copying the file into the filesystem cache, which is a region of virtual memory. Buffered I/O incurs a performance penalty the first time the file is accessed because it has to copy the file into memory; however, because memory access is faster than disk access, subsequent file access should be faster. The operating system takes care of synchronizing file writes back to disk, and reads can be pulled directly from memory.

    The usage note mentions large files vis-à-vis buffered I/O because:

    1.The up-front cost is expensive. The performance penalty with buffered I/O is substantially worse for large files.

    2.You get little in return. Large file blocks don't tend to stay in the cache for very long anyway, unless you have a ton of memory relative to the file size.

    3.It may not avoid disk I/O. Reads and write of large file data blocks increase the probability of requiring disk I/O.

    4.You probably don't need to buffer anyway. Large files tend to be less frequently accessed in practice than smaller files.

    You could get more information in the link below about How to: Copy very large files across a slow or unreliable network.

    https://blogs.msdn.microsoft.com/granth/2010/05/10/how-to-copy-very-large-files-across-a-slow-or-unreliable-network/

    Also see Microsoft's Ask The Performance Team Blog for more.

    https://blogs.technet.microsoft.com/askperf/2007/05/08/slow-large-file-copy-issues/

    Hope it will be helpful to you


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, November 15, 2017 7:00 AM
  • Hi Shane,

    Haven't received your message a few days, was your issue resolved? 
    I am proposing previous helpful replies as "Answered". Please feel free to try it and let me know the result. If the reply is helpful, please remember to mark it as answer which can help other community members who have same questions and find the helpful reply quickly.
    Best regards,
    Carl 


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Saturday, November 18, 2017 9:37 AM
  • Hi Shane,

    Any update?


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Tuesday, November 21, 2017 12:16 PM
  • I'm really sorry about the late reply.

    Robocopy's unbuffered I/O (/j) seems to be faster.

    I have questions about multi-threaded copies (/mt). Should I start a new thread?

    Thanks,
    Shane.


    Windows 10 Home 64-bit

    Sunday, December 17, 2017 9:14 AM
  • Hi cShane,

    It’s OK. Yes, Robocopy's unbuffered I/O (/j) could be faster. It can improve some performance.

    If you have other question, to help you better, I suggest that you start a new thread.

    Regards,

    Carl


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Monday, December 18, 2017 1:29 AM
  • I started a new thread.

    Thanks,
    Shane.


    Windows 10 Home 64-bit

    Monday, December 18, 2017 9:02 AM