locked
copy & file-compare to multiple destinations at once? RRS feed

  • Question

  • I have a backup routine for my virtual-machine-based server that copies the virtual drives to another drive in the same computer and over a gigabit network to another computer that is capable of running the server VM.

    My routine consists of an "Xcopy /j" (unbuffered copy) and an "FC" file compare to make sure the copy worked. Since I have two destinations, I now run Xcopy and FC for one destination, then another Xcopy and FC for the other destination. It takes rather long (160GB of data).

    Basically it's like this

    Xcopy /j source destination1
    FC source destination1
    Xcopy /j source destination2
    FC source destination 2
    The data is pulled 4 times, once for each Xcopy, and once for each FC.

    Is there a way to pull chunks of data from the source once, then copy & compare to multiple destinations without having to do a separate copy for each destination, like this?

    do
      read a chunk from source into a variable
      write variable contents to destination1 and destination2
      compare the variable contents to new chunks in destination1 and destination 2
      if the compare is bad, try it again
    loop until source is read through completely

    I'm familiar with VBS scripts and batch files, but I'll try anything else that might work. (I have tried rsync, but that takes even longer than a direct Xcopy in my tests.)
    Wednesday, March 21, 2012 1:41 PM

Answers

  • Here is your data chunker on steriods:

    FRS Technical Reference

    File Replication service (FRS) is a technology that replicates files and folders stored in the SYSVOL shared folder on domain controllers and Distributed File System (DFS) shared folders. When FRS detects that a change has been made to a file or folder within a replicated shared folder, FRS replicates the updated file or folder to other servers. Because FRS is a multimaster replication service, any server that participates in the replication of a shared folder can generate changes. In addition, FRS can resolve file and folder conflicts to make data consistent among servers.

    http://technet.microsoft.com/en-us/library/cc759297(v=ws.10).aspx


    ¯\_(ツ)_/¯

    • Marked as answer by Bill_Stewart Thursday, March 22, 2012 6:31 PM
    Thursday, March 22, 2012 6:27 PM

All replies

  • I have a backup routine for my virtual-machine-based server that copies the virtual drives to another drive in the same computer and over a gigabit network to another computer that is capable of running the server VM.

    My routine consists of an "Xcopy /j" (unbuffered copy) and an "FC" file compare to make sure the copy worked. Since I have two destinations, I now run Xcopy and FC for one destination, then another Xcopy and FC for the other destination. It takes rather long (160GB of data).

    Basically it's like this

    Xcopy /j source destination1
    FC source destination1
    Xcopy /j source destination2
    FC source destination 2
    The data is pulled 4 times, once for each Xcopy, and once for each FC.

    Is there a way to pull chunks of data from the source once, then copy & compare to multiple destinations without having to do a separate copy for each destination, like this?

    do
      read a chunk from source into a variable
      write variable contents to destination1 and destination2
      compare the variable contents to new chunks in destination1 and destination 2
      if the compare is bad, try it again
    loop until source is read through completely

    I'm familiar with VBS scripts and batch files, but I'll try anything else that might work. (I have tried rsync, but that takes even longer than a direct Xcopy in my tests.)

    This is a scripting forum.  What you are asking can only be done in compiled code like C or C#.

    For multiple copies a far better way to do this is with either RoboCopy using mirrored copies.  This will only update what has changed and it also does an  internal verify. It is very fast.

    For multiple targets consider setting up BITS Server Service and subscribing to a download from each of the target machines.  This can also be set up to do updates only.

    For you I suspect that RoboCopy is the better, more efficient and easier solution.

    I don't understand why you are doing an FC.  XCOPY does that internally and will return an error on any failed read or write.  No error then all files are complete.

    unbuffered copy is only rally useful for extremely large files.

    Note that RoboCopy is like XCOPY on steroids.  It was written to handle large scenarios just like yours.

    Her is yet another way to copy large file but it is really only useful for very large files.  If you are just copying a lot of small files (under 100Mb) then this si not a good solution:
    http://blogs.msdn.com/b/granth/archive/2010/05/10/how-to-copy-very-large-files-across-a-slow-or-unreliable-network.aspx

    It is from the Microsoft Performance Team.


    ¯\_(ツ)_/¯

    Wednesday, March 21, 2012 7:10 PM
  • Thanks, jrv, for the info.

    (Should've said I was running Windows 7 64-bit as the host machine's OS...)

    I did look at robocopy, but i could not find anything that indicates robocopy does an unbuffered copy of large files. Robocopy is built-in on Windows 7 and there's no switch (that I saw, at least) to make it work unbuffered, like Xcopy has. (I don't know if this means Robocopy automatically switches to unbuffered when needed, or it doesn't do unbuffered, though.)

    I had also seen Grant Holliday's ESEUTIL post before, and the Technet post it was based on (http://blogs.technet.com/b/askperf/archive/2007/05/08/slow-large-file-copy-issues.aspx). I used ESEUTIL successfully on the old 32-bit server before we moved up to the 64-bit monster, and it worked very well. I'm still running the old 32-bit SBS virtual machine on the new Windows 7 64-bit server box, though, and I don't have access to 64-bit Exchange Server (yet, until we upgrade to SBS2011 in another year), so I can't try the 64-bit ESEUTIL.

    I also looked into Xcopy regarding whether it verifies its copy. There was a /v switch, but the info from the command-prompt "xcopy /?" says it only verifies the file size, not the contents, and is apparently there only for compatibility with older scripts. I assume (perhaps incorrectly) that this means Xcopy doesn't verify if the /v switch isn't there. The reason I suspect Xcopy's ability to verify the destination is:

    I have monitored the I/O read and write bytes in task manager when Xcopy was running, vs. when FC was running. When Xcopy does its thing, the I/O read bytes and write bytes start increasing, showing the same number in each column up until the total bytes of the file, then Xcopy closes and disappears off Task Manager. When FC runs, the I/O write bytes have a little spike of a few bytes, then the read bytes shoot up to twice the size of the file before FC quits. It seems Xcopy is just reading the source file and writing to the destination, whereas FC is reading both the source and destination. I would expect that Xcopy would at least have to read the destination file back in at some time to see if the destination computer actually wrote and can read the data correctly, thus making the I/O read bytes number twice the write bytes number, one set of reads to get the source, one set of writes to copy the data, and another set of reads to check the destination. (As I recall, ESEUTIL also did the same as Xcopy, regarding the I/O read and write bytes, same value for both when it finished, seemingly indicating no comparing of source and destination.)

    Are you sure Xcopy verifies the contents of the copied file, not just the file size?

    I never did get into C or its derivatives, regrettably. I'll have to dig into that for a custom utility...


    • Edited by ScottGus1 Thursday, March 22, 2012 1:36 PM edits
    Thursday, March 22, 2012 12:10 PM
  • This seems more like a general Windows OS issue and not really a scripting question.

    Bill

    Thursday, March 22, 2012 3:30 PM
  • Possibly, just asking if there was a scripting way to do it. I hadn't heard of a way right off, although I have picked up something about using an ADO object to read a file binary-wise. Trying some experiments...

    Meanwhile, can anyone take a look into whether XCOPY does compare the contents of a copied file to see whether it was copied and is readable correctly, like FC does?

    Thursday, March 22, 2012 4:05 PM
  • Possibly, just asking if there was a scripting way to do it. I hadn't heard of a way right off, although I have picked up something about using an ADO object to read a file binary-wise. Trying some experiments...

    Meanwhile, can anyone take a look into whether XCOPY does compare the contents of a copied file to see whether it was copied and is readable correctly, like FC does?

    On NTFS volumes file creation and writing is transacted.  Files never get damaged in a copy.  They get damaged in use.  FC is a legacy tool used to compare files sets on DOS and has been maintained for other similar uses.

    XCOPY is the chosen method of Net installation and can build huge files sets for thigs likke web sites.  Microsft XCPOY installs bever do a verification.

    As noted earlier.  unbuffered copies are unnexessary except if you are low on memory.  Windows will choose the number of buffers and manage them.  It seldom has an issue.  Robocpoy will adjust its buffers dynamically and may unbuffer on very large files although it chould not be necessary.

    You are making too much of a project of something that I have done for more than a decade without ever seeing an issue.  Much of the info yu have is legacy pre-W2k when we had memory issue with NT.  XP and later do not have memory issues.

    As Bill has pointed out.  This is a discussion for the Windows OS forum for the OS you are using.  I think you might find moore info on RoboCopy in the Windows 7 forum.


    ¯\_(ツ)_/¯

    Thursday, March 22, 2012 4:17 PM
  • Meanwhile, can anyone take a look into whether XCOPY does compare the contents of a copied file to see whether it was copied and is readable correctly, like FC does?

    Hi,

    This sounds like a request for volunteers to go do research for you. Some might construe this as a bit rude. Please remember that this forum is moderated by busy volunteers who have full-time jobs. We (with the exception of the Scripting Guy) are not paid, are not Microsoft employees, and we have the same access to the Microsoft documentation that you have.

    Bill

    Thursday, March 22, 2012 4:19 PM
  • This sounds like a request for volunteers to go do research for you. Some might construe this as a bit rude. Please remember that this forum is moderated by busy volunteers who have full-time jobs. We (with the exception of the Scripting Guy) are not paid, are not Microsoft employees, and we have the same access to the Microsoft documentation that you have.

    Bill,

    Hoity-toity. If you've read my first respone to JRV's very knowledgeable (seriously, no sarcasm here) answer, I have done a considerable amount of research on this question, both through web searches and monitoring my own results with Task Manager. JRV, who has helped me much in the last coulple of months in some (to me) thorny issues I had, reported an unexpected capability in Xcopy that I had thought, based on my own research on the web, didn't exist. I was just trying to see if there was further info on the subject.

    Some might construe as a bit rude your implcation that I'm sponging off others without trying to do my own research. Read more carefully next time and you'll see that I'm not trying to do this. Stand down, brother...

    JRV,

    Naturally, I appreciate and have appreciated all the help you have taken your personal time to provide. I do hope you weren't insulted by my double-checking on the Xcopy verify question. It's just a new piece of info which, as I explained to "friend" Bill above, I had seen differently in my own research. When you said, "I don't understand why you're doing an FC. Xcopy does that internally", I understood you knew something that the folks who write the /? help for Xcopy didn't know. Maybe I misunderstood...

    I of course had no idea how long you've been in this field. (You of course don't know how long I've been in it either. :) Much longer than  the couple months I've been posting here.)

    But despite the improvements in Windows' memory usage capabilities, a search for "windows 7 copy large file memory" on Google will reveal some folks having memory issues with recent (7 & 2008R2) MS OS's when copying large files. Same issues I was having. And I have 48 gigbytes of memory to play with. Buffered copies of my two virtual disk files totalling 160GB (30GB and 130GB) kill the server. Unbuffered "Xcopy /J" doesn't.

    But Xcopy says in the /? help and in various places on the web it doesn't verify the contents of the copied file, only the size. I really need to know the copy worked and that nowhere in the movement of data was there a failure. This is my boss's business I'm supporting. Thus the FC.

    However, the Xcopy and FC take some time to complete. Last night's two-destination backup to a local drive in the server and to a gigbit-networked PC started at 11:35 PM and finished at 5:16 AM this morning. I'd like to cut that down a little. This is why I'm 'making too much of a project of something you've done for more than a decade.' 

    So I had a thought that maybe there was a command for multi-destination copying and comparing that I hadn't heard of, in my ten years of writing batch files, VBS's, and HTA's for my home network, and my home and work $150,000 CNC cutting robots, as well as for my boss's network , for whom I'm the IT guy. Just like that Date-Converter command you helped me with, when WMI started reporting the event times strangely....

    I get it that there isn't such a command for what I want.

    Again, thank you very much for the help you've given.

    To all:

    I have noted that there isn't one command that will do what I want and I'll have to get into C. In my own research I have also found a way to get VBS to read a file by bytes, not by text lines. Maybe this will help me, too. I certainly will look more into Robocopy, which I am using on my home server now to maintain a mirrored copy of one drive on another. I'll see if it can do the backups on my boss's server faster. I'll drop the question of whether Xcopy does or does not verify its copying and ask such on another forum.

    Thursday, March 22, 2012 5:58 PM
  • Hi,

    No offense intended. But I just thought you should be aware of the constraints here, especially given you aren't really asking a scripting question.

    Bill

    Thursday, March 22, 2012 6:03 PM
  • JRV,

    Naturally, I appreciate and have appreciated all the help you have taken your personal time to provide. I do hope you weren't insulted by my double-checking on the Xcopy verify question. It's just a new piece of info which, as I explained to "friend" Bill above, I had seen differently in my own research. When you said, "I don't understand why you're doing an FC. Xcopy does that internally", I understood you knew something that the folks who write the /? help for Xcopy didn't know. Maybe I misunderstood...

    I will trey to take thisa bit at a time,

    #1 - Bill you can move this to eh WIndows 7 forum as it is now an OS discussion and not a scripting issues.

    #2 - Scott - the switch on XCOPY i snot documented because it is not longer recommended.  All versions of NT since W2K have had teh file buffering API updated to use OS control on NTFS volumes.  The volumes are transacted.  If teh write fails before a close teh voume will roll back thhe copy and xcopy willfail with an error mesasge.  That is why teh retatrtrable switch was added.  If an XCOPY fails it can be restarted from where it left off.  It is transacted in the OS.  You cannot use the buffering switch along with the restart switch.

    Turning off buffering will not help performance by any noticable amount.  FC will not really work as expected since XCOPY completion wth no errors will amount to the same thing.  The OS has certified that a file IO request has been successfully stored on disk.  This, of course, cannot be guaranteed on FAT volumes.  If the target volume is FAT then you will not have proof of success (FAT is not transacted) and will have to use FC or a checksum.

    In my opinion and due to the design of Windows NTFS I say that FC is a complete waste of time and network traffic.

    #3 - The ESEUtil copy generate=s a checksum.  There are also checksum generators  Generating checksums for all files is very fast. After a copy you can ask the remote system to generate a checksum and compare it to the source file checksum. 

    #4 Robocopy does a complete verificcation and transacted copy and is restartable.  It can mirror to/from multiple servers.  Mirror mode is best and is incremental.  Files are updated on teh target as they chage on the source.

    #5 FRS is a very good method of synchronizing folder sets over the network.  It i swhat Active irectory uses to maintain teh SYSVOL and AD instaces.  If it was not totally reliable Microsoft would be out of business.

    BITS is also a good a secure way to replicate data and can provide buffered access to multiple destinations. If the destinations are synching at the same time BITS will only need to read the file once. It is designed to allow large numers of users access to very large files over the Internet.  Windows Update runs on BITS.  BITS absolutely has to be reliable.


    ¯\_(ツ)_/¯

    Thursday, March 22, 2012 6:23 PM
  • Bill,

    When I ask, "Is there a way to pull chunks of data from the source once, then copy & compare to multiple destinations without having to do a separate copy for each destination, like this?" in a forum on scripting, I'm asking, is there a scripting solution for my problem. I inform folks I'm familiar with VBS and batch files. I'm thinking, someone may say, "Yes, there's a new version of VBS that just got released that has 'fso.CopyMultiDestFC()' that'll cover you, Take a look for it." Or someone may say, "Not in VBS or batch files, but Sysinternals just put out a multi-destination copy utility with auto-file-compare." Or "You'll have to get into C or C#".

    That's a scripting answer for a scripting question.

    I see that the Xcopy-verify question which came up later based on some info I thought JRV was saying, would be better asked elsewhere. It's just that JRV and I were talking familiarly on the subject before you came on board. If you'd said something like "To the best of my knowledge Xcopy does/doesn't do an FC-style verify, try on the Windows 7 General forum", istead of implying I'm trying to get you all to do my IT for me, it would have come off a lot less insulting...

    Thursday, March 22, 2012 6:26 PM
  • Here is your data chunker on steriods:

    FRS Technical Reference

    File Replication service (FRS) is a technology that replicates files and folders stored in the SYSVOL shared folder on domain controllers and Distributed File System (DFS) shared folders. When FRS detects that a change has been made to a file or folder within a replicated shared folder, FRS replicates the updated file or folder to other servers. Because FRS is a multimaster replication service, any server that participates in the replication of a shared folder can generate changes. In addition, FRS can resolve file and folder conflicts to make data consistent among servers.

    http://technet.microsoft.com/en-us/library/cc759297(v=ws.10).aspx


    ¯\_(ツ)_/¯

    • Marked as answer by Bill_Stewart Thursday, March 22, 2012 6:31 PM
    Thursday, March 22, 2012 6:27 PM
  • Publishing Applications Using a Hub-and-Spoke FRS Topology

    Publish Applications Using Hub-and-Spoke Topology

    ¯\_(ツ)_/¯

    Thursday, March 22, 2012 6:28 PM
  • JRV, thanks for laying all that out. It sounds like I have more options available. I'll look into these and see what I could do...
    Thursday, March 22, 2012 6:30 PM
  • The following discuses how/why DFS replication can be used to provide redundancy (backup paths and storage) to an organizzation.

    FRS and DFS replication can be scheduled as well as continuous.  If yu only want an update after, say, 1 AM then schedule the replication to begin at 1 AM and terminate at 6 AM or any other schedule.  You can receive replication reports periodically and on all errors tthough WMI or Evntlog events.

    FRS is faster and more efficient than XCOPY or Robocopy.  Once implemented DFSR can be leveraged into all kinds of backup and redundancy scenarios. It is also the only technology that cooperates with vituralization and clustered systems.  I believe that XCOPY and Robocopy will have issues in a clustered environment as there is no support during failover or VM resets.


    ¯\_(ツ)_/¯


    • Edited by jrv Thursday, March 22, 2012 6:38 PM
    Thursday, March 22, 2012 6:38 PM