locked
Robocopy problem under Windows 8 RRS feed

  • Question

  • I use ROBOCOPY to synchronize files between my desktop and laptop machines via a USB flash drive. ROBOCOPY is set to copy only updated files. It worked perfectly under Windows 7, reporting and copying only "newer" files; under Windows 8 it marks and copies the newer files and marks all the others as "modified" and copies them as well, even when the "modified" files haven't been accessed or written to.
    Here is the command line I am using:

    %SystemRoot%\system32\robocopy C:"\users\Robert Jacobs\Pix" i:\Pix /PURGE /E /XO /FFT /XA:SH /DST /R:5 /W:2 /LOG+:C:"\users\Robert Jacobs\My Documents\work"\leap_down.log /TS /NP

    As I said, this worked perfectly under Windows 7; it seems to be broken in windows 8. How can I fix it?

    All help will be gratefully appreciated.

    Robert

    Friday, December 14, 2012 4:30 PM

Answers

  • Could this be caused by the same problem that causes Windows File History to backup unmodified files?

    The following thread discusses this topic: Windows 8 File History - Excessive backup - Excessively saving copies of files (photos, videos, documents, etc.)

    The fix there was to disable Homegroup functionality.

    The question there was:

    Why is File History saving so many duplicate copies of my media, when I'm not making any changes to the media or metadata?!?
     Does anybody know? 

    and the final solution to date is:

    Since I switched HomeGroup off, File History appears to be working sensibly. Only changed files appear to being backed up. My NAS is noticably quieter and my laptop far more responsive (as it isn't constantly backing up!!)
    
    Thanks all - nearly gave up on File History. More than happy to live without HomeGroup as it really isn't needed in Win 8 as Noel helpfully pointed out above.

    Could this be it? As far as I know, BOTH the File History and Robocopy use Shadow Copy functionality to copy locked files on-the-fly. So it looks like, the problem lies with how Shadow Copy works when Home Group functionality is turned on.

    Gentlemen, could you please verify that and check if switching Home Group off helps here as well or not?


    Well this is the world we live in And these are the hands we're given...



    • Edited by Exotic Hadron Sunday, January 20, 2013 10:12 AM
    • Marked as answer by formerprof Monday, January 21, 2013 8:03 PM
    Sunday, January 20, 2013 10:05 AM

All replies

  • It can be caused by a different file systems. Have a look at this switch:

    /FFT :: assume FAT File Times (2-second granularity).


    Niki Han
    TechNet Community Support

    Wednesday, December 19, 2012 7:08 AM
  • Dear Niki,

    I'm already using that switch on both machines -- please look again at the command line I cited in the original post:

    %SystemRoot%\system32\robocopy C:"\users\Robert Jacobs\Pix" i:\Pix /PURGE /E /XO /FFT /XA:SH /DST /R:5 /W:2 /LOG+:C:"\users\Robert Jacobs\My Documents\work"\leap_down.log /TS /NP

    Both machines and the flash drive are formatted in the NTFS file system.

    Thanks so much for the suggestion. I'm coming to the opinion that the problem is a bug in the Windows 8 ROBOCOPY.

    Robert

    Wednesday, December 19, 2012 2:16 PM
  • Hi Robert,

    Just wanted to mention that I am experiencing the same issue, where my Robocopy scripts which copied files correctly under Vista and Win7 now copies many more files, with this new file class "Modified".  For example:

    robocopy "D:\Development\NTIS" "S:\Development\NTIS" /S /XA:RSH /XF *.scn *.tps /XO /NP /LOG+:"UpdatePassport.txt"
    
    now copies files that have not been touched for years.

    I cannot find any documentation for this "Modified" class, so I suspect there have been new "features" added to Robocopy in Win8 for which we have not yet received documentation.  Hopefully something will surface soon and we can exclude these files.

    Best,
    Scott

    Thursday, January 17, 2013 7:22 PM
  • Dear Scott,

    Thanks for your note. I'm glad to learn that I am not the only sufferer - i.e., it's not my imagination!

    I expected better of the TechNet forum -- one would thing there would be someone who knows enough about Win8/Robocopy to provide a solution to this most annoying problem.

    All good wishes. If I learn anything I'll be sure to share it with you.

    Robert

    Saturday, January 19, 2013 7:03 AM
  • hmmm, I haven't noticed any change so far...but I'm using a simple mirror

    Set FileDate=%date:/=%
    Robocopy "C:\Data" "F:\DataBackup" /MIR /XA:SH /XD AppData /XJD /R:5 /W:15 /TS /NP /LOG:"F:\BackupLogs\DataBackup %FileDate%.log"

    Saturday, January 19, 2013 6:20 PM
  • Dear Mark 522010,

    Thanks for your note. I don't think your command line differs from mine in essentials; I've tried it with and without MIR.

    I'm starting to wonder whether it has anything to do with the transfer vehicle: the intermediate carrier is an NTFS-formatted USB flash drive. Both the desktop and laptop machines are also NTFS formatted, a RAID array in the case of the former.

    Two of the files being synchronized are encrypted and may not be transfer over an FAT32 flash drive. I'll experiment with that and report back.

    Robert

    Later: FAT32 will not transfer encrypted files, so it's of no help to me. In any event the commands I use worked perfectly under Windows 7.


    • Edited by formerprof Sunday, January 20, 2013 2:03 AM
    Saturday, January 19, 2013 9:15 PM
  • Could this be caused by the same problem that causes Windows File History to backup unmodified files?

    The following thread discusses this topic: Windows 8 File History - Excessive backup - Excessively saving copies of files (photos, videos, documents, etc.)

    The fix there was to disable Homegroup functionality.

    The question there was:

    Why is File History saving so many duplicate copies of my media, when I'm not making any changes to the media or metadata?!?
     Does anybody know? 

    and the final solution to date is:

    Since I switched HomeGroup off, File History appears to be working sensibly. Only changed files appear to being backed up. My NAS is noticably quieter and my laptop far more responsive (as it isn't constantly backing up!!)
    
    Thanks all - nearly gave up on File History. More than happy to live without HomeGroup as it really isn't needed in Win 8 as Noel helpfully pointed out above.

    Could this be it? As far as I know, BOTH the File History and Robocopy use Shadow Copy functionality to copy locked files on-the-fly. So it looks like, the problem lies with how Shadow Copy works when Home Group functionality is turned on.

    Gentlemen, could you please verify that and check if switching Home Group off helps here as well or not?


    Well this is the world we live in And these are the hands we're given...



    • Edited by Exotic Hadron Sunday, January 20, 2013 10:12 AM
    • Marked as answer by formerprof Monday, January 21, 2013 8:03 PM
    Sunday, January 20, 2013 10:05 AM
  • I just disabled the homegroup on both computers. It will be a day or so before I can tell for sure whether the problem is gone. Thanks so much for the suggestion!

    Robert

    Sunday, January 20, 2013 8:08 PM
  • Robert, could you please notify us of the result no matter positive or negative?

    Well this is the world we live in And these are the hands we're given...

    Sunday, January 20, 2013 9:47 PM
  • As I told you, I disabled the homegroup on both machines. Then I worked on the laptop for several hours, viewing a number of files and writing to two or three. On running Robocopy the files to which I had written were marked "newer" and the files which had been viewed but not written were marked "modified." Apparently any access to a file is detected by the new Robocopy which then marks them "modified." There must be some background process which accesses the files; in my case that might be Acronis True Image Home, but I doubt it -- I think it's something inside Windows 8. Not enough time has elapsed yet to see if it is the Homegroup process. I'll report on that in another day or so.

    The underlying problem is not the background access to files; it seems to be the Win8 Robocopy's insistence on copying "modified" files which have not in fact been updated.

    Please -- if anyone else has any insight into this matter I'd be most grateful for any help.

    Robert

     
    Monday, January 21, 2013 12:34 AM
  • Yesterday, I disabled both Homegroup services, rebooted, and ran my Robocopy script (to flush any current file class attributes).  I worked a bit yesterday and today and ran the Robocopy script again.  There are still hundreds of files in folders that have not been accessed by any software being copied by Robocopy with a class of "Modified".

    Glad to get the Homegroup link out of Explorer, but wish that would have been the fix.  It wasn't.

    Best,
    Scott

    Monday, January 21, 2013 3:38 PM
  • Pity...

    Could you confirm that affected files (those that appear as modified although have never been) have different formats, or are they mostly media formats like pictures, music, and videos?

    If these are mostly media files, do you by chance have Media Streaming enabled?


    Well this is the world we live in And these are the hands we're given...

    Monday, January 21, 2013 3:46 PM
  • Actually, these are source code files (primarily text, but some binary).  There are a couple of image files, and they are repeatedly copied, but so are EXE, DLL, and other binaries (such as proprietary database files, etc.).  Also - none of these folders are under "My Documents", so I wouldn't expect most Windows processes to be aware of them.  I do have the indexing service running, and configured this drive to "Allow files on this drive to have contents indexed in addition to file properties".  I am going to disable that and see what happens.

    Thanks,
    Scott

    Monday, January 21, 2013 4:02 PM
  • Dear Scott,

    So far my experience with the disabled homegroup is much more promising. After the first run of my script, on which a few "modified" files still turned up, the back and forth between my laptop and my desktop has been clean. I don't discount the possibility that the problem may arise again. Let me suggest that you try (I kind of hope you haven't already done so!) the /FFT switch for Robocopy. Several people have reported that time variations between the machines can produce the "modified" problem.

    I should add that my automatic Acronis system backup took place overnight. Every file in the machine must have been touched somehow, but when I ran the script this morning no files were listed as modified.

    Dear Exotic Hadron,

    All files of whatever type (within the directories which I am synchronizing) have shown up. There's been no distinction between media files, WORD files or database files.

    Thanks to you both for helping with this -- all good wishes.

    Robert

    Monday, January 21, 2013 4:05 PM
  • Hi,

    Your problem is most surely related to this one.

    http://social.technet.microsoft.com/Forums/en-US/w8itprogeneral/thread/88fc0286-4328-48d9-b945-5c280bce58b6

    W8 files are being "touched" on a continuous basis by some  Windows background process.

    So both Robocopy and Backup see your files as "modified".


    Sebastian Sajaroff Senior DBA Pharmacies Jean Coutu

    Monday, January 21, 2013 4:11 PM
  • FWIW, I'm running W8 x64 Enterprise, there are no HomeGroups in my domain, and File History is not on.   I'm also experiencing Robert's problem with robocopy when copying Intel drivers from my E: drive to a USB key.
    Monday, January 21, 2013 4:21 PM
  • Dear Sebastian,

    Thanks for the pointer. There seem to be at least two controllable functions which touch files. One is the "file history" function reachable via the control panel. I have that switched off. The other is file indexing, also seen in the control panel. I seem to have indexing set to "on" for the directories which are included within my script.

    At the very least Robocopy should have a switch which lets it ignore modified files which have not in fact been changed. Under windows 7 this problem didn't exist; of course, I don't know whether Robocopy has been changed, or, if it has, why.

    I'm going to try to force the indexing to see if "modified" recurs; I haven't seen it since I left the homegroup yesterday.

    Robert

    later: I forced the reindex, ran the script and had no "Modified" files in the Robocopy log.

    • Edited by formerprof Monday, January 21, 2013 4:47 PM
    Monday, January 21, 2013 4:28 PM
  • Hi Sebastian,

    the topic you are linking to looks to be exactly the same as described in this topic, which already provides some possible workarounds: Windows 8 File History - Excessive backup - Excessively saving copies of files (photos, videos, documents, etc.).

    P.S.: That's just to link similar topics altogether.


    Well this is the world we live in And these are the hands we're given...

    Monday, January 21, 2013 4:57 PM
  • By the way, gentlemen, since File History utilizes NTFS USN journal, what if we check it with fsutil to find modifications?

    And, by the way, some gentleman here confirms he has been able to workaround the issue with robocopy considering files as modified by installing previous version on robocopy.


    Well this is the world we live in And these are the hands we're given...


    Monday, January 21, 2013 4:59 PM
  • This topic reports some workaround.

    Robocopy to Network Share, is it just completely broken?

    "I found it.  It's the ARCHIVE attribute on the source files.  On the new Robocopy, the presence of the "A" attribute seems to force the file to be copied.  The problem is, Robocopy doesn't then reset the attribute. 

    And if you use /M (copy only "A" files, then reset "A"), any files without the "A" attribute set get ignored - even if they should have been copied because they don't exist on the target, etc.  So that doesn't work.

    The solution "appears" to be to reset the "A" attribute on the structure you want to copy BEFORE you do the copy.  Then your script "should" work.  (Mine did.)  I used: ATTRIB -A C:\MYDIR\*.* /D /S to clear the attributes.  Then Robocopy started working.

    Of course, going forward, modified files will get "A" set, and then Robocopy will start copying those files every time again, so what I did was to add the ATTRIB command AFTER the Robocopy command in my script.  Run Robocopy, then reset "A"s.  That "should" yield the desired result, I hope!"

    There's a note however: "Resetting the archive attribute greatly reduces modified file copies, but does not totally eliminate it."


    Well this is the world we live in And these are the hands we're given...

    • Proposed as answer by scotteb Thursday, January 24, 2013 2:09 PM
    Monday, January 21, 2013 5:13 PM
  • Aha! The archive bit does seem to invoke Robocopy's "modified" switch. I've tried the Robocopy switch COPY:DT (the default is /COPY:DAT) and tried setting and resetting the archive bit on one of the directories in my script. I ran Robocopy both ways and did not get any "modified" files with the /COPY:DT switch set.

    The archive bit may not be the only thing which causes files to be marked modified, but it certainly seems to be one of them.

    Robert

    Monday, January 21, 2013 6:09 PM
  • Well,  both workarounds provided by Exotic Hadron are worth a try.

    Sebastian Sajaroff Senior DBA Pharmacies Jean Coutu

    Monday, January 21, 2013 6:10 PM
  • Drat! I reestablished the homegroup, ran the script with /COPY:DT and was deluged again with modified files. Now I have to try adding the ATTRIB commands found in the link suggested by Exotic Hadron.

    Robert

    Monday, January 21, 2013 6:42 PM
  • I think that we had better stop using robocopy for now as it's starting to be a problem... When it poses so many possible problems with incorrect comparison/backup, who would know when it quirks the other day?

    Well this is the world we live in And these are the hands we're given...

    Monday, January 21, 2013 6:54 PM
  • I'd be happy to abandon robocopy were there something which offers the same facilities: I have to be able to move encrypted files and to exclude certain files, certain subdirectories and, of course, to copy only newer files.

    So far, abandoning the homegroup seems to be the most useful.

    Robert

    Monday, January 21, 2013 7:44 PM
  • Would you please mark it as a temporary answer then for others to be able to clearly see the solution without having to read through all the posts in topic?


    Well this is the world we live in And these are the hands we're given...

    Monday, January 21, 2013 7:49 PM
  • Done. I'm a little slow at catching on to how this forum works, but I think I got it right.

    Robocopy is the villain -- how does one attract Microsoft's attention so that they will fix the bug, preferably before I get too senile (Wednesday would be excellent) to appreciate it?

    Many thanks for your help.

    R.

    Monday, January 21, 2013 8:11 PM
  • Same issues here....   I've reverted back to the one from the 2003 resource kit which seems to work ok.

    http://www.microsoft.com/en-us/download/details.aspx?id=17657

    Arjan

    • Proposed as answer by Exotic Hadron Thursday, February 21, 2013 3:57 PM
    Wednesday, February 20, 2013 6:03 PM
  • My question is how do I find and delete all the copies that were made? My new computer's hard drive 150 GB is FULL and I don't know even how to find the copies to delete them all!!! Yikes!!!
    Wednesday, June 5, 2013 7:54 PM
  • I assume you are talking about file history taking up your hdd.

    Press windows + W to search settings.
    Type "File History" and click on it.

    Click advanced settings

    Click on Clean up versions

    OR

    c:\windows\system32\FhManagew.exe -cleanup age

    where age = days (0=all but current)

    Arjan


    Thursday, June 6, 2013 11:34 AM
  • Does anyone know if there is a ROBOCOPY.EXE update in the works to fix this issue?

    I, too, started having issues with ROBOCOPY.EXE copying files that were not modified after upgrading to a Windows 8 Professional computer. 

    Wednesday, August 21, 2013 4:28 PM
  • Any news on this topic?

    I just switched from Windows 7 to Windows 10 and ran into this problem as well.
    So it still persists. I am doing backups from NTSF to ext4 on a Linux-NAS and having the issue.
    Same robocopy script ran fine with Vista and Windows 7.

    A lot of files in my backup are reported as "same", but since Windows 10 many are regarded as "changed" which was not the case with robocopy running on Vista or Windows 7.

    Just to be clear:
    Known and resolved issues already present in Windows 7 (especially for backups to non-NTSF file-systems):

    - using the /FFT switch

    - using the /DST switch

    - i'm using /COPY:DT and /DCOPY:T


    Why are many files regarded as "changed" by robocopy?

    Any help still appreciated.

    Monday, September 7, 2015 9:26 AM
  • Yep, Windows 10 same issue here.  Too bad MS does not want to fix it.
    Sunday, February 21, 2016 1:01 AM
  • Just wanted to post that I have not seen this problem. 

    The command line I use successfully in Windows 10 is:

    robocopy "C:\Users\ron\Documents\" "\\ServerName\MyDocs" /E /ZB /IPG:333 /XF "Thumbs.db" /NP /NFL /NDL /LOG:"C:\Users\ron\Documents\backup.log" /R:3 /W:600

    Granted I am not doing a mirror image but only those files that are new or I have modified are copied.  I will point out that I have a large wait time on retires because of real-time file protection in a virus checker.  I found that a file scan was initiated when Robocopy tried to copy a file. 

    I recommend starting with the above set of parameters on a test directory and slowly add each of your parameters back until you identify the one that breaks it.  This will help isolate the issue and can then be reported as a bug.

    Thursday, February 9, 2017 1:17 AM