locked
Windows 7 64-bit Corrupting (Altering) Large Files Copied to External NTFS Drives

    Question

  •  

    I discovered yesterday by chance that when copying a large file (larger than 500MB or so) to an external USB drive, roughly 30% of the time, the copy is not identical to the original.

    Specifics:

    - No error occurs / no error message appears during the copy

    - The file size of copy is always identical, but content is not

    - Subsequent attempts to copy the same file will ultimately yield an identical copy.  This would seem to rule out interference by an external program such as an anti-virus program (and the only AV I am running is Microsoft Security Essentials)

    - The issue impacts copies made via the Windows GUI or via command-line copy or xcopy

    - So far, the third party utility "TeraCopy" somehow manages to consistently produce clean copies, and therefore is a temporary workaround

    - The issue occurs with multiple external USB drives, no matter what make or model.

    - UPDATE:  The issue appears to occur only with NTFS-formatted external USB drives

    - The USB drives involved pass error checks, and copies made to these drives on other (non W7) systems produce identical copies

    - Differences in the copied versions were revealed via the behavior of software operating on these files, and are confirmed via WinDiff as well as command-line "FC" (file compare).

    - The problem does not appear to impact relatively small files (1 to 100MB or so).   I have not found any particular threshold, but I have seen the issue impact numerous files in the 500MB neighborhood.

    - The problem is not new, as verification of files copied over a year ago has revealed.


    This strikes me as a very serious issue, and one which would likely go undetected in many cases.  I discovered it only by chance, but after spending hours testing and narrowing down the issue, I can easily reproduce it.





    • Edited by KT-Retro Thursday, January 26, 2012 7:22 PM
    Tuesday, September 13, 2011 6:12 PM

All replies

  • Here is my system information:

    OS Name    Microsoft Windows 7 Professional
    Version    6.1.7600 Build 7600
    Other OS Description     Not Available
    OS Manufacturer    Microsoft Corporation
    System Name    KIPP-XPS
    System Manufacturer    DELL Inc.
    System Model    Studio XPS 435T
    System Type    x64-based PC
    Processor    Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz, 2668 Mhz, 4 Core(s), 8 Logical Processor(s)
    BIOS Version/Date    DELL INC. A07, 2/6/2009
    SMBIOS Version    2.5
    Windows Directory    C:\Windows
    System Directory    C:\Windows\system32
    Boot Device    \Device\HarddiskVolume3
    Locale    United States
    Hardware Abstraction Layer    Version = "6.1.7600.16385"
    User Name    kipp-XPS\kipp
    Time Zone    Eastern Daylight Time
    Installed Physical Memory (RAM)    4.00 GB
    Total Physical Memory    3.99 GB
    Available Physical Memory    2.51 GB
    Total Virtual Memory    7.98 GB
    Available Virtual Memory    6.21 GB
    Page File Space    3.99 GB
    Page File    C:\pagefile.sys

    Wednesday, September 14, 2011 11:23 AM
  • I have the identical problem.  I appreciate the work around and hope someone has an answer.
    Wednesday, September 14, 2011 9:36 PM
  • Hi,

    I experience a known issue about advanced Format Disks that have a 4KB physical sector size. Please see the details in http://support.microsoft.com/kb/982018.  This article describes an update that improves the compatibility of Windows 7 and Windows Server 2008 R2 with Advanced Format Disks that have a 4KB physical sector size. This hotfix is only applicable to Advanced Format disks which report themselves as having a 4 KB physical sector size, and which emulate a logical addressing interface of 512 bytes.

    Download link: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=12248

    Please apply the hotfix and reboot to see if the problem persists.

    If this doesn't fix the issue, please let me know the output of the following command:

    fsutil fsinfo ntfsinfo <problematic drive>

    Thanks.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
    • Marked as answer by Niki HanModerator Monday, October 10, 2011 3:27 AM
    • Unmarked as answer by KT-Retro Thursday, October 13, 2011 6:46 PM
    Friday, September 16, 2011 12:12 PM
  • None of the various external drives on which I am experiencing this problem are Advanced Format Disks, so unfortunately, the suggested hotfix is not a cure.

    Please review the symptoms again.   I am convinced there is a bug in the Windows 7 OS related to I/O buffering...a bug that third-party software such as TeraCopy circumvents due to the way in which it performs file copies.
    Thursday, October 13, 2011 6:49 PM
  • Hi,

    Would you please update the following compoonent to a lastest version and see if it works?

     

    Package: ntfs.sys 6.1.7600.20714
    ----------------------------------------------------------- 
    KB Article Number (s) : 983466  
    Language: All (Global)  
    Platform: x64  
    Location: ( http://hotfixv4.microsoft.com/Windows%207/WindowsServer%202008%20R2/sp1/Fix317778/7600/free/413990_intl_x64_zip.exe )

     

    Package: Usbstor.sys 6.1.7600.21009


    ----------------------------------------------------------- 
    KB Article Number (s) : 2581464  
    Language: All (Global)  
    Platform: x64  
    Location: ( http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix373164/7600/free/435480_intl_x64_zip.exe )

     

    Package: Storport.sys 6.1.7600.20959
    ----------------------------------------------------------- 
    KB Article Number (s) : 2528357  
    Language: All (Global)  
    Platform: x64  
    Location: ( http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix368063/7600/free/432658_intl_x64_zip.exe )

    NOTE: Be sure to include all text between '(' and  ')' when navigating to this hot fix location!

     

    Thanks.

     


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
    Wednesday, October 19, 2011 3:00 PM
  • I ran into this issue back in Dec 2010 when I upgraded to Windows 7.  I have since verified this happens on 4 different Windows 7 PCs, with different usb drives and different usb cables.  I ended up having to switch to Teracopy.  This is a very serious and widespread issue that needs to be addressed.

    Monday, October 24, 2011 6:27 AM
  • Hi,

    Would you please try to upgrade to windows 7 SP1 and see effort?

    Did you ever format the extra disk?

    Thanks.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
    Tuesday, October 25, 2011 2:36 PM
  • Hi,

    I have exactly the same problem as KT-Retro, word by word!!. The problem still happens in SP1 and it is very annoying because you have to compare all the big files that you transfer to NTFS formatted USB drives. I think that the problem is in the write cache of NTFS file systems,that cannot be disabled and is not working correctly for external "slow" USB 2.0 drives.

    Please, take it seriously, it is a really big problem of Win 7 and the consequences are enormous (files are not reported to be damaged until used).

    Thanks.

    Sunday, November 27, 2011 12:31 AM
  • Any news about the problem. Is microsoft addressing it? I am using TeraCopy as a workaround, but I think that it should not be needed for a high level OS like Windows 7.

     

    Best regards,

    Iker.

    Wednesday, December 07, 2011 9:44 AM
  • Same problem as you, I think I am finding the solution ! Do you have microsoft security essentials ?
    Wednesday, December 07, 2011 6:27 PM
  • Yes I have. But I think that I didn't have it installed when I realised about the problem one year ago. Anyway, the problem continues even when microsoft has updated to SP1, so maybe Security Essentials has something to do with this.
    Wednesday, December 07, 2011 7:43 PM
  • Same problem for me. Some more details:

    • I experience the problem with external eSATA drives (and only with external drives), so it's not USB-specific, but specific to external drives.
    • It is not antivirus related. I tested before installing any sort of antivirus.
    • It is not directly related to big files, only to large transfers. Transferring many (tens of thousands) relatively small (<10MB) files also produces some corrupted files (in the order of 0.01% to 0.1%).

    As others mentioned, the problem is specific to Windows Explorer or cmd copy tools. Teracopy or Total Commander copies do not show this problem.

    An example fsutil info for an external drive I write to:

    C:\Windows\system32>fsutil fsinfo ntfsinfo f:
    NTFS Volume Serial Number : 0x5c0889d20889ac18
    Version : 3.1
    Number Sectors : 0x00000002ba9d37ff
    Total Clusters : 0x000000005753a6ff
    Free Clusters : 0x0000000023b316c3
    Total Reserved : 0x0000000000000000
    Bytes Per Sector : 512
    Bytes Per Physical Sector : <Not Supported>
    Bytes Per Cluster : 4096
    Bytes Per FileRecord Segment : 1024
    Clusters Per FileRecord Segment : 0
    Mft Valid Data Length : 0x000000000a280000
    Mft Start Lcn : 0x00000000000c0000
    Mft2 Start Lcn : 0x0000000000000002
    Mft Zone Start : 0x00000000000c8240
    Mft Zone End : 0x00000000000cc840
    RM Identifier: 88F6537B-07D2-11E1-A59E-D8D38594CBF6

     

    I have Win7 x64 SP1, and have not tried the hotfixes yet. Will try these hotfixes today.

     

    Update:

    • The hotfixes made no difference. Of the 33000 copied files, 1300 have been checked by now, and there are 4 corrupted files (sizes vary: 18MB, 800MB, 3GB and 265GB).
      I applied all fixes suggested above: "Windows6.1-KB2528357-v2-x64.msu" and "Windows6.1-KB2581464-x64.msu". The other 2 (982018 and 983466) were not installed by the system, because one did not apply to my machine, and the other was already installed (with SP1?).
    • I also experienced the problem with Teracopy, when I enabled an option which is off by default: "use system write cache".

    • Edited by 25Albert Wednesday, December 14, 2011 11:43 PM more info
    Wednesday, December 14, 2011 4:06 PM
  • The problem seems clearly related to the NTFS buffering used while copying. If the destination drive is not fast enough (like external drives), the buffer overflows or some kind of data corruption happens. TeraCopy does not use a buffer so it is not affected.

    Anybody from Microsoft can say anything? It is a first priority problem of Windows 7!!!


    • Edited by ikerrg Wednesday, December 14, 2011 6:52 PM
    Wednesday, December 14, 2011 6:51 PM
  • Is there any progress on this issue?  It is incredible that such a serious flaw is not being addressed by Microsoft...
    Tuesday, January 24, 2012 7:36 PM
  •  

    For those impacted by this issue, what are your experiences when copying large files over the network to mapped drives, for example?    Is write caching used in the same manner as it is with external drives?  (UPDATE:   This problem is so old at this point, I had forgotten that it was NTFS-format specific, so it's unlikely that a network copy operation would be impacted...but I am testing at this moment... 1/25/12, 8:20PM EST)

    By the way, TeraCopy does indeed resolve the issue with copying to USB-connected external drives, however, the program tends to crash when copying a large group of files.   Seems one can't win...

    What can be done to convince MS that there is a serious bug in the Windows 7 OS w/ regard to copy operations to directly-connected external drives?


    • Edited by KT-Retro Thursday, January 26, 2012 1:18 AM
    Tuesday, January 24, 2012 7:56 PM
  • Assuming say a 1.5 gigabyte LargeTestFile.tif, an external drive G:, and a no.txt file with a couple of n characters followed by returns in it, do you find the following batch can be used to reliably detect the problem?

     

    :TESTLOOP

    COPY LargeTestFile.tif G:\TEMP

    COMP G:\TEMP\LargeTestFile.tif LargeTestFile.tif <no.txt

    IF NOT ERRORLEVEL 1 GOTO TESTLOOP

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Tuesday, January 24, 2012 9:52 PM
  • Assuming say a 1.5 gigabyte LargeTestFile.tif, an external drive G:, and a no.txt file with a couple of n characters followed by returns in it, do you find the following batch can be used to reliably detect the problem?

     

    :TESTLOOP

    COPY LargeTestFile.tif G:\TEMP

    COMP G:\TEMP\LargeTestFile.tif LargeTestFile.tif <no.txt

    IF NOT ERRORLEVEL 1 GOTO TESTLOOP

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    I am not clear on the purpose of the "<no.txt" in the copy command.   In any event, Windows 7 itself never detects an error during the copy operations which corrupt large files on transfers to external drives, however, "fc" finds the differences, as does WinDiff.   It appears that the corruption is always in the form of an insertion of a string of null characters at a random location in the file.   See the earlier posts for details (including that corruption occurs intermittently and not with every attempted copy of the same or any other file).
    Thursday, January 26, 2012 1:08 AM
  • Also, ikerrg is 100% correct.   This problem occurs with NTFS drives (any make or size I have tested), and does not occur with FAT-32 drives.   Also, he is correct that TeraCopy avoids this problem, apparently by not using the same caching or buffering mechanism for copies to such drives.   Unfortunately, TeraCopy is prone to crashing when a large group of files is being copied...

    How can one engage Microsoft to convince them that there is such a serious bug in the Windows 7 which needs to be addressed?
    Thursday, January 26, 2012 1:11 AM
  • The <no.txt is so that when COMP asks if you want to compare more files you're answering "No" without the procedure stopping.

    If you have another sequence of commands you can use to reproduce the problem unambiguously, please list them.

    I'd like to see if I can reproduce the problem on two different systems here with external MyBook USB drives.

    • If I can, I'll try to get to the bottom of why it's happening.
    • If I can't, we can compare things about our systems and configurations to see if we can discover why there's a difference.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Thursday, January 26, 2012 1:48 AM
  • How many different systems have you tested this on, by the way?

    I suspect it's not a systemic problem with Windows, because as far back as I can remember all my external drive accesses have been 100% fine.  I'd have noticed if there were files corrupted. 

    But I want to make sure I'm doing exactly what you're doing before claiming I don't have the problem.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Thursday, January 26, 2012 1:52 AM
  • I have only one Windows 7 (64-bit) system, but please read back through the thread, and you'll see that others are experiencing identical symptoms.    I was unaware that there was a problem until when, in September, I retrieved a large file from an external NTFS drive and, after experiencing issues with the file, I began comparing a number of backup copies of large files with source files still on my hard drive, and discovered several of the backed-up copies to be different (to have been essentially corrupted during the transfer).  I also discovered I could reproduce the issue with experimentation (I use the term "issue" rather than "error" because there is no outward indication of a problem, and there is no structural corruption of the file or the disk).

    Here are the specifics again:

    - Copies of large files, files typically larger than 500MB, are corrupted roughly 30% of the time when copying them under Windows 7 64-bit to USB-connected NTFS-formatted external drives.

    - No error occurs / no error message appears during the copy

    - The file size of copy is always identical, whether or not data was corrupted during the copy process.

    - File differences are confirmed via either the command-line "FC" command or a utility such as WinDiff

    - The issue impacts copies made via the Windows GUI -OR- via command-line copy or xcopy

    - The issue occurs with multiple external USB NTFS-formatted drives, no matter what make or model.

    - Subsequent attempts to copy an affected file will ultimately yield an identical copy.  This would seem to rule out interference by an external program such as an anti-virus program (and the only AV I am running is Microsoft Security Essentials)

    - The USB drives involved pass error checks, and copies made to these drives on other (non W7) systems produce identical copies

    - So far, the third party utility "TeraCopy" manages to consistently produce clean copies, and therefore is a temporary workaround.   This utility apparently works because it, by default, bypasses the NTFS memory caching operation used by the Windows 7 OS...a caching system which I have so far found no way of disabling.

    - The problem does not appear to impact relatively small files (1 to 100MB or so).   I have not found any particular threshold, but I have seen the issue impact numerous files in the 500MB neighborhood.

    - The problem seems to date at least to the version of Windows 7 that was in release as far back as the Fall of 2010, as I discovered corrupted backup copies of files dating back that far.   Again, the files are corrupted with respect to the original copy...NOT with respect to file structure itself.

    I hope this is helpful, and I greatly appreciate your interest.   If you look back in the thread, you will see that my experience is not unique.  Also, I should warn that I might still be unaware today of the corruption had circumstances not required that I access such an affected file in a manner that revealed the corruption, as there is otherwise NO DIRECT INDICATION when the corruption occurs of an issue or the fact that the copied version of the file is different from the source.

    -Kipp

    (P.S.  It's worth noting that, even with "Optimize for Removal" set on an external drive, the write cache is still minimally utilized, in a "write-through" mode.)






    • Edited by KT-Retro Thursday, January 26, 2012 9:31 PM
    Thursday, January 26, 2012 6:19 PM
  • How many different systems have you tested this on, by the way?

    I suspect it's not a systemic problem with Windows, because as far back as I can remember all my external drive accesses have been 100% fine.  I'd have noticed if there were files corrupted. 

    But I want to make sure I'm doing exactly what you're doing before claiming I don't have the problem.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options


    Please see my detailed response just posted & subsequently edited, but I want to point out that what is being described here is not an issue that would impact "access" or that would otherwise yield an overt indication that a copy of a file was not identical to the original.   This is why, in my view, this issue has gone undetected for so long now and uncorrected.    What I am referring to as "corruption" here is not a structural issue that would be evident in an error scan or that would yield any sort of OS error.   In addition, when such large files are videos, it is even possible for playback to be unaffected if the sequence of data which has been replaced (apparently by nulls) in the copy by chance does not impact the decoding of the particular video format.   Likewise, for a database file for example, if the sequence of nulled data is in a database field, it might not corrupt the database structure.   The point is, the nature of this issue makes it difficult to detect unless one happens to stumble on it OR go looking for it.  So, if you are using Windows 7 64-bit and NTFS drives connected externally via USB, I don't think it's at all safe to assume that any copies you have made of large files to such drives are unaffected.    Is it possible that the issue could be limited to a specific hardware platform?  I suppose so, but all signs so far sure seem to point to an issue in the operating's system handling of NTFS write caching.

    UPDATE:  I see from a post above that the issue is not USB-specific, but it also occurs on externally-connected eSATA drives.   25Albert didn't explicitly indicate NTFS, but that's a near certainty in his case given the collective evidence.   (I repeatedly experimented with FAT-32 external drives and could not reproduce the problem)


    • Edited by KT-Retro Thursday, January 26, 2012 6:52 PM
    Thursday, January 26, 2012 6:42 PM
  • To test this I've just copied a bunch of big files, from 100 MB to 1.5 GB, to and from my external Western Digital USB MyBook drives and done binary comparisons of them with the originals, and I didn't detect any corruption whatsoever.  All copies were 100% perfect.

    It may be that this problem happens only with some systems, with some configurations, or with some kinds of disks, but I think I've just shown that it doesn't happen in all cases, or that it's so infrequent that it didn't happen in the copies I've done (I copied about 100 GB of data total).

    Let's try to be absolutely clear:

    With what you've seen, would you feel you would see the problem with virtual certainty in 100 GB of file copies, as I have done?

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Thursday, January 26, 2012 10:53 PM
  • Another open question to all who are seeing this problem...  Do you have ECC memory in your system - i.e., if you had a RAM error would your system either correct it or halt? 

    I ask because mine has ECC, and before I changed out a pair of DIMMs that tested otherwise fine it did occasionally detect some faults.  I wasn't as aware that RAM errors were something to be concerned with before that experience.  Now I wouldn't have another system without ECC RAM.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Thursday, January 26, 2012 11:03 PM
  • Noel,

    Thanks.   You are using Windows 7 64-bit, correct?   What about the drive...is it NTFS?    I believe that both are common elements.    Also, what setting are you using for Generic Flash Disk USB Device Properties, Quick Removal or Best Performance?   The former makes minimal use of write caching, while the latter uses it heavily.   This is a setting I have not yet checked on my system (I am not at the Windows 7 computer at the moment).    Yes, at present, I would absolutely experience the problem copying, for example, even only five files of the size 800MB to 1.5GB.

    Have a look at my system specs earlier in the thread and see if you notice any specific differences that you think might be a factor.

    Regarding memory, I will have to look into this...

    Thanks!

    Kipp


    • Edited by KT-Retro Thursday, January 26, 2012 11:32 PM
    Thursday, January 26, 2012 11:29 PM
  • Yes, Windows 7 x64 Ultimate, Western Digital MyBook 2 TB formatted for NTFS.  At the moment I'm set for "Quick Removal", but I will try the other setting in another test.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Thursday, January 26, 2012 11:36 PM
  • Something I missed before...  You say you're looking at "Generic Flash Disk USB Device Properties".

    Are you using a flash memory card?

    As I showed in my response to your other thread, my disk has a specific driver:

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Thursday, January 26, 2012 11:43 PM
  • Yes, Windows 7 x64 Ultimate, Western Digital MyBook 2 TB formatted for NTFS.  At the moment I'm set for "Quick Removal", but I will try the other setting in another test.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

     

    Mine is also set for Quick Removal, which is apparently the default upon insertion of any drive.

    I think what I should do next is perform another series of tests...to rule out the possibility that a Windows update in the last few months has fixed the problem.   I doubt this is the case, but I have not tested in a while.   Assuming the issue is still present, it would probably also be a good idea for me to enable "Better Performance" on the device to see if by changing the caching algorithm used, it resolves the issue.

    Do you see anything different in our configurations that might be a factor?   I am running Windows 7 Professional,
    Version 6.1.7600 Build 7600

     

    Thursday, January 26, 2012 11:49 PM
  • Something I missed before...  You say you're looking at "Generic Flash Disk USB Device Properties".

    Are you using a flash memory card?

    As I showed in my response to your other thread, my disk has a specific driver:

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    No, the pane title for this setting on my system is specific to the particular drive.  I was earlier citing a generic reference found on the web.
    Thursday, January 26, 2012 11:52 PM
  • Another open question to all who are seeing this problem...  Do you have ECC memory in your system - i.e., if you had a RAM error would your system either correct it or halt? 

    I ask because mine has ECC, and before I changed out a pair of DIMMs that tested otherwise fine it did occasionally detect some faults.  I wasn't as aware that RAM errors were something to be concerned with before that experience.  Now I wouldn't have another system without ECC RAM.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    How can I determine this?  CPU-Z identifies my memory as DDR3, but I don't see any mention of ECC.
    Friday, January 27, 2012 12:04 AM
  • I just tried another set of copies, 12.9 GB in 234 files, this time with the Better Performance setting enabled.  They all compared OK.

    The write cache was doing its job too, as initially it reported the speed as 400 MB/sec, then it gradually went lower, ending up at about 30 MB/sec.  The drive activity light flashed for a good while longer after the copy was said to have completed as the written data was being flushed.  The Windows Resource Monitor reported the actual read-back speed of around 20 MB/sec, which is pretty decent considering it's USB 2.

    My thoughts tend to wondering whether you could have a hardware problem - e.g., a RAM error in a part of the memory that gets used for the cache - but I wouldn't rule out a difference in settings causing a bug or even particular software you run that I don't have.  My system is a Dell Precision T5400 workstation.  My antivirus is Avast Internet Security.  I've tweaked some system settings, but none that one would think should improve the reliability of data transfers...  I have disabled indexing on my system. 

    If you have any ideas about settings you want to compare to mine, please let me know.  I'm breathing a bit of a sigh of relief that I don't have the problem, but I feel bad for you and those who do.  If you can't trust a simple file transfer, then things are pretty bad!  I have to be able to rely on my system (as I'm sure do you).

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, January 27, 2012 12:25 AM
  •  

    I just ran another test, copying the same 1.4GB three times in succession.   Compares of the first two copies were successful, but the third yielded this difference list fairly quickly into the COMP operation:

    D:\tv\720p-mkv>comp test.mkv g:\test\test.mkv

    Comparing test.mkv and G:\test\test.mkv...
    Compare error at OFFSET 6660000
    file1 = E8
    file2 = 3
    Compare error at OFFSET 6660001
    file1 = 1D
    file2 = 1
    Compare error at OFFSET 6660002
    file1 = 9F
    file2 = EE
    Compare error at OFFSET 6660003
    file1 = 19
    file2 = D6
    Compare error at OFFSET 6660004
    file1 = 9F
    file2 = 44
    Compare error at OFFSET 6660005
    file1 = 60
    file2 = 7C
    Compare error at OFFSET 6660006
    file1 = 65
    file2 = 82
    Compare error at OFFSET 6660007
    file1 = C6
    file2 = D7
    Compare error at OFFSET 6660008
    file1 = C0
    file2 = F8
    Compare error at OFFSET 6660009
    file1 = 8F
    file2 = 7E
    10 mismatches - ending compare

    My system is up-to-date with required patches.    This is truly bizarre, and indeed, it is extremely frustrating to not be able to trust the OS's copy command.   As I've mentioned TeraCopy consistently produces flawless copies, but it's ridiculous to have to depend on a third-party application for basic functionality.

     

    Kipp

    Friday, January 27, 2012 12:33 AM
  • This problem occurs with NTFS drives (any make or size I have tested)


    Please list a sizes of the drives and their sector sizes (see a fsutil command above).

    Were the drives formatted to NTFS at the factory? If not what a program did you use?

    Friday, January 27, 2012 12:43 AM
    Answerer
  • Have to leave at the moment, but will follow up in the a.m. with sector size information.   All of my external drives are factory-formatted.  The issue occurs on all NTFS drives I have test, including Toshiba and Seagate models.  Thank you!

    Noel, what build of Windows 7 64-Bit Ultimate are you running?   Surely our systems must share the same copy and write-caching functionality as it applies to NTFS drives...    What could be different?

    Friday, January 27, 2012 12:55 AM
  • My drive came formatted, but I reformatted it myself anyway, just to make sure everything is perfectly compatible (I have no idea how they format the drives at Western Digital).

    C:\TEMP>fsutil fsinfo ntfsinfo G:

    NTFS Volume Serial Number :       0xc6b04293b0428a3f
    Version :                         3.1
    Number Sectors :                  0x00000000e8df7fff
    Total Clusters :                  0x000000001d1befff
    Free Clusters  :                  0x000000000a661356
    Total Reserved :                  0x0000000000093d60
    Bytes Per Sector  :               512
    Bytes Per Physical Sector :       <Not Supported>
    Bytes Per Cluster :               4096
    Bytes Per FileRecord Segment    : 1024
    Clusters Per FileRecord Segment : 0
    Mft Valid Data Length :           0x00000000050c0000
    Mft Start Lcn  :                  0x00000000000c0000
    Mft2 Start Lcn :                  0x0000000000000002
    Mft Zone Start :                  0x0000000010d4f5a0
    Mft Zone End   :                  0x0000000010d5bdc0
    RM Identifier:        8B9093D5-4A76-11E0-B6E4-001D9208583E

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, January 27, 2012 1:46 AM
  • When I start a CMD window I see:

     

    Microsoft Windows [Version 6.1.7601]
    Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

    C:\TEMP>

     

    You mentioned Toshiba and Seagate.  I'm using Western Digital.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, January 27, 2012 2:18 AM
  • OK. I agree that the problem may only occur on some specific hardware (maybe the USB ports of an specific Intel chipset). However, I think that my system is very different from yours, and I also suffer the corruption of big files (copying 7 to 18 Gb .mkv files, never tried with small files). My system is a DELL Vostro 1700 laptop with 4 GB RAM. The chipset is Intel GM965 with Nvidia 8600GT. Maybe we should compare the USB chipset ports used in our systems. The buffer overflow is possibily only happening on some USB port types. Noel, are your USB ports from Intel or any other manufacturer?

    The problem also happens in my job computer (DELL Precision T1500 with 16 GB RAM).

    Anyway, the IDE to USB conversion chipset in the external hard drives is possibily the guilty factor. One year ago I noticed that some USB hard drives were more prone to corrupt the big files. I do not remember the IDE/SATA chipset used but after testing some external USB drives I finally bought a external SATA drive (with a eSATA card controller) to fix my problems. I cannot get any corruption when copying to my eSATA drive, maybe because it is faster than the laptop's internal SATA hard drive and the buffer is barely used.

    I have tried with several USB 2.0 external drives and I always get the data corruption (always formatted at home to NTFS). It doesn't matter if I use my 320 GB transcend USB hard drive or a custom enclosure with an old 500 GB IDE hard drive. I also suspect that the corruption is more probable if you are actively using the PC while you copy the files.

     


    • Edited by ikerrg Friday, January 27, 2012 8:53 AM
    Friday, January 27, 2012 8:46 AM
  • On my Precision T5400:

    FYI, I was using my system pretty heavily when I performed the above successful tests.  I was going through all the processes to make a software release, including builds in Visual Studio, use of Tortoise subversion, LAN server access, FTP site access, browsing, etc...

    Do you see any errors in the Windows event logs when you get corruption?  I only rarely find they help with problems, but there's always a possibility and you should probably look.

    I'm going to find a really huge single file now and test it again...

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, January 27, 2012 1:06 PM
  • Noel, with regard to logged errors, this bug is extremely surreptitious and 100% transparent to Windows.   There are no structural issues as a result, an affected file has integrity with respect to the file system (with a byte count identical to the original), and no errors are detected or logged by Windows during the copy operation.  The latter was the very first thing I looked for after discovering that the issue can essentially be reproduced at will.
    Friday, January 27, 2012 1:22 PM
  • OK. I agree that the problem may only occur on some specific hardware (maybe the USB ports of an specific Intel chipset). However, I think that my system is very different from yours, and I also suffer the corruption of big files (copying 7 to 18 Gb .mkv files, never tried with small files). My system is a DELL Vostro 1700 laptop with 4 GB RAM. The chipset is Intel GM965 with Nvidia 8600GT. Maybe we should compare the USB chipset ports used in our systems. The buffer overflow is possibily only happening on some USB port types. Noel, are your USB ports from Intel or any other manufacturer?

    The problem also happens in my job computer (DELL Precision T1500 with 16 GB RAM).

    Anyway, the IDE to USB conversion chipset in the external hard drives is possibily the guilty factor. One year ago I noticed that some USB hard drives were more prone to corrupt the big files. I do not remember the IDE/SATA chipset used but after testing some external USB drives I finally bought a external SATA drive (with a eSATA card controller) to fix my problems. I cannot get any corruption when copying to my eSATA drive, maybe because it is faster than the laptop's internal SATA hard drive and the buffer is barely used.

    I have tried with several USB 2.0 external drives and I always get the data corruption (always formatted at home to NTFS). It doesn't matter if I use my 320 GB transcend USB hard drive or a custom enclosure with an old 500 GB IDE hard drive. I also suspect that the corruption is more probable if you are actively using the PC while you copy the files.

     


    This is very helpful, and I think you may be onto something here.  Yes, let's at least you, Noel and I compare USB particulars.  You mentioned that this problem occurs on two of your Windows 7 machines, so what have you found in common between the two?


    • Edited by KT-Retro Friday, January 27, 2012 2:48 PM
    Friday, January 27, 2012 1:26 PM
  • I'm going to find a really huge single file now and test it again...

    I turned on my test batch file earlier and forgot it running while I was doing other work.  It copied a 7 gigabyte file to my external MyBook drive and verified a perfect copy now 13 times, so I really do have a good system to compare with.

    I do hope we can get to the bottom of why this is happening on some systems and not others.  Please let me know what additional info you'd like from my system.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, January 27, 2012 2:42 PM


  • The above is from my Studio XPS 435T running 64-bit Windows 7 Professional...
    Friday, January 27, 2012 2:55 PM
  • I'm going to find a really huge single file now and test it again...

    I turned on my test batch file earlier and forgot it running while I was doing other work.  It copied a 7 gigabyte file to my external MyBook drive and verified a perfect copy now 13 times, so I really do have a good system to compare with.

    I do hope we can get to the bottom of why this is happening on some systems and not others.  Please let me know what additional info you'd like from my system.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options


    As the Device Manager output reveals (look back a few messages in the thread), your system and mine are using different USB controllers, so that may indeed be a factor...  I am anxious to learn the specifics of ikerrg's two systems, both of which are exhibiting the issue.

    • Edited by KT-Retro Friday, January 27, 2012 3:00 PM
    Friday, January 27, 2012 2:59 PM
  • When I start a CMD window I see:

     

    Microsoft Windows [Version 6.1.7601]
    Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

    C:\TEMP>

     

    You mentioned Toshiba and Seagate.  I'm using Western Digital.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options


    Another difference...    My Windows version is 6.1.7600
    Friday, January 27, 2012 3:02 PM
  • Another difference...    My Windows version is 6.1.7600


    Hm, I believe that means you haven't installed Windows 7 Service Pack 1.

    Do you not keep your system up to date via Windows Update?  SP1 was pushed out via Windows Update.

    In any case, I'd definitely advise doing a full System Image backup (ideally to media other than your external drive) and applying SP1.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, January 27, 2012 5:11 PM
  • Another difference...    My Windows version is 6.1.7600


    Hm, I believe that means you haven't installed Windows 7 Service Pack 1.

    Do you not keep your system up to date via Windows Update?  SP1 was pushed out via Windows Update.

    In any case, I'd definitely advise doing a full System Image backup (ideally to media other than your external drive) and applying SP1.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options


    I do apply updates weekly, but it would appear that I deferred on SP1, and will need to catch up.   In any event, this is a red herring, as ikerrg and others (including 25Albert) have noted that the issue still occurs under SP1.

    I believe we are on the right path concentrating on hardware differences between systems.  ikerrg has two which exhibit the issue (at least one of which is on SP1 by the way), I have one, and you have a system that doesn't.
    • Edited by KT-Retro Friday, January 27, 2012 6:22 PM
    Friday, January 27, 2012 6:19 PM
  • Hi,

    This is my home laptop USB hardware. It is in spanish, but you can see the USB type: Intel ICH8. The version of the controllers is SP1 (6.1.7601.17586).

    I cannot access my job computer until next monday, but I can confirm that it is also SP1 and it corrupted some files some time ago, so I decided to directly install Teracopy without more testing as I cannot tolerate any errors in my job data. When I have time I can do further testing, because testing is very time consuming.

    The chipset of the external USB hard drive (prolific, cypress, etc.) could also be the problem. However, I cannot see the chipset without physically opening the external enclosure.

    Best regards.

    • Edited by ikerrg Friday, January 27, 2012 8:10 PM
    Friday, January 27, 2012 8:07 PM
  • ...The chipset of the external USB hard drive (prolific, cypress, etc.) could also be the problem. However, I cannot see the chipset without physically opening the external enclosure.

    I suppose this is possible, but I have extensively tested the same collection of NTFS drives I am using, under Windows XP Professional, and have never seen XP produce a flawed file copy.   Does XP use "write-through" caching for external drives by default (essentially only minimal caching) as does Windows 7, or does XP use no caching whatsoever by default?

    Thank you very much for continuing to engage on this...  I suspect that some who have discovered the problem and its workaround (e.g. TeraCopy) have simply switched to the workaround and moved on.

    Out of curiosity, what background software are you running?   I am using Microsoft Security Essentials.

    Friday, January 27, 2012 9:02 PM
  • Would one have to open a paid support case with  Microsoft in order to bring awareness of this issue to the Windows 7 OS team?   The issue is at least a year and a half old at this point...    That you (ikerrg) see it on two separate Windows 7 machines is a good indication that it is far more widespread than the handful of accounts of the issue here.  The data alteration aspect of this issue is so stealthy that it's likely to have impacted far more users who aren't even aware that affected files are compromised.
    Friday, January 27, 2012 9:13 PM
  • Would one have to open a paid support case with  Microsoft in order to bring awareness of this issue to the Windows 7 OS team?  

    There is a way to send feedback without any payments but I'd suggest (at least) to make a list of hardware with this problem first.

    It is very hard to convince developers to investigate a problem without repro steps. Especially if a problem may be related with hardware.

    Friday, January 27, 2012 9:36 PM
    Answerer
  •  Does XP use "write-through" caching for external drives by default (essentially only minimal caching) as does Windows 7, or does XP use no caching whatsoever by default?
    XP spits large file operations to 64KB blocks and Windows 7 to 1MB. This may be a difference.
    Friday, January 27, 2012 9:38 PM
    Answerer
  • Here is the fsutil information for two of the makes and models of drives on which I can reproduce this issue with relative ease:

    C:\Windows\system32>fsutil fsinfo ntfsinfo g:
    NTFS Volume Serial Number :       0x4e48e75448e738fb
    Version :                         3.1
    Number Sectors :                  0x000000003a384c01
    Total Clusters :                  0x0000000007470980
    Free Clusters  :                  0x0000000000628e4f
    Total Reserved :                  0x0000000000000f00
    Bytes Per Sector  :               512
    Bytes Per Cluster :               4096
    Bytes Per FileRecord Segment    : 1024
    Clusters Per FileRecord Segment : 0
    Mft Valid Data Length :           0x0000000001ec8000
    Mft Start Lcn  :                  0x0000000000000002
    Mft2 Start Lcn :                  0x0000000003a384c0
    Mft Zone Start :                  0x0000000000eb4c00
    Mft Zone End   :                  0x0000000000ebf5a0
    RM Identifier:        273AE4FE-1B1E-11DF-9968-0023AEE6DF0F

    C:\Windows\system32>fsutil fsinfo ntfsinfo q:
    NTFS Volume Serial Number :       0x963410bc3410a173
    Version :                         3.1
    Number Sectors :                  0x0000000038cfd6ef
    Total Clusters :                  0x000000000719fadd
    Free Clusters  :                  0x000000000362eed6
    Total Reserved :                  0x0000000000002580
    Bytes Per Sector  :               512
    Bytes Per Cluster :               4096
    Bytes Per FileRecord Segment    : 1024
    Clusters Per FileRecord Segment : 0
    Mft Valid Data Length :           0x0000000000280000
    Mft Start Lcn  :                  0x00000000000c0000
    Mft2 Start Lcn :                  0x0000000000000010
    Mft Zone Start :                  0x00000000000c0280
    Mft Zone End   :                  0x00000000000cc840
    RM Identifier:        50274B03-FE19-11D5-9F14-000C296E3B52

    Friday, January 27, 2012 10:42 PM
  • Would one have to open a paid support case with  Microsoft in order to bring awareness of this issue to the Windows 7 OS team?  

    There is a way to send feedback without any payments but I'd suggest (at least) to make a list of hardware with this problem first.

    It is very hard to convince developers to investigate a problem without repro steps. Especially if a problem may be related with hardware.


    I have just posted the fsutil info for two different makes and models of drives on which Windows 7 will intermittently create altered copies of large files.   My machine profile is near the top of the thread.   Is there more that I can provide?

    So far, we know that some W7 64-bit systems with both Intel ICH8 and ICH10 Family USB Universal Host Controllers are impacted, but we don't know that all W7 64-bit systems with these controllers are.

    Thank you Igor.

    • Edited by KT-Retro Friday, January 27, 2012 10:50 PM
    Friday, January 27, 2012 10:44 PM
  • Here is the fsutil information for two of the makes and models of drives on which I can reproduce this issue with relative ease:


    Thank you. Problem is not related with a sector size.

     

     

    Friday, January 27, 2012 10:49 PM
    Answerer
  •    Is there more that I can provide?  


    The exact models of external drives. Can you try these drive with another computers? Do you have a possibility to try the drives with USB High Speed cable? But the test will be slow unfortunately.

    Unfortunately it will not be sufficient information I'm afraid. But let's start to collect it. 

     


    Friday, January 27, 2012 10:59 PM
    Answerer
  • Igor, are you with MSFT?    There was a company employee (Aaron) in the thread early on, but he does not appear to still be engaged, and I do not see any way to contact him.

    I would like to take whatever steps are necessary to bring this to the attention of Microsoft and to seek a resolution, if possible, in the form of a hotfix.

    Thank you.

    -Kipp

    Friday, January 27, 2012 11:01 PM
  • Igor, are you with MSFT?   


    No, MVP cannot be a MSFT empoyee. See mvp.support.microsoft.com please.

    Friday, January 27, 2012 11:11 PM
    Answerer
  •    Is there more that I can provide?  


    The exact models of external drives. Can you try these drive with another computers? Do you have a possibility to try the drives with USB High Speed cable? But the test will be slow unfortunately.

    Unfortunately it will not be sufficient information I'm afraid. But let's start to collect it. 

     


    One is a 500MB Seagate FreeAgent Go 9KW2AH-500, and the other is a 750MB Toshiba 593400-C.   The drives work flawlessly with two other machines I use regularly, but both are running XP Professional.  It would seem to be quite the coincidence if the problem lies with the drives (unless as was suggested earlier, it is because of a shared chipset that does not work properly with Windows 7 64-bit drivers and write-caching).   I have another Toshiba USB drive, but it is formatted FAT-32, and large-file copies to it under W7 are consistently perfect.  I am not familiar with a "USB High Speed" cable...  (?)
    Friday, January 27, 2012 11:15 PM
  • Note ikerrg's experience and two additional types of external drives...

    Quoting ikerrg:  "I have tried with several USB 2.0 external drives and I always get the data corruption (always formatted at home to NTFS). It doesn't matter if I use my 320 GB transcend USB hard drive or a custom enclosure with an old 500 GB IDE hard drive. I also suspect that the corruption is more probable if you are actively using the PC while you copy the files."

    Friday, January 27, 2012 11:21 PM
  • Aaron,

    Are you still available in this thread?    I am no closer to a solution on this problem as I was four months ago when I started the thread.

    -Kipp

    Saturday, January 28, 2012 12:00 AM
  • Here is another potential clue...     Up until February 2010, I was running the Windows 7 RC (64-bit version) on my system.   Prior to upgrading from the RC, I copied a significant number of files, including some very large video files I had mastered, to one of my NTFS drives.    I still have these copies, and they are perfect.    That is, the Windows 7 RC did not alter/corrupt the copies of files, yet the released Windows 7 does.    What changed between RC and released version that would have impacted write operations to NTFS external drives??
    Saturday, January 28, 2012 12:28 AM
  • USB hign speed cables do not support full USB 2.0 speed and work at USB 1 speed.

    How do you connect hard drives, with additional cables, to the front of computer or to the back?

    Since Win7RC hardware became older, this may be a reason too. Can you try a new hard drives?

    Can you try a copying a lot of small files (25 GB total) and verify them?



    Saturday, January 28, 2012 8:33 AM
    Answerer
  •  Does XP use "write-through" caching for external drives by default (essentially only minimal caching) as does Windows 7, or does XP use no caching whatsoever by default?
    XP spits large file operations to 64KB blocks and Windows 7 to 1MB. This may be a difference.


    That is a good point!!! I remember several years ago that I started having problems when comparing identical files in USB drives while using CDCheck (http://kvipu.com/CDCheck/) I noticed that if I modified the default read buffers (2048 KB) in the configuration of the program, I could always get a perfect reading. I contacted the developer of CDCheck and below you can see my email and his response:

    Hi!
    Please try to use the /normalread parameter when starting CDCheck. This prevents unbuffered reading which is sometimes the cause of this kind of problems (usually due to some driver misbehaviour).
    Best regards,
    Mitja Perko

    On Fri, May 15, 2009 at 9:13 AM, Iker Rodríguez wrote:

    Hi,

    I've seen that large files compared between an external USB 2.0 hard drive and the computer's internal hard drive report comparison errors with CDCheck 3.1.14.0. However, fc console command does not report differences. I've seen that large buffers are responsible of this error (read buffer=1024 circular buffer=2048) If these values are made smaller (read buffer=32 circular buffer=64) the comparison is good. This does not happen when comparing between internal hard drives. Please, check this bug, is critical!

    Best regards.

    I do not remember if I was using Vista or Win 7 at that time, but Mitja Perko thought that a driver misbehaviour could be the cause (this includes a hardware problem). I remember that the buffer size did not affect the comparison quality when comparing between internal drives (or fast eSATA drives). I think that it was the starting point of all the data corruption suspicions, and that's why I stopped using USB 2.0 drives as a hard drive extension for my laptop. But I still use USB 2.0 drives for long term storing of files, and there it is where I noticed the problems.

    Maybe Microsoft can provide us with a low-level utility to check the quality of data transmission when using different buffers on our affected systems.

    I am also using security essentials, but you can disable the real time protection and see if the problems still occur.

     

    Saturday, January 28, 2012 11:52 AM
  • USB hign speed cables do not support full USB 2.0 speed and work at USB 1 speed.

    How do you connect hard drives, with additional cables, to the front of computer or to the back?

    Since Win7RC hardware became older, this may be a reason too. Can you try a new hard drives?

    Can you try a copying a lot of small files (25 GB total) and verify them?



    My (two) Toshiba Canvio 750GB drives are newer than my system and newer than the final release of Windows 7.    For these drives as well as the Seagates, I use the manufacturer-supplied short cable only, and I have experimented connecting them both to the front (top) as well as to the rear USB ports of my Dell XPS.   Data alteration occurs in either case on roughly 30% of copies involving large files.  I will experiment with a batch of small files and post results.
    Saturday, January 28, 2012 2:00 PM
  •  Does XP use "write-through" caching for external drives by default (essentially only minimal caching) as does Windows 7, or does XP use no caching whatsoever by default?
    XP spits large file operations to 64KB blocks and Windows 7 to 1MB. This may be a difference.


    That is a good point!!! I remember several years ago that I started having problems when comparing identical files in USB drives while using CDCheck (http://kvipu.com/CDCheck/) I noticed that if I modified the default read buffers (2048 KB) in the configuration of the program, I could always get a perfect reading. I contacted the developer of CDCheck and below you can see my email and his response:

    Hi!
    Please try to use the /normalread parameter when starting CDCheck. This prevents unbuffered reading which is sometimes the cause of this kind of problems (usually due to some driver misbehaviour).
    Best regards,
    Mitja Perko

     On Fri, May 15, 2009 at 9:13 AM, Iker Rodríguez wrote:

    Hi,

    I've seen that large files compared between an external USB 2.0 hard drive and the computer's internal hard drive report comparison errors with CDCheck 3.1.14.0. However, fc console command does not report differences. I've seen that large buffers are responsible of this error (read buffer=1024 circular buffer=2048) If these values are made smaller (read buffer=32 circular buffer=64) the comparison is good. This does not happen when comparing between internal hard drives. Please, check this bug, is critical!

    Best regards.

    I do not remember if I was using Vista or Win 7 at that time, but Mitja Perko thought that a driver misbehaviour could be the cause (this includes a hardware problem). I remember that the buffer size did not affect the comparison quality when comparing between internal drives (or fast eSATA drives). I think that it was the starting point of all the data corruption suspicions, and that's why I stopped using USB 2.0 drives as a hard drive extension for my laptop. But I still use USB 2.0 drives for long term storing of files, and there it is where I noticed the problems.

    Maybe Microsoft can provide us with a low-level utility to check the quality of data transmission when using different buffers on our affected systems.

    I am also using security essentials, but you can disable the real time protection and see if the problems still occur.

     

    I want to be clear that, for the problem I am describing, data alteration is occurring during the write operation and is permanent.   My testing has conclusively confirmed this, and that there is no misbehavior of comparison applications.  That is, I have used fc, comp and WinDiff, and also, I have copied the affected files back to the hard drive and the comparison results remain unchanged.
    Saturday, January 28, 2012 2:06 PM
  • I agree with KT-Retro, the problem is permanent and during write operation and I have experimented it many times when copying big files. My last post only shows that a big buffer can also break the read transferences in USB 2.0 drives (when using CDCheck or maybe other programs), so there is a big possibility that a write operation is also affected by a big OS buffer. That is why FAT32 drives are not affected when writting to them.

     

    Saturday, January 28, 2012 3:57 PM
  • Just trying to summarize, here, to see whether we have a complete set of information for comparison:

    Please correct this list if I have gotten anything wrong and augment it with info not present...

    Combinations on which the problem is seen:

    • (Iker's) Dell Vostro 1700 laptop, Intel(R) ICH8 USB controller, external USB drive (Transcend 320 GB)
    • (Iker's) Dell Vostro 1700 laptop, Intel(R) ICH8 USB controller, external USB drive (another model?  wording says "several")
    • (Iker's) Dell Vostro 1700 laptop, Intel(R) ICH8 USB controller, external 500 GB SATA drive (model?) in enclosure (model?)
    • (Iker's) Dell Precision T1500, (USB controller?), external USB drive (model(s)?)
    • (Kipp's) Dell Studio XPS 435T, Intel(R) ICH10 USB controller, external USB drive (model(s)?)
    • (25Albert's) computer (model?), eSATA controller (model?), eSATA drive (model?)

    The problem is not seen on these combinations: 

    • (Iker's) Dell Vostro 1700 laptop, eSATA card controller (model?), external SATA drive (model?)
    • (Noel's) Dell Precision T5400, Intel(R) 631xESB/6321ESB/3100 USB controller, external WD MyBook 1140 (2 TB)

    It would be good to get the specifics for (model?) in the above list.

    There's too little information to determine any pattern, really, but I notice that the only Western Digital external drive listed (mine) seems to work, as well as the failure being seen with all the Intel ICHx controllers listed.

     

    A complete shot in the dark...  Early in Windows 7's release history (prior to SP1) there was a very specific NTFS bug having to do with Atomic Oplocks.  One manifestation was that, when Indexing was running and certain other file operations attempted, the system could report a drive corruption problem and schedule CHKDSK to be run.  A Microsoft-approved workaround at the time was to disable indexing, and subsequently Microsoft created a hotfix and rolled it into Service Pack 1.  But I wonder if there could be a lesser bug related to indexing...  One key thing is that I have completely disabled indexing on my system.  I thought of this because it's being said to be NTFS-specific, and reported more likely to happen if there's other system activity.

    If you'd like to try to disable indexing on your system to see if it works around this problem, I can provide the steps here to do so (it's not trivial).  Let me know.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Saturday, January 28, 2012 5:44 PM
  • This is some information about my systems:

    One of the external hard drives is a Transcend model TS320GSJ25M. I can confirm corrupted big files when copying to this drive using my Vostro 1700 laptop.

    The other external hard drive is an external enclosure TOOQ model TQE-3518 where I have installed a Seagate IDE,500 GB,5.2Krpm hard drive (I can give you the exact model if required). I can also confirm corrupted big files when copying to this drive using my Vostro 1700 laptop.

    My job's computer (Precision T1500) uses a "Intel(R) 5 Series/3400 Series Chipset Family USB Enhanced Host Controller - 3B34" and 3B3C. I have not thoroughly tested the errors on this system because I cannot stay "playing" in my job. That's why I directly installed TeraCopy on it.

     

    The correctly working drive is a LaCIE d2 Quadra 1TB eSATA on the Vostro 1700 laptop, connected using a eSATA expansion ExpressCard from DeLOCK, model 61382 (it uses a SiI3132 SATA300 chipset from Silicon Image). It is faster than internal hard drives of the Vostro laptop so the copy errors cannot be produced (the NTFS buffer is almost no used). Anyway, I think that eSATA drives are not affected because they are exactly the same as internal drives (eSATA controllers are connected to a PCI Express bus in workstations and new laptops and they are identical to mainboard SATA controllers).

    All the drives are NTFS formatted.

     

    Monday, January 30, 2012 8:33 AM
  • How do those of us experiencing this serious problem...on a variety of Intel USB chipsets...and on a variety of NTFS-formatted external drives...seek a resolution from Microsoft?    The common element here is the Windows 7 64-bit operating system, and it's there that this problem almost certainly lies.
    Tuesday, January 31, 2012 6:25 PM
  • How do those of us experiencing this serious problem...on a variety of Intel USB chipsets...and on a variety of NTFS-formatted external drives...seek a resolution from Microsoft?    The common element here is the Windows 7 64-bit operating system, and it's there that this problem almost certainly lies.


    And for sure there are many other people that have not noticed the errors!!! If you usually copy big video files to your USB drives it is difficult to notice the corrupted data because the video data is well designed for corruption and transfer errors. Only a binary comparison (which is not usually done by normal users) can show the problem.

     

    Tuesday, January 31, 2012 6:37 PM
  • How do those of us experiencing this serious problem...on a variety of Intel USB chipsets...and on a variety of NTFS-formatted external drives...seek a resolution from Microsoft?    The common element here is the Windows 7 64-bit operating system, and it's there that this problem almost certainly lies.


    And for sure there are many other people that have not noticed the errors!!! If you usually copy big video files to your USB drives it is difficult to notice the corrupted data because the video data is well designed for corruption and transfer errors. Only a binary comparison (which is not usually done by normal users) can show the problem.

     

    You are absolutely correct.   This is a very stealthy, insidious issue whose damaging effect may for some go undetected by many for years, if ever.   Most users are not going to go to the trouble to test their systems based on our reports, and many users don't work with large files at all...but as another poster has noted, it can also impact large transfers, for example, a large batch of small files.   In any event, I can easily reproduce the issue and demonstrate that, on roughly 30% of copy operations of a file even as small as 1GB, data will be altered if the standard Windows 7 copy mechanism is used.

    What's also troubling is that this thread is now so large that it takes forever to open it...at least on Firefox.   It's doubtful any casual forum visitors, or Microsoft, for that matter, is paying any attention to it anymore.

    Tuesday, January 31, 2012 6:56 PM
  •   The common element here is the Windows 7 64-bit operating system, and it's there that this problem almost certainly lies.


    isn't Dell a common element too?

    What is the size of corrupted data? Do the corruption occurs once or several times for a file? Are the offsets similar an all cases or very different?

    Tuesday, January 31, 2012 7:07 PM
    Answerer
  • Igor, it works fine on my Dell Precision T5400 - no corruption.   Running Windows 7 x64 as well.

    Honestly, for those who have the problem I'm sure it's frustrating, and it would be nice to have Microsoft on your side, but I'm not convinced yet that it's Microsoft's problem.  Could it be a common part in many (but not, for example, Western Digital) external drives?

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Tuesday, January 31, 2012 7:20 PM
  • Noel, yes, I see you have Dell.

    Unfortunately I do not see enough information for bug report.

    KT-Retro, ikerrg, do you have box version or Dell's OEM?

    Tuesday, January 31, 2012 7:26 PM
    Answerer
  • Noel, yes, I see you have Dell.

    Unfortunately I do not see enough information for bug report.

    KT-Retro, ikerrg, do you have box version or Dell's OEM?

    I upgraded from Windows 7 RC to full Windows 7. Regarding the nature of the altered data, the same file can be copied multiple times, and roughly 70% of the time, copies will be clean and identical. In the other instances, data alteration seems to occur in one or more large chunks randomly dispersed throughout the copied file.

    • Edited by KT-Retro Tuesday, January 31, 2012 7:48 PM
    Tuesday, January 31, 2012 7:44 PM
  •   The common element here is the Windows 7 64-bit operating system, and it's there that this problem almost certainly lies.


    isn't Dell a common element too?

    What is the size of corrupted data? Do the corruption occurs once or several times for a file? Are the offsets similar an all cases or very different?

    The number of chunks of corrupted/altered data varies from copy to copy (including of the same file) from none to two or three, but I do not know if the size of the corrupted areas each vary. I haven't scrutinized it that closely.

    P.S. - I have had to block access to i3.social.s-msft.com in order to continue reading/replying to this thread. I cannot access it all under Internet Explorer.
    Tuesday, January 31, 2012 7:53 PM
  • Noel, yes, I see you have Dell.

    Unfortunately I do not see enough information for bug report.

    KT-Retro, ikerrg, do you have box version or Dell's OEM?

    Igor, are you not able to reproduce this problem? If the problem were occurring on your system, what specific information would you collect that would be considered sufficient to submit a bug report? I will go through such a list of necessary diagnostic steps thoroughly in order to move this forward. Thanks.
    Tuesday, January 31, 2012 7:55 PM
  • I upgraded from Windows 7 RC to full Windows 7.


    Unfortunately this is unsupported scenario. Any bug report will be closed immediately.
    Tuesday, January 31, 2012 7:57 PM
    Answerer
  • I upgraded from Windows 7 RC to full Windows 7.


    Unfortunately this is unsupported scenario. Any bug report will be closed immediately.
    Apologies, as I misused the term "upgraded." No, I backed up my system and then INSTALLED Windows 7 64-bit. Again, sorry for the misinformation.
    Tuesday, January 31, 2012 8:00 PM
  • I upgraded from Windows 7 RC to full Windows 7.


    Unfortunately this is unsupported scenario. Any bug report will be closed immediately.
    I reviewed my e-mails from February 2010. Even Microsoft misused the term "upgrade" in their reminders to users about moving from RC to full W7, but no, all of my correspondence confirms that I performed a 100% CLEAN INSTALL of Windows 7 at some point in February 2010...prior to the forced every-two-hour shutdown of the RC which was to begin on March 1, 2010. So now that we have cleared this up, please advise on how I can go about substantiating a bug report. Thanks.
    Tuesday, January 31, 2012 8:08 PM
  • Igor, are you not able to reproduce this problem?


    I did not try. First, I am very busy and will be busy at least for a week.

    Second, motherboard in my main computer is dying slowly so the test can prove nothing.

    Third, I have no free external hard drive for tests.

    And at this moment I do not see a nesessity to do any tests. I'd like to be at least 30 % sure the problem is in Windows.

    About information: what are the exact sizes of corrupted blocks: 512 bytes, 4KB, 64KB, 1MB?

    512 seems to be a hardware problem, 1MB very likely be a Windows problem.

    Tuesday, January 31, 2012 8:26 PM
    Answerer
  • Igor, it works fine on my Dell Precision T5400 - no corruption.   Running Windows 7 x64 as well.

    Honestly, for those who have the problem I'm sure it's frustrating, and it would be nice to have Microsoft on your side, but I'm not convinced yet that it's Microsoft's problem.  Could it be a common part in many (but not, for example, Western Digital) external drives?

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Noel, is your MyBook drive considered "Advanced Format?" If so, that might account for you not observing the issue if it is an OS issue as I believe it to be. In any case, do you have access to an older, say vintage 2008 or 2009 NTFS USB drive for testing? Thanks
    Tuesday, January 31, 2012 8:29 PM
  • Igor, are you not able to reproduce this problem?


    I did not try. First, I am very busy and will be busy at least for a week.

    Second, motherboard in my main computer is dying slowly so the test can prove nothing.

    Third, I have no free external hard drive for tests.

    And at this moment I do not see a nesessity to do any tests. I'd like to be at least 30 % sure the problem is in Windows.

    About information: what are the exact sizes of corrupted blocks: 512 bytes, 4KB, 64KB, 1MB?

    512 seems to be a hardware problem, 1MB very likely be a Windows problem.

    Will a "FC" or "COMP" comparison reveal the corrupted block size? (WinDiff only reports that files are different) Thank you
    Tuesday, January 31, 2012 8:31 PM
  • fc /b file1 file2 shows all mismatches one by one.

    Tuesday, January 31, 2012 8:37 PM
    Answerer
    • (25Albert's) computer (model?), eSATA controller (model?), eSATA drive (model?)

    HP Z800, Rocketraid 2314 eSATA controller with latest and previous driver, various G-RAID 2TB or 4TB external drives.

    I am told that the problem might actually only occurs when copying between this and an HP (? or AVID?) external RAID. Not between internal disks and eSATA disks. (Nor between the big external RAID and internal disks).

    The external RAID is VERY fast. So how does Win7x64 treat eSATA differently than internal SATA, and how would a big speed difference between source and destination impact only eSATA and not the plain internal SATA.

    • Edited by 25Albert Tuesday, January 31, 2012 9:32 PM
    Tuesday, January 31, 2012 9:30 PM
  • With the variety of USB chipsets and the variety of external drives on which this is occurring, it seems very unlikely that this is a hardware issue. Also, there is not yet enough evidence, but it seems that the 64-bit version of Windows is a common element. I have access to additional Windows 7 systems at my job, but unfortunately they are 32-bit... I will test them anyway.
    Tuesday, January 31, 2012 9:33 PM
    • (25Albert's) computer (model?), eSATA controller (model?), eSATA drive (model?)

    HP Z800, Rocketraid 2314 eSATA controller with latest and previous driver, various G-RAID 2TB or 4TB external drives.

    I am told that the problem might actually only occurs when copying between this and an HP (? or AVID?) external RAID. Not between internal disks and eSATA disks. (Nor between the big external RAID and internal disks).

    The external RAID is VERY fast. So how does Win7x64 treat eSATA differently than internal SATA, and how would a big speed difference between source and destination impact only eSATA and not the plain internal SATA.

    Only an external RAID? Did you not indicate earlier that you experience the corruption copying to NTFS-formatted external USB drives? If not, can you test to a standalone NTFS drive?
    • Edited by KT-Retro Tuesday, January 31, 2012 9:38 PM
    Tuesday, January 31, 2012 9:37 PM
  • Noel, is your MyBook drive considered "Advanced Format?" If so, that might account for you not observing the issue if it is an OS issue as I believe it to be. In any case, do you have access to an older, say vintage 2008 or 2009 NTFS USB drive for testing? Thanks

    I don't know if this drive is "Advanced Format"...  In what context does that phrase have meaning?  I bought it in October of last year, so it's pretty new.  I did find this:

    http://wdc.custhelp.com/app/answers/detail/a_id/3876

    From the above page:

    Starting in late 2009, the My Book, WD Elements, and other WD External Storage products began shipping with Advanced Format drives.

    I do have a smaller, older 1TB MyBook drive plugged into another system (running Vista x64, not Windows 7 x64), and it's from late 2008.  I will relocate it to see if I can reproduce the problem, but I doubt that I will because that drive did service on my Windows 7 system for 18 months, and in that time I'm sure I would have noticed a failure in a copy.  I did many (large) file operations in both directions and I would have definitely noticed a failure such as you describe.  But I will test it for completeness.

    I mentioned before, but I'll say again:  I formatted these drives myself from Windows 7.  I didn't want to rely on the factory formatting.  Have any of the problems here been seen on user-formatted drives?  I've lost track in the big thread above...  Edit:  Yes, I see Iker formatted his own drives.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options




    • Edited by Noel Carboni Wednesday, February 01, 2012 12:50 AM
    Wednesday, February 01, 2012 12:41 AM
  • Noel, is your MyBook drive considered "Advanced Format?" If so, that might account for you not observing the issue if it is an OS issue as I believe it to be. In any case, do you have access to an older, say vintage 2008 or 2009 NTFS USB drive for testing? Thanks

    I don't know if this drive is "Advanced Format"...  In what context does that phrase have meaning?  I bought it in October of last year, so it's pretty new.  I did find this:

    http://wdc.custhelp.com/app/answers/detail/a_id/3876

    From the above page:

    Starting in late 2009, the My Book, WD Elements, and other WD External Storage products began shipping with Advanced Format drives.

    I do have a smaller, older 1TB MyBook drive plugged into another system (running Vista x64, not Windows 7 x64), and it's from late 2008.  I will relocate it to see if I can reproduce the problem, but I doubt that I will because that drive did service on my Windows 7 system for 18 months, and in that time I'm sure I would have noticed a failure in a copy.  I did many (large) file operations in both directions and I would have definitely noticed a failure such as you describe.  But I will test it for completeness.

    I mentioned before, but I'll say again:  I formatted these drives myself from Windows 7.  I didn't want to rely on the factory formatting.  Have any of the problems here been seen on user-formatted drives?  I've lost track in the big thread above...  Edit:  Yes, I see Iker formatted his own drives.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options




    Please keep in mind that the copy operation itself never indicates a failure. Unless you suspect an issue, you might never notice it. I used my W7 64-bit system for a year and a half before I stumbled onto the problem last September, then was able to reproduce it (to this day) at will.
    Wednesday, February 01, 2012 1:48 AM
  • I understand completely.  Please rest assured that the way I use these drives and with the file types I use I *DEFINITELY* would have detected a failure to copy data at 100% accuracy.

    Plus, I have now tested my drives thoroughly, with many GB of copies / compares.  I'm just not getting seeing any data corruption at all in the copy operations.  I just copied a bunch of GB of data to my ca. 2008 1 TB MyBook and compared it back without error.  The issue you're having sure makes me glad I have drives that work right.

    I know it involves spending some $$$, but you might want to try getting yourself a new USB drive.  They're not hugely expensive, and it might just get rid of the problem for you.  Hey, if it fails in the same way as your old one, print the comparison results out and take it back, claiming it doesn't work right.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Wednesday, February 01, 2012 3:06 AM
  • I understand completely.  Please rest assured that the way I use these drives and with the file types I use I *DEFINITELY* would have detected a failure to copy data at 100% accuracy.

    Plus, I have now tested my drives thoroughly, with many GB of copies / compares.  I'm just not getting seeing any data corruption at all in the copy operations.  I just copied a bunch of GB of data to my ca. 2008 1 TB MyBook and compared it back without error.  The issue you're having sure makes me glad I have drives that work right.

    I know it involves spending some $$$, but you might want to try getting yourself a new USB drive.  They're not hugely expensive, and it might just get rid of the problem for you.  Hey, if it fails in the same way as your old one, print the comparison results out and take it back, claiming it doesn't work right.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    A new drive? This is occurring on five different USB drives, two relatively new, all of two different manufacturers, and three of similar but not identical models. No, I do not need to buy a new USB drive. There is a defect in the support operating system software, and this is what the evidence points to.
    Wednesday, February 01, 2012 3:25 AM
  • All I know is that if I met someone who was unable to reproduce the problem I was having I'd try to duplicate their setup.

    We need to face the fact that this is not open software, so outside of futzing with it long enough to divine a pattern, all we as users can realistically do is mix and match parts and tweak settings until we find a combination that works. 

    Your plight reminds me of an expensive nVidia video card I could never get to work quite right a long while back (in the first days of Aero with Vista).  My whole system was just plain flaky.  I finally just gave up and bought a new video card, and I was as happy as a clam to find out that my system got tight and reliable after that.  Turned my whole impression of Vista around.  I had wasted thousands of dollars in time to avoid paying a hundred dollars for a new part.

    Note that we haven't really discussed the mix of software that's running on your system where you see the problems.  It's entirely possible something only indirectly related to the disk is involved...  Your anti-malware software comes to mind.  What software do you have in common with the others having the same problem?  You guys should list all the packages and see what you have in common, if anything.

    Unfortunately, as pointed out earlier in this thread, it's practically impossible to engage Microsoft.  And even if you did get them to write up a problem report, they might just up and close it again...  Looking for info on a Visual Studio "invalid pointer" problem my chief engineer was having earlier today I browsed into a fair number of internal Microsoft problem reports that were just closed because the users had found workarounds, or (my favorite) after having gone to the trouble of recording a video clearly showing the problem, because he didn't provide additional info within 6 days, they just closed the problem report on one guy without any resolution whatsoever.  Sheesh!

    Anyway, I don't know what else I can provide to help you with this, but if you think of anything drop a note in this thread.  I'll continue to monitor it.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Wednesday, February 01, 2012 6:01 AM
  • I have tested at my employer on a Windows 7 64-bit system with an AMD processor and using the "OpenHCD USB Controller," using one of the same NTFS-formatted drives I use at home. All comparisons were clean...quite slow, but clean. The drive was, as on my home system, set for "Optimize for Removal" (write-through caching). I realize that there are a number of factors potentially at play with regard to the data corruption issue, but this test would at least appear to rule out the drive itself...and by deduction, the drives I am using in general. In other words, this test demonstrates that it IS possible for Windows 7 64-bit to successfully create clean copies of files on a specific drive. Also, the driver date & version for the external drive are identical to those in use on my home system (6/21/2006, 6.1.7600.16385) The most obvious difference in the two systems is that one has an Intel motherboard and USB controller chipset, and the other has an AMD motherboard and OpenHCD USB.
    Wednesday, February 01, 2012 9:17 PM
  • I have created a test file 1.4GB in size which contains nothing but a repeating alphabet string (A-Z). My intention is to scrutinize the data alteration to determine if the chunk of erroneous file data in a corrupted copy is random or if it follows a pattern. This should help in narrowing down the nature of the write cache mismanagement.

    An important clue in the root cause of this problem lies in the fact that the workaround to this issue...the third-party TeraCopy, by default totally bypasses the write cache, unless the program's option to use the system write cache is enabled via checkbox...in which case the same data corruption issue presents itself with the same frequency. This clearly points to write cache management, by whatever system-level entities are involved.

    I will report on my testing with the patterned data file later today. (I am not currently at the system which exhibits the corruption issue)

    • Edited by KT-Retro Thursday, February 02, 2012 6:52 PM
    Thursday, February 02, 2012 6:27 PM
  • I think you may be fixating on the write cache connection a little too strongly...  The thing is, the Teracopy documentation states this:

    Error recovery In case of a copy error, TeraCopy will try several times to recover

    Ask yourself:  What do they mean by "copy error"?

    THIS IS ALL JUST A THEORY:

    Could the TeraCopy documentation mean that the data is written, then read back and compared, and if it doesn't match, TeraCopy writes it again, repeating until it DOES match.  Now imagine how that activity could be affected by enabling/disabling the use of cache...

    1.  Without cache enabled:  Teracopy does an actual disk write, an actual disk read, and compares what was written to what was supposed to be written, in its own RAM buffer.  Upon detecting an error writing to the disk, it repeats this process until what's read back matches what's written.

    2.  With cache enabled:  Teracopy does a write (into cache), a read (back from cache), and compare.  There is no problem with the system's RAM, so the compare succeeds and Teracopy moves on.  Then later there's an error writing from the cache to the disk drive itself and the corruption remains permanent, because Teracopy isn't there to try again.

    The use of cache changes the sequencing of the process so as to make a read-after-write check succeed (because it's all done from RAM), when in fact corrupted data is actually written to the disk.

    The problem could be in the way the Windows 7 disk driver is controlling the disk, and Teracopy is just hiding the problem by retrying when the data isn't written correctly.  Does TeraCopy have the ability to list statistics (e.g., how many writes it had to retry in the process of performing the advertised recovery)?

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, February 03, 2012 12:52 AM
  • I think you may be fixating on the write cache connection a little too strongly...  The thing is, the Teracopy documentation states this:

    Error recovery In case of a copy error, TeraCopy will try several times to recover

    Ask yourself:  What do they mean by "copy error"?

    THIS IS ALL JUST A THEORY:

    Could the TeraCopy documentation mean that the data is written, then read back and compared, and if it doesn't match, TeraCopy writes it again, repeating until it DOES match.  Now imagine how that activity could be affected by enabling/disabling the use of cache...

    1.  Without cache enabled:  Teracopy does an actual disk write, an actual disk read, and compares what was written to what was supposed to be written, in its own RAM buffer.  Upon detecting an error writing to the disk, it repeats this process until what's read back matches what's written.

    2.  With cache enabled:  Teracopy does a write (into cache), a read (back from cache), and compare.  There is no problem with the system's RAM, so the compare succeeds and Teracopy moves on.  Then later there's an error writing from the cache to the disk drive itself and the corruption remains permanent, because Teracopy isn't there to try again.

    The use of cache changes the sequencing of the process so as to make a read-after-write check succeed (because it's all done from RAM), when in fact corrupted data is actually written to the disk.

    The problem could be in the way the Windows 7 disk driver is controlling the disk, and Teracopy is just hiding the problem by retrying when the data isn't written correctly.  Does TeraCopy have the ability to list statistics (e.g., how many writes it had to retry in the process of performing the advertised recovery)?

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options


    By "copy error," TeraCopy is referring to a detectable lower-level disk error signaled by the hardware.  TeraCopy does not read as it writes, but rather, and only if Verification is selected, it compares the copy AFTER its completion.

    The only aspect of this problem on which I am fixating is observable evidence.  Your theory of TeraCopy's operation is not sound, it is not consistent with the software's performance (TeraCopy is as fast as Windows in copying files, which would not be the case in your scenario), and it is based on a fundamental misstep in your interpretation of "copy error."   There is a distinction between a detectable and a non-detectable error, and we are absolutely dealing with a non-detectable error here.  TeraCopy, like Windows, is considering that an error-free result code from a device driver means that the copy was error-free.  It is the optional Verification process which a user can use to safeguard against a bad copy.

    I would appreciate others weighing in, as otherwise, the effort I am putting forth here (including post such as this one) to seek a resolution to this problem is becoming more of a hassle than the problem itself.

    Friday, February 03, 2012 2:07 AM
  • I also try to help on this problem, but I don't know what else can I do. We have explained and given all the data that we can get from our systems. How should we continue to fix this problem? Do I need to start debugging Windows with a Ring 0 disassembler?
    Friday, February 03, 2012 1:01 PM
  • Here is an important new factor it seems. The size of the file is not the only factor, but apparently its structure is relevant to whether the issue occurs or not. I can consistently reproduce the error on roughly one out of every three copies of a 1.4GB movie (.MKV) file, HOWEVER, I cannot produce the error with a 1.4GB text file created using VBS and containing a repeating alphabetic string.
    Friday, February 03, 2012 3:05 PM
  • Here is an important new factor it seems. The size of the file is not the only factor, but apparently its structure is relevant to whether the issue occurs or not. I can consistently reproduce the error on roughly one out of every three copies of a 1.4GB movie (.MKV) file, HOWEVER, I cannot produce the error with a 1.4GB text file created using VBS and containing a repeating alphabetic string.

    Please, try other file types, e.g. zip, doc, docx.
    Friday, February 03, 2012 3:13 PM
    Answerer
  • Can you suggest a location from which I can get a sample 1.4GB .mkv file to test with, and with which you have verified you've seen the problem?  I'll be happy to verify my setup again with a different test data file.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, February 03, 2012 3:21 PM
  • Can you suggest a location from which I can get a sample 1.4GB .mkv file to test with, and with which you have verified you've seen the problem?  I'll be happy to verify my setup again with a different test data file.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options


    Well, this is an interesting issue. I have to test by myself because I have found corrupted files in DVD ISO images (4.37GB). The data inside the ISO images that I tested is compressed (ZIP, RAR, 7ZIP, CAB, etc), so it can also be considered random, like the video data. Anyway, the MKV is only a container for audio and video, i.e. an AVI file could contain the same data (MPEG4 video+MP3 or AC3 audio), though it cannot be larger than 2GB. You can try with any big AVI file. If you don't have one, you can convert any DVD for testing purposes (www.handbrake.fr ). You can also find a big video file in any of the P2P sharing programs. Don't care about copyrigh issues because you are not going to see the video, only use its data.

     


    • Edited by ikerrg Friday, February 03, 2012 3:49 PM
    Friday, February 03, 2012 3:45 PM
  •  Don't care about copyrigh issues because you are not going to see the video, only use its data.


    Please, look at a local laws. In some countries the downloading itself breaks a law.
    Friday, February 03, 2012 3:53 PM
    Answerer
  • Yes, testing with a file could be considered deriving value from it, and I will not illegally download copyrighted material.

    I think the files I've already tested with will suffice as "random" data, as some of them have been large zip files, but for completeness I thought testing with the very same file might be a good idea.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, February 03, 2012 4:04 PM
  •  Don't care about copyrigh issues because you are not going to see the video, only use its data.


    Please, look at a local laws. In some countries the downloading itself breaks a law.

    Good point!
    Friday, February 03, 2012 4:06 PM
  • Can you suggest a location from which I can get a sample 1.4GB .mkv file to test with, and with which you have verified you've seen the problem?  I'll be happy to verify my setup again with a different test data file.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options


    Well, this is an interesting issue. I have to test by myself because I have found corrupted files in DVD ISO images (4.37GB). The data inside the ISO images that I tested is compressed (ZIP, RAR, 7ZIP, CAB, etc), so it can also be considered random, like the video data. Anyway, the MKV is only a container for audio and video, i.e. an AVI file could contain the same data (MPEG4 video+MP3 or AC3 audio), though it cannot be larger than 2GB. You can try with any big AVI file. If you don't have one, you can convert any DVD for testing purposes (www.handbrake.fr ). You can also find a big video file in any of the P2P sharing programs. Don't care about copyrigh issues because you are not going to see the video, only use its data.

     


    Yes, I am not suggesting the issue is limited to .MKV, but I am questioning the varying structure of files. What, for example, would be fundamentally different at a structural level about a file written 28 bytes at a time via a VBS script vs. a video file (regardless of video format)? Is there an internal block and record structure that could impact the manner in which a file is copied?
    • Edited by KT-Retro Friday, February 03, 2012 4:07 PM
    Friday, February 03, 2012 4:06 PM
  •  Don't care about copyrigh issues because you are not going to see the video, only use its data.


    Please, look at a local laws. In some countries the downloading itself breaks a law.

    Good point!

    1) MKV is merely an example of a large file, and not meant to spark a debate about copyright

    2) Regardless of file format, what are the structural variables that could come into play that may impact copying of large, similarly-sized files? That is the point here.

    • Edited by KT-Retro Friday, February 03, 2012 4:12 PM
    Friday, February 03, 2012 4:11 PM
  • Okay, I found a legal .mkv, downloaded it, then appended it enough times into the same file to make it into a 1.4 GB test file.

    I just copied it and 18 GB of other files to my 2 TB MyBook drive, including many compressed files. This was enough file content that it exceeded my RAM capacity, so the test file could not possibly be read back from the cache.

    All files still verified perfectly against the originals when read back via COMP.

    It took quite a while for the copy and compare to complete, showing that the write cache was flushed (I have enabled the "Better Performance" write cache setting, which is supposed to be worst case for this issue).

    I can see how specific data sequences could trigger failures in marginal hardware, causing analog issues (e.g., in the cable) and resulting in unexpected changes to data values. Or it could just be an intermittent software problem with specific disk drivers. Those sequences, because of the way the system works, may not occur in XP or with different drivers.

    I think you really do need to test a WD MyBook, KT, which I have seen to be rock solid. You might also want to try different USB cables while you're at it. Lastly, have you subjected your system to a rigorous RAM test?

    Best of luck getting to the bottom of this. Rest assured it CAN work.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Friday, February 03, 2012 5:07 PM
  • Okay, I found a legal .mkv, downloaded it, then appended it enough times into the same file to make it into a 1.4 GB test file.

    I just copied it and 18 GB of other files to my 2 TB MyBook drive, including many compressed files. This was enough file content that it exceeded my RAM capacity, so the test file could not possibly be read back from the cache.

    All files still verified perfectly against the originals when read back via COMP.

    It took quite a while for the copy and compare to complete, showing that the write cache was flushed (I have enabled the "Better Performance" write cache setting, which is supposed to be worst case for this issue).

    I can see how specific data sequences could trigger failures in marginal hardware, causing analog issues (e.g., in the cable) and resulting in unexpected changes to data values. Or it could just be an intermittent software problem with specific disk drivers. Those sequences, because of the way the system works, may not occur in XP or with different drivers.

    I think you really do need to test a WD MyBook, KT, which I have seen to be rock solid. You might also want to try different USB cables while you're at it. Lastly, have you subjected your system to a rigorous RAM test?

    Best of luck getting to the bottom of this. Rest assured it CAN work.

     

    -Noel


    Detailed how-to in my new eBook: Configure The Windows 7 "To Work" Options

    Thanks Noel. I will borrow another NTFS-formatted drive such as the MyBook, but I have tested multiple drives from two different manufacturers, and in every case, I have used a different (and very short) USB cable. These drives work flawlessly in other settings, and two in particular have worked flawlessly for at least three years. I have also subjected my system to thorough memory tests. No issues there, and besides, if there was flaky memory, there would be other indications, not a behavior peculiar to copying a certain type of file to external NTFS drives.
    Friday, February 03, 2012 6:04 PM
  • I ran a few more tests with my alphabet-sequence text file of 1.4GB, and this time, a comparison failure popped up quickly. The discrepancy between source and copy begins at offset 42B0000 (according to COMP), and the bad data is random data, not simply "misaligned" data from the source. Also, the affected area in the file is quite large...several hundred bytes. So, scratch the earlier observation that only certain types of files are corrupted, and disregard my questions about file structure. The only characteristic of the file that matters is its size. I don't know the specific threshold, but it has to be a relatively large file, on the order of 800MB or more.
    Saturday, February 04, 2012 1:23 AM
  • Two recent observations: - The mismatched data in an affected copy is not misaligned data, but is entirely random, and is potentially (if not likely) pre-existing disk data in pre-allocated file space which is not being populated during the copy. - So far, I cannot reproduce the error on an affected system if I am using Remote Desktop to access the system. In other words, it appears that only when I am physically using the keyboard, mouse and monitor of the system will a faulty copy occur. I discovered this difference in behavior by chance, and not via a deliberate experiment. As another individual in this thread experiencing the problem observed, it seems to occur only when there is other significant system activity during the copy, and now it appears that such activity *may* merely be the use of a USB keyboard and/or mouse. In that the problem never occurs when the write cache is totally bypassed (e.g. using TeraCopy), one possibility is that there is a conflict in handling of USB interrupts which specifically interferes with write cache buffer flushing. The corruption occurs either because the cache memory buffer is clobbered while in use, or the caching pointers are corrupted in a way which results in the buffer not being completely flushed.
    Sunday, February 05, 2012 1:55 PM
  • Two recent observations: - The mismatched data in an affected copy is not misaligned data, but is entirely random, and is potentially (if not likely) pre-existing disk data in pre-allocated file space which is not being populated during the copy. - So far, I cannot reproduce the error on an affected system if I am using Remote Desktop to access the system. In other words, it appears that only when I am physically using the keyboard, mouse and monitor of the system will a faulty copy occur. I discovered this difference in behavior by chance, and not via a deliberate experiment. As another individual in this thread experiencing the problem observed, it seems to occur only when there is other significant system activity during the copy, and now it appears that such activity *may* merely be the use of a USB keyboard and/or mouse. In that the problem never occurs when the write cache is totally bypassed (e.g. using TeraCopy), one possibility is that there is a conflict in handling of USB interrupts which specifically interferes with write cache buffer flushing. The corruption occurs either because the cache memory buffer is clobbered while in use, or the caching pointers are corrupted in a way which results in the buffer not being completely flushed.

    I totally agree with your observations, as I already pointed that the system usage was related to the corruption. Indeed, just before knowing about TeraCopy, I used to copy my big files leaving the system alone (as a workaround), and I usually was succesful. It sounds very good that the usage of other USB ports is the common factor in this problem. Please, check that you external hard drive is connected to the same USB controller as the mouse, the keyboard or any other active USB device. Try connecting the hard drive to other controller (sometimes you have to test the different USB connectors to know which one is connected to each internal controller). My laptop has many USB connectors, but only two of them (not adjacent) are connected to a different hardware controller. When I have time, I'll check it myself. I think you have cornered the problem!!

     

    Sunday, February 05, 2012 5:51 PM
  • Just to add my 2p .. I have recently noticed this issue with my desktop machine; when copying anything over about 25meg the files may become corrupted, and copying anything large (such as a DVD ISO) will always become corrupted. The corruption is always different (fc /b on multiple copies always says they are different). I have tried multiple USB drives, multiple ports (front and back), upgraded to the latest Intel chipset drivers, verified there are no memory issues with MemTest. Using the same USB drives on another machine (also Win7) works fine and the MD5s match my "master" versions. I'm about to boot off a Linux live CD to see if the problem manifests there to try and rule out a hardware issue.
    Friday, February 10, 2012 11:44 AM
  • Well I get sick with this large file copy issue that doesn't go away 
    I tried the bits of ideas shared here and it didn't work

    I have a Dell 1705 and Toshiba Satellite ...  thermatake BlacX Duo USB2 ... WD20 - 2 TB Drive ... been through this problem since Vista ... problems copying large vmk files over to USB ... like 9.6gb, 13.3gb, etc

    Well I check other similar treads ... I figured out my issue ... when I read something that made me think ...

    The file I had a problem with the vmk's ... It appears that inside VMware disk file there is already files compressed in there. I don't think it matter if the file sat compress or regular on the outside.

    My system would go crazy locking up, source file stops reading, system still trying to write something ...

    I read somewhere that some compress files ... would complicate copying

    so the lesson learned ... don't used compressing so much ... use it wisely

    I guess MS doesn't really up keep their compressing routines ... unlike in the old days of DOS ... there were two principle compressing vendors ... now they are gone because of MS doublespace ... 

    I hear that as MS moves to a different drive format ... compression will .. cease to exist

    It appears the this old compressing routine is the reason why VMWare stress that you should not run vmk compressed on the drive or it will lock up ... now I see the source of why .. without running vmware ... its ms waiting to deprecate sw that's prone to issues that takes away manpower to advance their vision
     
    I hope this little insight helps any one with large file copy issues ... remember remove the compression on the drive or directory to where your copying

    good luck

    Tuesday, March 06, 2012 1:32 AM
  • I have a Dell 1705 and Toshiba Satellite ...  thermatake BlacX Duo USB2 ... WD20 - 2 TB Drive

    Interesting.  If I understand you correctly you have a thermaltake external SATA drive adapter with a Western Digital 2 TB drive plugged-into it.

    Based on my observations that Western Digital external MyBook drives don't seem to exhibit the problem, I have been wondering whether something about the drive contoller vs. the USB might be involved.

    In your case there's still a USB driver involved, but this tends to say the drive controller itself is less likely to be the issue, because yours is Western Digital, and points the suspicion more strongly at the USB itself.

     

    -Noel


    Detailed how-to in my eBook: Configure The Windows 7 "To Work" Options

    Tuesday, March 06, 2012 1:51 PM
  • After a bunch of testing and research on this issue, here's some things I've discovered

    If you set the USB drive removal policy from "Quick removal" (which is the default) to "Better Performance", the corruption issue seems to go away.  On quick removal, Windows does write-through caching... Maybe an issue with that?

    Another thing I discovered is if I disable or uninstall Microsoft Security Essentials, the corruption issue seems to go away as well.  I'm am currently installing other anti-virus software and testing to see if I can get the corruption to occur.

    In all these, I say "seems to go away", because it's hard to verify that the problem is gone, and takes a long time to transfer the large files needed for the testing over usb speed.  But so far under the conditions I've listed above, which I have tested repeatedly, I have not had corruption occur on my USB drive.

    Monday, March 19, 2012 3:45 AM
  • Hello,

    I’d like to raise my hand as yet another person struggling with file corruption
    issues.

    I first realized something was awry about two years after I first installed Windows 7
    (Ultimate) 64-bit. I recently started getting file corruption messages from a
    game I play, City of Heroes, which uses game files with a *.pigg extension;
    these files use a variant of the zip-file format. I re-downloaded the game
    files, and then backed up my game folder by creating a zip file of it. Then I
    tested the zip file I created and got CRC errors.

    Large zip files (can’t determine a threshold size) seem to be created okay (w/o error
    messages), but when I perform a test of the zip file or try to extract files, I
    get alerts of CRC errors from WinZip and WinRar, and the generic built-in Windows
    handler mentions an error as well.

    To examine this behavior I created a zip file on another computer (using Windows XP
    Profession 32-bit), tested it using WinZip (it tested okay), and ran an MD5
    utility on the zip file to obtain a checksum. Then I burned the file to a DVD.
    I tested the file on the DVD (it tested okay), and ran the same MD5 utility on
    the zip file (burned to the DVD) to obtain a checksum. The checksum for the zip
    file on my Win XP computer hard disk matched that of the checksum for the file
    on the DVD.

    Then I put the DVD in my Windows 7 64-bit computer, and tested the zip file with WinZip
    and WinRar. I got CRC errors. Further, when running the MD5 utility on the test
    file, I got a different checksum! I tried copying the zip file to my hard drive
    and the MD5 utility generated yet another checksum for that file. Using the
    command prompt copy with verify (copy /v *.zip), I got a message that verify
    failed. When I reboot my computer, the WinZip and WinRar CRC error messages
    flagged different files within the zip as being corrupt and yet another MD5
    checksum would be generated for the zip file. (This MD5 utility is a command
    line, 64-bit compatible program.) Sometimes the MD5 utility generates consist
    checksums (though different than the “correct” checksum) on a given command
    window basis, but sometimes it’ll always generate a different checksum.

    I got the idea to try these tests in "Safe Mode". Well, in Safe Mode everything
    checked out fine. No CRC errors. The MD5 utility generated the same checksum
    for the zip file as in Win XP 32-bit. The zip file could be copied successfully
    (and verified) to my hard drive.

    I ran extensive tests on my processor, memory, hard disks, optical drives, etc. and
    they all came back as okay. I ran extensive virus/malware checks, and used the
    sfc.exe utility. These things all came back okay.

    For another test I booted Windows 7 (using Msconfig.exe) in "Diagnostic Mode". In
    this mode only one more Windows service is added (compared to Safe Mode) and a
    few more drivers are loaded as well. In Diagnostic Mode, the zip file issue
    appears again. This is the closest I’ve come to targeting where the problem is
    happening. I have comparisons of the two modes (from the boot log) but nothing
    jumps out (i.e. all the extra services and drivers loaded correspond to Microsoft
    system files.) I can post the files loaded in safe mode versus diagnostic mode
    if that will help.

    Now I always ensure file copies are verified, if they are of an archival nature, using
    WinDiff or an MD5 utility. I don’t recall having file copy problems with my
    Windows 7 machine previously, but then I don’t remember trying to save anything
    on it in an archival way before. It’s possible this problem has been around all
    along (or just recently, I don't know). But I have the problem now.

    My only conclusion is that something in the operating system is
    "intercepting" access to large files. This seems to make sense
    because that would explain interference with the MD5 utility being able to
    generate a correct (and consistent) checksum. I also have the problem with large
    ISO image files, in terms of MD5 checksums. My next step will be to test a
    variety of file sizes to see if I can find a threshold file size. If the game I
    play (City of Heroes)
    is any indication though, the threshold is probably on the order of 200MB, but
    for now I can’t say for sure. The fact that an MD5 hash cannot be reproduced with consistency is extremely troubling
    to me. I think to myself, “If an MD5 hash can’t be reproduced, how is my
    computer even booting up?” For what it’s worth my “test” zip file was 3.88GB in
    size.

    MSSE is not to blame since I had it disabled during testing (executable and service). I
    don't know if Microsoft is aware of this but something is causing a really big
    problem. A search on the Internet shows that this problem is quite prevalent. Troubleshooting
    this is an ongoing process for me. I would like to resolve the problem without
    having to reinstall the operating system (assuming that would resolve this
    particular issue). If I don’t know what is causing the problem with large files
    then, should the problem appear again, I’d hate reinstallation to be the only
    solution.

    Are there any new developments lately? (I especially ask this of KT-Retro and anyone else
    having issues, but a response from anyone would be helpful) Are there any ideas
    about what’s going on, or better yet any solutions? Any help would be greatly
    appreciated! Thanks for reading this, I know it is long.

    Best
    Regards, Davis 67


    • Edited by Davis 67 Tuesday, April 03, 2012 8:32 PM
    Tuesday, April 03, 2012 8:31 PM
  • Unfortunately, no, I have nothing new on the issue I am experiencing, and the only solution I know is to use a third party utility to bypass Windows write caching. The errors I am seeing, however, are specific to writing large files to a USB-connected NTFS drive. I have not seen any issues with files on local hard drives or DVD.
    Tuesday, April 03, 2012 8:44 PM
  • I've been reading this tread with great interest because I am having a very similar problem.

    I am getting intermitted corruption fetching data FROM an external drive.

    I have about 140GB spread across 14,000+ files.  The files are anything from small text files to large video files.

    The external drive is a Phantom 1TB unit with both USB 2 and eSATA connections.

    I have been writing to/from this drive for several years with a Vista (32-bit) laptop and a Win7 Pro (64-bit) desktop.  I always use Beyond Compare (or FolderMatch) to verify the data after writing it.

    I've just picked up a new Win7 Pro (64-bit) laptop and am trying to transfer the data to it.  I copy/paste the files and then follow up with a Beyond Compare scan to verify.  Sure enough, many times I will end up with 1 byte in 1 file that is wrong.  It is usually a large video file with the problem, but on at least one occassion it was a small mp3.

    Because of the length of time involved in moving and copying this much data I've been using the eSATA connection.  I did one test with the USB 2 connection and got the same results.

    I have another Hitachi 1TB unit with a USB 2 connection. I did a single test with it and ended up - again - with 1 byte in 1 file being wrong.

    Earlier in the week I was able to replicate this problem with almost every test.  I've been testing almost non-stop for the last 24 hours and have yet to hit the problem again.

    I have been using both of these external drives for years, sometimes transferring almost an entire TB of data in one fell swoop and have never encountered this problem.

    This is very frustrating because I'm trying to decide whether or not I should simply return the laptop under the 30-day return policy (taking a sizeable hit on the restocking fee) and shop for something else.

    Friday, April 06, 2012 4:08 PM
  • Hi. Same thing happening to me on Win7x64-Prof SP1. I had made backups of large video files from internal Sata RAID drive onto an external WD passport 1GB USB3 drive as well as on a Toshiba 1GB USB2 drive (2nd level backup), and luckily checked target and source with CRC comparison of BeyondCompare. I was terrified seeing about 10% of the saved files show different CRCs, and corrupted ZIPs on the target. Reformatting the target drives (I even did a low-level format on one drive after no-fault surface analysis) did not help. So I tried copying via Robocopy instead of BC, and verified by MD5 this time. Same result, quite a lot of files reported differences between source and target. Strangely, I also noted that some CRCs already denoted on files that did not differ, now were showing different values. Hex dump showed quite isolated errors mostly affecting a single byte. Of course, I suspected some virus activity, and ran additional checks beyond my routinely activated F-Secure via 4 different scanners residing on a bootable CD drive plus internet signatures update. No problem reported again.

    With 3 days wasted for digging around, and getting nervous about the safety of my family video archive, finally, I stumbled across this thread. Thanks for your professional discussions so far, and for assuring that I am not suffering from sudden loss of IT competence.

    The damage potential of this problem is really high - especially if one had already deleted local files after doing a backup to free internal drive space. I hope MS is coming up with a solid solution fast !

    For the time being, I am trying FastCopy64 as it can automatically run a MD5 verification of source and target files - so at least I hope I will see immediate feedback about copy success or failure. Does anyone have a suggestion for verifying file integrity (like "7zip t" for zips, PvaStrumento for MPGs / VOBs) - especially for exe/dll/..., as I am mistrusting other backup archive contents as well.
    Regards - Robert S.



    • Edited by RobStroe Friday, April 13, 2012 6:33 PM
    Friday, April 13, 2012 6:20 PM
  • I used to also get errors writing large files when I would backup files to other internal HD's and especially to external USB drives.  Well I have a WD MyBook with USB2.0 and eSata.  I obviously always try to use eSata, but I rarely leave it plugged in as I know a big enough % of them don't last that long.  So then instead of rebooting PC I just use USB for transferring files under 2gb or sometimes less than 5gb.

    I ran into TeraCopy a few years ago and absolutely just love it.  I did have a problem with v2.22 but now I'm sure it was just due to the _System Write Cache_ being checked.  Now with the free TeraCopy 2.27 and Win 7 x64 and haven't had that option checked since initial Win7 install 8 months ago and I don't remember getting an error yet.  I have it set to always do a CRC check after I copy large files back and forth, like VMware WS files and such.
    The best part of TeraCopy is that you can copy and paste many, many files and it will wait for the next batch to be copied so you can go away and do other things instead of baby sitting it, or in my case I usually do it overnight and just go to bed. 

    Works great, one of the best programs out there for Win 7.  My 2nd all time favorite that only works in WinXP was FolderSize.  I am still PO'd on that one not working with Win7, I know there are alternatives but they're not the same.

    Thanks to posters above who did actual testing of the "Write Cache" problems, very informative and I think it is a __major__ problem for Windows.  Granted like others have said it maybe less than steller components in the PC, however I doubt that's the case most of the time.  I doubt there are too many problems in a MAC, but then again they only make a few components and you are locked down in they're North Korean like system, so no thanks.

    To RobStroe, I used to use SyncBack SE years ago and it might be able to compare MD5 or SHA-1 backup archives IIRC.  I just found it easier to delete like my whole backup My Docs folder and copy the new one in there again with TeraCopy.  Essentially I was too lazy to do the backups correctly.  I do highly recommend that program, I assume it's even better now but don't know.

    Again, please MS look into this further.  I would love to speed up TeraCopy with the "System Write Cache" option enabled.

    • Edited by SBMan Thursday, April 19, 2012 6:26 AM
    Thursday, April 19, 2012 6:24 AM
  • I ended up having the same problem with the internal drives of this laptop.

    I would shuffle the 140GB of files back-and-forth between the C: and D: drives about 9 times.  I then compared the initial set with the 9th set and sure enough one of the files was off by a single byte.

    I don't know if this was an OS issue or a hardware issue, but I wasn't willing to troubleshoot the problem on a new laptop any further.  The laptop went back and I purchased another one.

    • Proposed as answer by Alexus.P Saturday, April 21, 2012 4:13 AM
    Thursday, April 19, 2012 4:09 PM
  • Guys I am no computer gig just an average user but i bought new computer from i buypower with win7 64 bit have semilar problems i didnt transfer to external hard drives but i do have these problems with large files getting corrupted when downloading or sometimes installing a game had some errors it seems to happen when ever there is a data transfer i hope there will be some fixes soon it really annoying!!!
    Saturday, April 21, 2012 4:22 AM
  • Data corruption during downloading or installing is not the same as what's being discussed here, and such corruption is not something to be expected or tolerated.  

    "Fixes" won't be forthcoming from others - you'll need to diagnose your computer and/or internet service and try to find out what's wrong with it/them.

     

    -Noel


    Detailed how-to in my eBook:  
    In development:

    Configure The Windows 7 "To Work" Options
    Configure The Windows 8 "To Work" Options

    Sunday, April 22, 2012 3:16 AM
  • Hey,

    I understand that this thread is talking about corruption of files copied to an external drive, but then there is an issue of file corruption in general. I posed this on the "Microsoft Answers" forum as I worked through my particular problem step-by-step:

    http://answers.microsoft.com/en-us/windows/forum/windows_7-windows_programs/problems-with-large-files-multi-gigabyte/c7beb33d-99a9-40e8-94ca-2133446fc63d

    It turned out that Creative Labs' SB X-Fi driver dated July 2010 was corrupting the copying of large files and interfering with the ability to generate a correct MD5 checksum (and interfering with the reading of known "good" zip-archives -- resulting is false positive CRC errors). I solved my problem by going back to an earlier version of Creative Labs' driver. It took a while to figure out that this was the source of my problem.

    I mention this because I imagine many people may have Creative Labs drivers on their computer. Hopefully this will help.

    I can't imagine why a sound card driver would cause the behavior I experienced -- maybe (this is a shot in the dark) -- the driver was intercepting file system activity (maybe to "better" parse large media files? -- I don't know).

    Best Regards, Davis

    Saturday, April 28, 2012 5:42 AM
  • Not that it's more than just a data point, but I don't have any Creative Labs software on my system, and I see no data corruption whatsoever. 

     

    -Noel


    Detailed how-to in my eBook:  
    In development:

    Configure The Windows 7 "To Work" Options
    Configure The Windows 8 "To Work" Options

    Sunday, April 29, 2012 2:17 AM
  • Hey,

    I just wanted to clarify that copying large files to an external USB drive and experiencing intermittent corruption was just a subset of the issue I was looking at (with regard to my last post).

    Surely a singular “just a data point” can be seen as random with no statistical significance. In a larger discussion about a particular problem occurring (where a data point is in context), such a data point adds to the discussion.

    The context being, it seems Windows 7 may be open to several avenues in which processing of large files reveals a corruption situation. I say "reveals a corruption situation" because it is not clear (to me) in a larger context what may cause a large file to be corrupted in affected systems. What are the "several avenues"? It might be installed software or maybe even a proclivity for Windows 7 itself to be affected by several different software and/or hardware configurations. I think I found (from my last post) a particular situation in which processing large files comes to reveal corruption issues.

    As KTRetro has said, copy scenarios where corruption issues occur -- -- viewed through the Window 7 OS -- don’t throw up error messages. The only message I’ve seen has been through the command line with an error in verifying file copy.

    Noel Carboni posted this batch file demonstrating his tests with copying large files to USB attached storage:

    :TESTLOOP

    COPY LargeTestFile.tif G:\TEMP

    COMP G:\TEMP\LargeTestFile.tif LargeTestFile.tif <no.txt

    IF NOT ERRORLEVEL 1 GOTO TESTLOOP

    An indication for me that the corruption problem is happening (in terms of a system message I’ve witnessed) would be if

    “COPY LargeTestFile.tif G:\TEMP /v”  (i.e. copy and verify)

    was used and returned with

    ERROR Verify – G:\TEMP\LargeTestFile.tif (since COPY will always work, just not as intended on affected systems).

    I’ve also found that if a verify error occurs, then TeraCopy will also be unable to copy an affected file without errors.

    In my particular testing it seemed (to me) that it was impossible to determine if file copy was actually 1:1 or if verify just failed. In fact on different reboots, different MD5 hashes would be generated for suspect files (either on attached USB drives or on internal disks). This essentially made suspect files inscrutable. I didn’t use the COMP command in my testing (though I did use FC), and that’s something I may go back and try (by testing my system in the “error” state -- which I can reproduce).

    Best Regards,

    <city><place>Davis</place></city>

    • Proposed as answer by Fletcher R Wednesday, May 23, 2012 1:44 PM
    • Unproposed as answer by Fletcher R Wednesday, May 23, 2012 1:45 PM
    Saturday, May 05, 2012 9:45 AM
  • I have almost exactly the same problem while copying files between 2 partitions on normal AHCI hard disk. I take Windows 7 image (iso) for test, and compare files with Total Commander. I'm unable to produce 2 identical copies of this file. What's interesting - copying works without corruption in safe mode.

    Disabling write cache in drives' options does nothing. All partitions are NTFS. I tried copying with all modes of Total Commander and Windows Explorer. Same results. In safe mode copied files are identical, in normal mode - new file is ALWAYS corrupted. Corrupted bytes occur on various positions and in various number. What's even more weird - copying speed is higher in safe mode - 100MB/s, while only 80MB/s in normal mode.

    I tried Intel RST driver, but it made things worse doubling windows boot time without disk data transfer increased.

    And the most interesting thing: I experienced identical problems with different motherboard (MSI), different memory, CPU and Windows 7 installation.

    Since the problem doesn't seem to occur in safe mode - it clearly looks like Windows 7 / AHCI / NTFS bug.

    My configuration (from MS System Information):

    OS: Version 6.1.7601 Service Pack 1 Build 7601

    Disks:

    Description Disk drive
    Manufacturer (Standard disk drives)
    Model SAMSUNG HD154UI ATA Device
    Bytes/Sector 512
    Media Loaded Yes
    Media Type Fixed hard disk
    Partitions 5
    SCSI Bus 2
    SCSI Logical Unit 0
    SCSI Port 4
    SCSI Target ID 0
    Sectors/Track 63
    Size 1,36 TB (1 500 299 297 280 bytes)
    Total Cylinders 182 401
    Total Sectors 2 930 272 065
    Total Tracks 46 512 255
    Tracks/Cylinder 255
    Partition Disk #0, Partition #0
    Partition Size 79,99 GB (85 888 341 504 bytes)
    Partition Starting Offset 32 256 bytes
    Partition Disk #0, Partition #1
    Partition Size 80,01 GB (85 911 515 648 bytes)
    Partition Starting Offset 85 889 907 712 bytes
    Partition Disk #0, Partition #2
    Partition Size 1,21 TB (1 328 500 486 656 bytes)
    Partition Starting Offset 171 801 423 360 bytes

    Description Disk drive
    Manufacturer (Standard disk drives)
    Model SAMSUNG HD642JJ ATA Device
    Bytes/Sector 512
    Media Loaded Yes
    Media Type Fixed hard disk
    Partitions 3
    SCSI Bus 4
    SCSI Logical Unit 0
    SCSI Port 6
    SCSI Target ID 0
    Sectors/Track 63
    Size 596,17 GB (640 132 416 000 bytes)
    Total Cylinders 77 825
    Total Sectors 1 250 258 625
    Total Tracks 19 845 375
    Tracks/Cylinder 255
    Partition Disk #1, Partition #0
    Partition Size 396,17 GB (425 386 773 504 bytes)
    Partition Starting Offset 32 256 bytes
    Partition Disk #1, Partition #1
    Partition Size 100,00 GB (107 372 805 120 bytes)
    Partition Starting Offset 425 386 805 760 bytes
    Partition Disk #1, Partition #2
    Partition Size 100,00 GB (107 373 133 824 bytes)
    Partition Starting Offset 532 760 494 080 bytes

    Description Disk drive
    Manufacturer (Standard disk drives)
    Model Generic USB SD Reader USB Device
    Bytes/Sector 512
    Media Loaded Yes
    Media Type Removable media
    Partitions 1
    SCSI Bus Not Available
    SCSI Logical Unit Not Available
    SCSI Port Not Available
    SCSI Target ID Not Available
    Sectors/Track 63
    Size 3,69 GB (3 964 584 960 bytes)
    Total Cylinders 482
    Total Sectors 7 743 330
    Total Tracks 122 910
    Tracks/Cylinder 255
    Partition Disk #2, Partition #0
    Partition Size 3,69 GB (3 961 809 920 bytes)
    Partition Starting Offset 31 744 bytes

    Description Disk drive
    Manufacturer (Standard disk drives)
    Model pqi IntelligentStick USB Device
    Bytes/Sector 512
    Media Loaded Yes
    Media Type Removable media
    Partitions 1
    SCSI Bus Not Available
    SCSI Logical Unit Not Available
    SCSI Port Not Available
    SCSI Target ID Not Available
    Sectors/Track 63
    Size 1,86 GB (1 998 743 040 bytes)
    Total Cylinders 243
    Total Sectors 3 903 795
    Total Tracks 61 965
    Tracks/Cylinder 255
    Partition Disk #3, Partition #0
    Partition Size 1,87 GB (2 003 522 560 bytes)
    Partition Starting Offset 31 744 bytes

    IDE:

    Name ATA Channel 1
    Manufacturer (Standard IDE ATA/ATAPI controllers)
    Status OK
    PNP Device ID PCIIDE\IDECHANNEL\4&3076FC63&0&1
    Driver c:\windows\system32\drivers\atapi.sys (6.1.7600.16385, 23,56 KB (24 128 bytes), 2009-07-14 01:19)

    Name ATA Channel 2
    Manufacturer (Standard IDE ATA/ATAPI controllers)
    Status OK
    PNP Device ID PCIIDE\IDECHANNEL\4&3076FC63&0&2
    Driver c:\windows\system32\drivers\atapi.sys (6.1.7600.16385, 23,56 KB (24 128 bytes), 2009-07-14 01:19)

    Name ATA Channel 3
    Manufacturer (Standard IDE ATA/ATAPI controllers)
    Status OK
    PNP Device ID PCIIDE\IDECHANNEL\4&3076FC63&0&3
    Driver c:\windows\system32\drivers\atapi.sys (6.1.7600.16385, 23,56 KB (24 128 bytes), 2009-07-14 01:19)

    Name ATA Channel 4
    Manufacturer (Standard IDE ATA/ATAPI controllers)
    Status OK
    PNP Device ID PCIIDE\IDECHANNEL\4&3076FC63&0&4
    Driver c:\windows\system32\drivers\atapi.sys (6.1.7600.16385, 23,56 KB (24 128 bytes), 2009-07-14 01:19)

    Name ATA Channel 5
    Manufacturer (Standard IDE ATA/ATAPI controllers)
    Status OK
    PNP Device ID PCIIDE\IDECHANNEL\4&3076FC63&0&5
    Driver c:\windows\system32\drivers\atapi.sys (6.1.7600.16385, 23,56 KB (24 128 bytes), 2009-07-14 01:19)

    Name ATA Channel 0
    Manufacturer (Standard IDE ATA/ATAPI controllers)
    Status OK
    PNP Device ID PCIIDE\IDECHANNEL\7&24833039&0&0
    Driver c:\windows\system32\drivers\atapi.sys (6.1.7600.16385, 23,56 KB (24 128 bytes), 2009-07-14 01:19)

    Name Standard AHCI 1.0 Serial ATA Controller
    Manufacturer Standard AHCI 1.0 Serial ATA Controller
    Status OK
    PNP Device ID PCI\VEN_8086&DEV_3B22&SUBSYS_83831043&REV_06\3&11583659&0&FA
    I/O Port 0x0000A880-0x0000A887
    I/O Port 0x0000A800-0x0000A803
    I/O Port 0x0000A480-0x0000A487
    I/O Port 0x0000A400-0x0000A403
    I/O Port 0x0000A080-0x0000A09F
    Memory Address 0xEFAF7000-0xEFAF77FF
    IRQ Channel IRQ 21
    Driver c:\windows\system32\drivers\msahci.sys (6.1.7601.17514, 30,38 KB (31 104 bytes), 2012-05-25 01:34)

    Name ATA Channel 1
    Manufacturer (Standard IDE ATA/ATAPI controllers)
    Status OK
    PNP Device ID PCIIDE\IDECHANNEL\7&24833039&0&1
    Driver c:\windows\system32\drivers\atapi.sys (6.1.7600.16385, 23,56 KB (24 128 bytes), 2009-07-14 01:19)

    Name Standard AHCI 1.0 Serial ATA Controller
    Manufacturer Standard AHCI 1.0 Serial ATA Controller
    Status OK
    PNP Device ID PCI\VEN_1B4B&DEV_9123&SUBSYS_84001043&REV_10\6&F15016A&0&002800E0
    I/O Port 0x0000DC00-0x0000DC07
    I/O Port 0x0000D880-0x0000D883
    I/O Port 0x0000D800-0x0000D807
    I/O Port 0x0000D480-0x0000D483
    I/O Port 0x0000D400-0x0000D40F
    Memory Address 0xEFFFF000-0xEFFFF7FF
    IRQ Channel IRQ 17
    Driver c:\windows\system32\drivers\msahci.sys (6.1.7601.17514, 30,38 KB (31 104 bytes), 2012-05-25 01:34)

    Name ATA Channel 0
    Manufacturer (Standard IDE ATA/ATAPI controllers)
    Status OK
    PNP Device ID PCIIDE\IDECHANNEL\4&3076FC63&0&0
    Driver c:\windows\system32\drivers\atapi.sys (6.1.7600.16385, 23,56 KB (24 128 bytes), 2009-07-14 01:19)

    Saturday, May 26, 2012 9:28 PM
  • Since the problem doesn't seem to occur in safe mode - it clearly looks like Windows 7 / AHCI / NTFS bug.

    I'm having trouble following that reasoning...  To me it sounds like a bug caused by something that's running normally, but which is not running in Safe Mode.  Don't forget, there are systems where it doesn't fail.

    Since it's reproducible, I'd review all software running with something like the SysInternals Autoruns tool, and try temporarily disabling things to see if you can isolate it to a specific program or subsystem.

     

    -Noel


    Detailed how-to in my eBook:  
    In development:

    Configure The Windows 7 "To Work" Options
    Configure The Windows 8 "To Work" Options


    Sunday, May 27, 2012 9:22 PM
  • One question: Does anybody presenting corruption problems has the VirtualBox installed? VirtualBox installs a non-signed driver for the USB Virtualization (you can see in the legacy drivers, VirtualBox USB Monitor Driver). I do not know, but it could be a possible reason.

    Cheers,

    Iker

    Friday, June 01, 2012 11:51 AM
  • I had this problem way way back and it's good (in a way) to finally see I am not the only one. What is also good is that I found a workaround seeing as MS don't seem to care about fixing it.

    To stop the problem occuring on my Windows 64bit PC's (happens on several systems) I changed the "Removal Policy" for the USB drives to "Better Performance". I suspect this is a bug in the handling of large files when optimized for "Quick Removal".

    This is still a pain as I haven't found a way to set it to default for all drives yet so new drives have to have that change made atleast once. Worth it though as to date I haven't had any curruption issues with large files.

    Hope this works for others and hope MS fix the bug at some point!

    Friday, June 01, 2012 2:24 PM
  • Hope anyone can explain me how to reproduce this bug so I could see it with my eyes. If this is a Windows bug it should be reproducible.

    Sunday, June 03, 2012 10:59 PM
    Answerer
  • Hope anyone can explain me how to reproduce this bug so I could see it with my eyes. If this is a Windows bug it should be reproducible.

    I can reproduce it easily, and have been able to since I first observed it.   In my case, I only need to copy a large file, say 1GB, from my hard drive to ANY NSFT-formatted USB-connected external drive, and on roughly 20% of such copies, the resulting file will differ from the original when compared via any number of byte-for-byte file comparison tools...despite no errors or events occurring during the copy operation.  Subsequent comparisons of a corrupted file will consistently yield the same results.   That is, the corruption is absolutely occurring during the copy operation.   More particulars are in the thread under my name.
    Sunday, June 03, 2012 11:05 PM
  • But how can I reproduce this? Not you but me. If this is really Windows bug I want to see it. Otherwise I'd suggest the problem may be related not with Windows, but with specific hardware, drivers, software...

    Sunday, June 03, 2012 11:16 PM
    Answerer
  • But how can I reproduce this? Not you but me. If this is really Windows bug I want to see it. Otherwise I'd suggest the problem may be related not with Windows, but with specific hardware, drivers, software...

    I was implicitly suggesting that you try these steps with a similarly sized file (with Windows 7 64-bit and a USB-connected NTSF-formatted drive)...if you haven't already that is.    It may take several copies to see a corruption occur.
    Sunday, June 03, 2012 11:19 PM
  • I've took USB 2.0 hard drive NTFS formatted and copied 10.4GB file to it six times. Each time I compared the copy with the original file. No corruptions.

    Windows 7 x64 SP1 Retail.


    Sunday, June 03, 2012 11:32 PM
    Answerer
  • I've took USB 2.0 hard drive NTFS formatted and copied 10.4GB file to it six times. Each time I compared the copy with the original file. No corruptions.

    Windows 7 x64 SP1 Retail.


    I wish I knew how to duplicate your results and eliminate this problem.   I only know how to work around the issue using TeraCopy with system write cache bypassed.
    Sunday, June 03, 2012 11:39 PM
  • Incidentally, I never see corruption on copies of large (multi-gigabyte) files from the same system to network storage, and I have also never experienced corruption on internal disk-to-disk copies or on copies to FAT-32 external drives...only NTSF (external)
    Sunday, June 03, 2012 11:43 PM
  • Sure seems like either a person's got this problem or not.  Igor and I seem to have a magic combination of software and hardware on which we simply can't reproduce it.  And you can reproduce it no matter what you do.

    Since my earlier posts in this thread I've created a 700 GB System Image backup to my external USB drive, and successfully restored them onto a brand new disk array.  No problems whatsoever.  It's been perfectly stable.

    I do think several of the other folks who have posted in this thread may have different issues...  It only takes one of any number of things gone wrong to corrupt data; everything working perfectly is required for it not to be corrupted.

    I don't know what I'd do if I couldn't rely on the data I copy being accurately copied.  I'd probably go so far as to buy all new hardware, as I simply could not accept "close enough".  I can't imagine how you're living with it, and I really hope you can find a way to solve it.  I saw this thread come up and I initially thought "gee, I hope he's found a fix".

    Best of luck.

     

    -Noel


    Detailed how-to in my eBooks:  

    Configure The Windows 7 "To Work" Options
    Configure The Windows 8 "To Work" Options

    Sunday, June 03, 2012 11:55 PM
  • Sure seems like either a person's got this problem or not.  Igor and I seem to have a magic combination of software and hardware on which we simply can't reproduce it.  And you can reproduce it no matter what you do.

    Since my earlier posts in this thread I've created a 700 GB System Image backup to my external USB drive, and successfully restored them onto a brand new disk array.  No problems whatsoever.  It's been perfectly stable.

    I do think several of the other folks who have posted in this thread may have different issues...  It only takes one of any number of things gone wrong to corrupt data; everything working perfectly is required for it not to be corrupted.

    I don't know what I'd do if I couldn't rely on the data I copy being accurately copied.  I'd probably go so far as to buy all new hardware, as I simply could not accept "close enough".  I can't imagine how you're living with it, and I really hope you can find a way to solve it.  I saw this thread come up and I initially thought "gee, I hope he's found a fix".

    Best of luck.

     

    -Noel


    Detailed how-to in my eBooks:  

    Configure The Windows 7 "To Work" Options
    Configure The Windows 8 "To Work" Options

    If I did not have a reliable workaround/alternative, I would not be living with the issue, but given that at least a few of the others in this thread are experiencing identical symptoms, and given that no commonality has been found, I feel it's a crap shoot buying a new system...and my Dell XPS with i7 processor isn't old enough to warrant an upgrade.   But again, if it were not for TeraCopy which allows me to reliably copy files (and which has shell integration as the default copy handler), I would have NO CHOICE but to try new hardware.  Thanks.
    Monday, June 04, 2012 12:02 AM
  • Two others have indicated that the corruption problem vanishes when the Removal Policy is changed from "Optimized for Quick Removal" to "Better Performance."  Although I see I have mentioned the setting in my posts, it's not clear that I have actually tried this yet.   Such a change will change the behavior from Write-Through Caching to standard Write Caching.   Indeed, it's possible that the bug...wherever it lies...is linked to write-through caching, which *still* makes use of the cache, but in a very limited way (unlike TeraCopy which avoids the cache altogether).

    I am experimenting now.

    Monday, June 04, 2012 12:27 AM
  • After a bunch of testing and research on this issue, here's some things I've discovered

    If you set the USB drive removal policy from "Quick removal" (which is the default) to "Better Performance", the corruption issue seems to go away.  On quick removal, Windows does write-through caching... Maybe an issue with that?

    Another thing I discovered is if I disable or uninstall Microsoft Security Essentials, the corruption issue seems to go away as well.  I'm am currently installing other anti-virus software and testing to see if I can get the corruption to occur.

    In all these, I say "seems to go away", because it's hard to verify that the problem is gone, and takes a long time to transfer the large files needed for the testing over usb speed.  But so far under the conditions I've listed above, which I have tested repeatedly, I have not had corruption occur on my USB drive.

    I haven't tested exhaustively, but based on results I've obtained so far with the Removal Policy set to "Better Performance" for the external drive, I believe that you (Disordered) as well as CyDoNiAGB have found an answer within Windows itself that "fixes" the problem (!)   Disordered, I thought I had already attempted to disable Microsoft Security Essentials, but I will try that as well.

    Thank you both, and I am only sorry that I did not try this suggestion before now.

    I am reluctant to select "Mark As Answer," however, even if my testing continues to back this theory, because there is clearly still a bug that Microsoft needs to address.

    More soon on followup testing.

     


    Monday, June 04, 2012 12:46 AM
  • I have MSE and it was turned on during testing. I'd suggest to look into a list of USB drivers ot USB-related software you may have installed.

    The steps I'd trying to do are: make a clean install retail or TechNet Windows 7 copy. Really clean with all drivers out of the box or from Windows Update only  and noone driver or software from any other source including hardware manufacturers. Than try to reproduce.

    Monday, June 04, 2012 5:59 AM
    Answerer
  • Errors with MD5 hashes on large (multi-gigabyte) files <-- I mention this because it is a super-set of issues I experience when copying files to external USB drives.

    I haven't worked through all the scenarios in which problematic MD5 hashes can be "computed" but I see the issue as being eventually resolved (at least in my mind).

    Essentially, it seems to boil down to Windows (x64) running 32-bit programs in emulation. The upshot, if a person runs a 32-bit MD5 utility (or any 32-bit program) it should really be installed in the "Program Files (x86)" folder. While some 32-bit programs seem to operate okay outside this folder others do not -- and typical (32-bit) MD5 hash utilities seem to be in this class. I'm in the process of testing this and the results are favorable.

    Here are some links:

    http://www.samlogic.net/articles/32-64-bit-windows-folder-x86-syswow64.htm

    and

    http://www.youtube.com/watch?v=esrbDZK219c

    I'll post more details when I work them out.

    Some more things that I've noticed and am looking into is that file compression programs (WinZip, WinRar, etc.) seem to be problematic too, if not used with a bit of caution.

    For example, I've noticed if I run a 32-bit MD5 hash program from the folder "C:\MD5" on a large file that's several gigabytes, then I'll get an incorrect MD5 checksum for the large file. Next, if I run a 64-bit MD5 hash program on the same large file, I'll also get an incorrect checksum. What seems to happen here is that the large file is loaded into memory when first running the 32-bit program and whatever file handling goes on in the 32-bit emulation results in the 64-bit program reporting the same, but still incorrect, checksum as the 32-bit program. (I'm assuming the file is cached or put into memory as a result of the speedier computation of any subsequent MD5 hash on the same large file). Launching a 32-bit program from the "Program Files (x86)" folder seems to ensure the proper file system support is given. If a file is cached without proper file system support then it might cause more than one program to operate incorrectly with the desire data.

    So there could be a myriad of combinations in which 32-bit and 64-bit programs could be installed on a computer and a some data files are "mishandled".

    Best Regards, Davis

    Friday, October 05, 2012 3:12 AM
  • Less file system "magic" is one of the many reasons I prefer to completely disable UAC.  But I'm not convinced this could have anything to do with files being copied to/from external hard drives with errors.

    For what it's worth, I rely on my external USB drive for a number of critical things, with data copied and restored using both 32 and 64 bit programs on Windows 7 x64, on 3 different computer systems, and I still have never had one byte corrupted.  Can you show a specific, repeatable example?  What MD5 hash program is this you're using?

    Are you using ECC RAM?

     

    -Noel


    Detailed how-to in my eBooks:  

    Configure The Windows 7 "To Work" Options
    Configure The Windows 8 "To Work" Options

    Friday, October 05, 2012 9:32 PM
  • Hey, just a quickie update

    ** I have UAC disabled

    ** I'm still in testing phases and I'm not sure "specific repeatable example" would be meaningful to anyone not having the problem KT-Retro mentioned, unfortunately.

    ** It's unclear that data is necessarily corrupt with regard to processing large files. But, I am certain (at least on my software configuration under Win 7 x64) that utilities which compute checksums/hashes fall under a category (perhaps larger category) of programs that have issues with large files. (Zip utilities, for example, like to compute CRC32 checksums on files).

    ** Take md5deep for example. On a number of 32-bit machines, the hash for a large file will always be the same, but on my Win 7 machine the hash will change under different command windows (md5deep is command line) and/or after reboots. Under safe mode or the "Repair disc Windows 7 64-bit", however, the hash for a given file is always dead on with what was computed on a 32-bit machine. fc between two files that are "the same" may return different results after successive runs under the same command window. So, tricky.

    ** Still in testing phases. It's very hard to narrow down.

    ** Smaller files don't have an issue under the conditions I'm experiencing but it's hard to say where the boundary is (in file size)

    ** I would be interested (if anyone has the time) given some large file (a gigabyte or more), if the hash for that file matches up with what's computed on Windows 7 x64 versus a different 32-bit operating system. If it does... then what's the software configuration like on your Windows 7 x64 system? Yes this is all very vague/general... but I'm trying to narrow testing parameters.

    ** RAM tested fine. Not ECC RAM, though.

    This note is longer than I wanted it to be, still trying to find a culprit. It'll take time.

    Btw, nice book Noel.

    Monday, October 08, 2012 2:16 PM
  • Just found this discussion while trying to determine what the system write cache setting in TeraCopy was about. Did some testing and fc myself, and found no issues on my system, luckily.

    I'm on Windows 7 Ultimate x64 SP1, all updates applied.

    My USB controller is the "Intel N10 / ICH7 Family" controller, on a Dell Optiplex 380 motherboard (

    PCI\VEN_8086&DEV_27CC&SUBSYS_04001028&REV_01) and I'm using a  Western Digital 80 GB VelociRaptor drive in a KingWin EZ-Dock connected through USB 2.0 port, using a real USB 2.0 marked cable.

    Tried a few large files, including:

    ☺ 3 gig .RAR archive, using the regular Win7 copy util. FC reported no differences.

    ☺ 8 gig .ISO, using TeraCopy with System Cache enabled. FC says no diffs.

    ☺ 5 gig .MKV, using Windows Copy. FC: no diffs.

    Maybe I just got lucky, but I'm not having any issues on here.

    Friday, October 26, 2012 6:41 PM
  • Hi,

    After some months I have re-searched for the issue on the internet. It seems that many people are asking about this problem in many different places. Some of them:

    http://forums.cnet.com/7723-7588_102-231717/external-usb-laptop-hard-drive-storage-file-corruption/

    http://www.tomshardware.co.uk/forum/271012-14-files-corrupt-copying-external-hard-drive

    http://www.sevenforums.com/drivers/52122-usb-data-transfer-file-corrupted.html

    http://windows.bigresource.com/Win7-Usb-data-transfer-file-corrupted-NC99Sz5w.html

    http://forums.techguy.org/windows-vista/734623-large-files-corrupted-when-copied.html

    http://www.sevenforums.com/general-discussion/210800-windows-7-64-bit-corrupting-large-files-copied-external-ntfs-drives.html

    About myself, I’m still having the issue in all my systems, even in a new laptop (ASUS EEEPC 1215B with AMD chipset). The problem happens also when copying files to USB sticks formatted in FAT32, and it is veeeery annoying, because all the file transfers must be double checked (after reinserting the USB stick) in order to be sure that the data is correctly copied.

    Any progress in the past months?

    Best regards,

    Iker.

    PD. This thread is too big that it lasts a long time to load in my computer. IE seems to be frozen for some time when loading. Could it be split in several parts?

    Monday, December 17, 2012 11:10 AM
  • It's pretty clear there's SOME specific factor that makes this work perfectly for some folks and not for others.

    Some updates from me...

    Since I posted before, I have upgraded from a Dell T5400 workstation to a T5500 model.  I'm still seeing no problems whatsoever with file copies to/from the MyBook 2 TB backup USB drive I use with it.

    I just bought myself another Western Digital MyBook for Christmas.  This one is a 3 TB model.  Up to now I have had perfect reliability with 1 TB and 2 TB models.  I considered also buying a PCIe USB3.0 controller (my workstation just has native USB2.0 ports), but I thought twice when I remembered this thread and didn't add it to the cart.  Better not to mess with success, methinks.  I'll report back again after the 25th after I test it thoroughly for this issue.

     

    Btw, nice book Noel.

    Thank you for the nice feedback, Davis 67. :-)

     

    -Noel


    Detailed how-to in my eBooks:  

    Configure The Windows 7 "To Work" Options
    Configure The Windows 8 "To Work" Options

    Monday, December 17, 2012 9:39 PM
  • Hi,

    I ran across this thread because I think that I'm encountering this problem when I was copying some large (>8GB) VMDK files to a new 1.5TB Canvio USB3.0 drive.

    I've read through most, but not all, of this thread, and am I correct that there's no solution/resolution to this problem yet (a year later!!)?

    If so, is Microsoft aware of this (I saw the earlier msgs with the MS employee, but he disappeared?), and are they doing anything about it?

    Thanks,

    Jim

    Thursday, January 03, 2013 7:58 AM
  • Hi,

    There was some mention of Teracopy earlier.  I tried to download that from codesector.com(?), and got a setup.exe, but when I run it, it tries to install something called PDF Creator, and also AVG is detecting that it's trying to connect out and marks it as malware.  Where did you all get Teracopy from?

    Jim

    Thursday, January 03, 2013 8:29 AM
  • 'Where did you all get Teracopy from?'

    Jim: download from CNET.com

    Paul

    Saturday, January 05, 2013 2:03 AM
  • I just wanted to follow-up...

    My new 3 TB Western Digital My Book 1140 USB drive has now been hosting my backups for several weeks without errors.  I've just verified hundreds of gigabytes of backup files copied to the drive with XCOPY then read back with COMP - with zero errors.  Some files were very large (many gigabytes) and some small.

    My Dell T5500 system has USB2 via Intel ICH10.  I considered getting a USB 3 PCIe card, but since this system has no problem completing its 1 am System Image backups well before I need to use it in the morning, I really don't need it to be any faster.  And I've got to say, given the troubles exposed by others in this thread, I really don't want to mess with success, since it's utterly reliable.

    I wish for all of you experiencing problems for 2013 to be the year in which you discover and fix what's at the root of your external drive problems.  Keep the faith, it really is possible to get these things to work reliably.

      

    -Noel


    Detailed how-to in my eBooks:  

    Configure The Windows 7 "To Work" Options
    Configure The Windows 8 "To Work" Options

    Sunday, January 06, 2013 10:58 PM