none
Widows Search not working with Data Deduplication?

    General discussion

  • Hi,

    I noticed that many files were missing when searching for them on my Server 2012 Fileserver.

    After some troubleshooting I noticed that the ones missing, had a Size on Disk of 4KB and the "SparseFile" and "ReparsePoint" (PL) Flags set.

    So it looks like they were processed by the enabled Data Deduplication.

    Am I missing something here or is it really the case that deduplicated files cannot be indexed by windows search?

    Friday, March 1, 2013 1:46 AM

All replies

  • Hi,

    It seems that Junction Points will not be indexed, so I'm thinking if Data Deduplication is supported by Windows Search... Will do more research on this...


    TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tnmff@microsoft.com.

    Monday, March 4, 2013 9:26 AM
    Moderator
  • Hi,

    It seems that Junction Points will not be indexed, so I'm thinking if Data Deduplication is supported by Windows Search... Will do more research on this...


    TechNet Subscriber Support in forum |If you have any feedback on our support, please contact tnmff@microsoft.com.


    I'd appreciate that. I disabled deduplication for one of the Volumes, and the files there get indexed fine now. (Some still have the "P" flag but the "L" flag is gone)
    Thursday, March 7, 2013 2:43 PM
  • Hi Shaon.

    I am also interested in this.

    In last couple of years - I have "obsoleted" use of drive letter. That's why we got mount point's in the first place to mount and virtualize those file hierarchies. I.e. most of us are not using drive letters and "network-based" harddrives - but networked file hierarchies using shares.

    So I just ran into the same issue here on a test system. Made a share "...\Shares" and mounted those file hierarcies below. 5 resparse points.

    Windows 7, 8 libraries advertised the relocated shares could not be added to file liberaries unless indexed.

    So let's go add those folders.

    Reminder and bad news: Indexing complete in 1 sec. Do not follow reparse points.

    Very bad if this is true. It can not be we still have to use volumes mounted to drive letters only! Especially, now Windows Storage Spaces are really encouraging the use of volumes. I just recall why I had trouble using volumes in the first place ... Windows Search not working with resparse points. Because I can not just keep assigning drive letters to volumes. Simply running out ... THIS SOULD BE A HIGH PRIORITY ISSUE BECAUSE WINDOWS STORAGE SPACE AND VOLUMES HAVE COME ARROUND NOW. Before you would may be have a few very large volumes to deal with this issue of bad support in Windows Desktop Search for file and storage virtualization in Windows.

    We can not use liberaries in Windows 8 without Windows Search ON THE SERVER having indexed those remote files. File hierarchies in the Windows client are functionally dependent on Windows Desktop Search to work.



    Friday, March 8, 2013 8:45 AM
  • Just tested and can confirm resparse points does not work in Search indexing whether hard links, junctions or symbolic links ...

    mklink /d test \\?\Volume{1cf32c52-15c0-497f-91a0-7321ebadbd0a}\

    mklink /j test \\?\Volume{1cf32c52-15c0-497f-91a0-7321ebadbd0a}\

    ... can not browse in WDS: Control Panel>Index>Modify>browse folders ...


    Friday, March 8, 2013 9:00 AM
  • BTW! "... and mounted those file hierarcies below. 5 reparse points" so I my case I had consolidated the users Space into 5 overall concepts on 1 volume each. In that way I had already reduced my use of volumes.

    So for now I can add a drive letter to each for trying to make Windows Search work.

    But guess I am going to stumble into the same issue then with Windows Search not supporting resparse points again ....because some of the volumes are deduped.

    Great experience as ever working with file virtualization in the Windows client, i.e. Liberaries. Just wonder why the heck MS did not improove that experience ...

    Right now, with Server 2012, find myself "hacking" again this year 2013 ... to try and make Windows work on some basic issues. All the Flex is starting to go away now ... here comes reality ... NOT WORKING.

    NTFS resparse points, long file names, etc. Great Flex! However in theory only. For real when everything is pulled together and the system receives its more or less complete configuration ... unsupported in Windows. Go hack it.

    Please stop doing just components test ... and do complete flight tests also. More hollistic tests please across constraints! With a focus in each test ... from 0 to a fully configured system.

    How can no support in Windows Search for resparse points in NTFS be a miss during introduction of libraries (Windows 7/2008 R2) and now during introduction of data deduplication (Windows 8/2012) ... during just internal tests at MS? And we all have to use libraries (thus a remote Windows Search index on remote shares in a domain setup) ... and now Windows 8 WinRT is even more dependent on libraries? How can that be overlooked - or such a basic thing receive no priority - and just get skipped? How can that happen?



    Friday, March 8, 2013 9:16 AM
  • Now go and tell the WDS team to fix this compatibility with resparse points getting more and more in heavy use. If no mount points or similar flex in ReFS - then ReFS is a no go for me ... it's reliability AND FLEXIBILITY.

    Windows Search works ... but this built in dependency on Windows Search i.e. Windows client, WinRT and libraries ... and then no support reparse points. It's a no go! IT HAS TO GET FIXED.

    Or switch the search engine. An intermediary here has to be FULLY compatible with NTFS! This is not 3rd party development.





    Friday, March 8, 2013 9:29 AM
  • Or tell us how to override and configure Windows Search for reparse points :o)
    Friday, March 8, 2013 9:31 AM
  • And BTW:

    DFS still does not work with Windows Search:

    So problems with using Windows Search has now extended to:

    1) No DFS support

    2) No Reparse points support (i.e. no mount points)

    3) Thus no deduplication support (implied by 2) (NEW)

    ... and Windows Search is required for using File Libraries on Windows 8

    4) WinRT applications has to use file liberaries (NEW) - which has to use Windows Search ... which cannot use deduplication, reparse points or DFS.

    What a mess. Please fix Windows Search.


    Friday, March 8, 2013 12:43 PM
  • We reported the indexing / deduplication problem to Microsoft at the end of February.
    They got back to us confirming that it is a known problem related to design.
    The windows search service is not able to index deduplicated data. The following steps can be used as a workaround to get all data indexed:
    1. Deactivate deduplication (Server Manager)
    2. PS:> Start-DedupJob -Volume X: -Type unoptimization
    3. Rebuild Index
    4. Activate deduplication (Server Manager)
    5. PS:> Start-DedupJob -Volume X: -Type optimization
    Pay attention: The above mentioned steps decrease the server performance. Unoptimization additionally requires more space on the volume. Check with the PowerShell command "Get-DedupStatus" if you dispose over enough available space on the volume.
    PS: Microsoft told us that if a lot of customers report this issue, they will fix it. So feel free to report to Microsoft accordingly.

    Wednesday, March 27, 2013 7:55 AM
  • In Febraruary you wrote:

    We reported the indexing / deduplication problem to Microsoft at the end of February.
    They got back to us confirming that it is a known problem related to design.
    The windows search service is not able to index deduplicated data.

    Any news?

    Can we hope Microsoft will solve this problem?

    Deduplication should be useful for data storages but if it brokes indexing and searching..... nobody can use it :(

    Friday, August 9, 2013 3:00 PM
  • I too am interested in MS fixing this issue.  My index is not finding much of what is in our file system.  I'm a little unclear if the workaround removes dedupe, indexes, and then RE-dedupes the data, or if it simply unoptimizes the deduped volume and reindexes.   The point being, indexing has to occur with dedupe off, but will remain intact (indexed) if you turn on dedupe after the fact.

    Thanks for any help.


    DLovitt

    Tuesday, December 31, 2013 12:44 AM
  • If I do the initial indexing that way - what is happening with new files, or old moved
    files ... are they indexed correctly than?

    Regards

    Martin

    Friday, January 17, 2014 2:11 PM
  • Same issue here.

    For testing we moved some files that were not searchable and the Index was unable to find to a new location. Once we moved the files they Indexed immediately and we were able to search them successfully.

    It looks like you have to index your files first before you dedup.

    Currently have a case open with Microsoft, I will report back if they find anything.

    Wednesday, April 23, 2014 5:34 PM
  • Thanks to those who opened a case.

    Since we have OEM licenses for Windows Server (not Volume licensing), we would have to pay >200 $ just to report the issue. THEIR issue, as it clearly seems. Another lesson of "Just get big enough, then you don't have to bother anymore".

    Sorry, but I'm quite upset that those (well advertised) features do not work together, even in Server 2012R2.

    Regards,

    Andreas

    Wednesday, April 30, 2014 12:16 PM
  • Is there some news about this issue?

    Regards, Lorenzo.

    Monday, November 24, 2014 10:05 AM
  • This is ridiculous. We just came across this too. Our 2012 R2 file server search is now useless because all volumes are deduped.

    This is critical functionality that has to be fixed.  Nowhere is any dedupe documentation or reviews or write-ups or blogs have I ever seen it mentioned that dedup breaks indexing.

    Tuesday, December 9, 2014 2:36 AM
  • I'm having the same issue.
    I don't want to disable deduplication because that would need me to extend my disk volume which I won't be able to shrink afterwards.
    Think the idea of deduplication is to use less diskspace isn't it... :-)

    Microsoft, please fix this or provide us with a tool to create the initial correct windows search indexing database.

    Regards,

    Kristof

    Tuesday, December 30, 2014 1:26 PM
  • confirmed still not working on 2012R2 with latest windows updates as of June 2015 :(

    this really needs to be fixed!!

    Thursday, July 9, 2015 10:10 PM
  • Does anyone have any news on this one?
    Thanks,

    M

    Wednesday, September 9, 2015 1:20 PM
  • Hi everyone,

    i can confirm this with Microsoft Server 2012 R2.

    We got about 2,5 Mio. in our Indexservice bevor deduplication.
    After deduplication and rebuilding the index, i only got about 1 Mio. Files indexed.
    This is about 40% less Results than bevore and it is also 40% of saved dataspace.

    Now, we got a SharePoint2013 SP1, indexing our fileserver with all these 2,5 Mio. files und surprise, we still got only 1 Mio. Files indexed. All Hope ist gone for Microsofts deduplication.

    Next step ist changing the Primary Datastore to an EMC VNX5200 and we gonna deactivate dedup von MS-Filestore and active dedup on PrimaryStorage. This Solution ist expensive for shure.

    Sry for my bad english, my browser also does a german autocorrection :-(


    Monday, October 26, 2015 4:36 PM
  • We also have this issue and it needs a fix. They could have posted something on all these duplication articles on what it actually breaks so we knew what we were getting into beforehand.
    Monday, November 7, 2016 10:57 AM
  • Hi, we could need all of your guys votes for this fix to make MS listen to this customer issue, please go vote on the fix request I created here: https://windowsserver.uservoice.com/forums/295056-storage/suggestions/17888647-make-windows-search-service-work-with-data-dedupli

    Thanks and kind regards

    Sven

    Thursday, March 2, 2017 9:18 AM
  • I have recently re-encountered this issue after doing my due diligence before blatantly enabling Data Duplication. 

    Have a look at the following KB article, specifically the very last paragraph.

    TL;DR: Windows Search is still not compatible with Data Duplication.

    https://technet.microsoft.com/en-us/library/hh831600(v=ws.11).aspx

    My account hasn't been verified yet, So I can't insert links. Therefore you will have to just copy and paste the above URL into your browser. Doh!



    • Edited by Tricky83 Friday, September 29, 2017 5:05 AM tidy up of comment.
    Friday, September 29, 2017 5:01 AM