none
Problems with the disk space of Fast Search RRS feed

  • Question

  • Hi I have big problems with my diskspace.

    I'm crawling a longer time now and anything was allright, but after the last full crawl my diskspace is not enough...

    During the crawl there where no problems, the disk space was enoght and anything gets crawled. But now the space seems not to bee enought. The space starts by a half and then it gets filled up to the limit and then it gets cleaned out to the half again and this loops ervery 2 hours. If i stop the services from the servers my disk space is round about 300gb of free space and 300 used space. When i start the services the problems began like i explained. And it loops on every one of my three servers.

    So to see the staus of the indexer i run the folling command in the powershell "indexerinfo --row=0 --column=0 status" an there the status of the partition 0,1 and 3 are idle and the partition 2 an4 are "indexing" but they only go up to maximum 8% and after this the disk ist full! Do i need so much more diskspace for the harddisks to get the status to 100% or is there somthing else i can do? Becuse normally the space should be enough.

    Kind Regards

    Markus Becker

    Friday, July 27, 2012 7:52 AM

All replies

  • After a reboot, the partition 2 and 4 stop at about 44% but the storge didn't stop getting full...
    Friday, July 27, 2012 12:41 PM
  • Hi,

    How many items and of what kind are you indexing?

    If you are running out of space, the general rule would be to add more space.

    Thanks,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/MCT/MCPD
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Friday, July 27, 2012 8:33 PM
  • Hi,

    I have now round about 20 million Item. They are all form our file servers, where mostly are office documents. In avarage on file is 0,2 MB.

    The problem is, that the three Fast Search Server are physikal and i can't put more than 900GB of Data on them. In the moment i have split the data on two partitions. One with 650GB for the insatllation and the Data_index with 64k cluster size and one for the data_fixml with 250GB an 1k cluster size.

    I think the size should normaly fit, i think that there is a problem someware that the server fills up the sapce. Because the indexing stucks somewhere.

    Thanks

    Markus Becker

    Monday, July 30, 2012 7:20 AM
  • Hi,

    When splitting it, is it the index or fixml folder which fills up first? Also, have you checked the size of the var/log folder? Would be good to identify which folder is consuming your space. It might not be the index itself, but the analytics component.

    Thanks,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/MCT/MCPD
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Monday, July 30, 2012 7:02 PM
  • Hi,

    The volume with the index is the one that fills up the disk space. The most space needs the data_index Folder and this is also the one that fills up the space. 
     "2" (360GB) and "4"(100GB) under Data_index are the folders that take the most space and are also folders that changes the size on the admin server.
    On the no-admin server the folder "0" is the one with the most space (600GB).

    The thing that i worry mostly is that the status of the indexer starts relativ fast and than at one point it stop or only increase one or two percent until the space ist full.

    I have no idea what i can do. And a complet new full crawl woud take me about 1 or 2 month...

    Tuesday, July 31, 2012 9:32 AM
  • Hi,

    The following numbers are from the medium scenario at: http://technet.microsoft.com/en-us/library/ff599526.aspx

    44 million items (11TB original data) uses 1.63GB for the web analyzer and 2.2TB for the index.

    As you have 20 million items, you should expect to use ~1TB for your index if the data is comparable.

    For an item which averages around 200-250kb you should expect to use around 50kb for the index.

    Thanks,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/MCT/MCPD
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Wednesday, August 1, 2012 8:02 PM
  • Hi,

    but i have 3 Servers with 900GB, so overall 2,7 TB of space. In the moment i have 20 million items with 10TB of original size. So why does the index only grow to a maximum of 45%? So if I believe in the things my server says i need 6tb of space for the index! This could not be possible.

    In the 2.2 TB of index data is ther only the data_index folder or also the data_fixml?

    Thanks

    Markus Becker

    Thursday, August 2, 2012 6:48 AM
  • Hi Marcus,

    How is your three server farm set up? Feel free to share your deployment.xml.

    If the setup is three columns I agree you should have enough space.

    Thanks,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/MCT/MCPD
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Friday, August 3, 2012 6:45 PM
  • Hi Mikael,

    hier is my deployment.xml:

      <?xml version="1.0" encoding="utf-8" ?>
    - <deployment comment="FAST Search farm configuration" xmlns="http://www.microsoft.com/enterprisesearch">
      <instanceid>FAST Search Server</instanceid>
      <connector-databaseconnectionstring />
    - <host name="FastServer1.corp.int">
      <admin />
      <query />
      <content-distributor />
      <searchengine row="0" column="0" />
      <indexing-dispatcher />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="1" />
      <document-processor processes="12" />
      </host>
    - <host name="FastServer2.corp.int">
      <content-distributor />
      <searchengine row="0" column="1" />
      <document-processor processes="12" />
      </host>
    - <host name="FastServer3.corp.int">
      <content-distributor />
      <searchengine row="0" column="2" />
      <document-processor processes="12" />
      </host>
      <host name="DB01.corp.int" />
    - <searchcluster>
      <row id="0" index="primary" search="true" />
      </searchcluster>

     </deployment>


    The Problem is, I think after the disk space runs out of space once I didn't get ist to space again.
    My Problem was, that the cluster size of the disk was 64k, but because of the many small files for the data_fixml the space was full far to soon. So i splitted the disk in 2 partions with the fixml on 1k size and the rest with 64k and now i have  the problems like I explaind. And this on all three servers.


    • Edited by hb_markus Monday, August 6, 2012 8:55 AM
    Monday, August 6, 2012 8:50 AM
  • Hi,

    As you have three columns the content should be divided evenly on all servers (assuming everything work), and I guess you see volume fill-up on all three servers as well.

    I'm at a loss and recommend you file a support case with Microsoft on the issue. As for re-indexing, you can clear the FAST index and refeed from the fixml to get it up and running more quickly than crawling all the content. You can find details on how to reindex using the fixml files at http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=28548.

    Thanks,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/MCT/MCPD
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Monday, August 6, 2012 5:50 PM
  • Hi...I have also been struggling with FAST problems which seem to be related to disk.

    One of the key things we learned during this on going issue (in the midst of resolution) is that FAST requires significant disk IOPS and transfer speed to and from disk.

    In fact Microsoft support provided the following article http://technet.microsoft.com/en-US/library/gg604775  which at least provides some basic guidelines on how to validate your FAST server disk subsystem.

    You had mentioned you have 900G of disk space per server but do not say are you using a single disk or is a RAID configuration.

    As for allocation unit size be aware that the larger allocation size you use when you format can lead to an increase in fragmentation which will slow you down. We have used the default allocation unit size when formating the drives and also have disabled 8.3 name conversion.

    In a nut shell what I am learning is Disk IO is the number 1 factor in peformance of fast followed by memory and then the number of CPU's, it is has been an interesting journey thus far.

    Marco 

    Thursday, August 16, 2012 1:39 PM