locked
Index server cpu usage 100% when crawling huge data using fast search. RRS feed

  • Question

  • Hi,

              We are using fast search to crawl file shares which contains huge data (500 gb - 1 TB). Our index server is going 100% cpu usage and taking days to crawl. 

              Any recomendations to adjust the topology or to increase performance of crawl faster. We are using 8gb ram on the index server.

    Wednesday, March 28, 2012 7:14 PM

Answers


  • Hello,

    Did you get a chance to review Performance and capacity management (FAST Search Server 2010 for SharePoint)

    http://technet.microsoft.com/en-us/library/gg604778.aspx

    In particular the testing and tuning sections.  Did you run a SQLio test?  if so did you meet or exceed the minimum results?

    The Fast Search indexer in general requires significant Disk i/o.  I have seen cases where both memory and CPU spike when when the system is waiting for disk i/o.  Other possibile issues are software that could be monitoring the Fast Search files and folders (examples antivirus; backup; monitoring).

    Perfmon Counters Low % Idle Time and High CurrentDisk Queue Length can help you diagnos Disk i/o issue.

    Hope this helps,

    Best Wishes,

    Michael Puangco | Senior Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                         Microsoft| Services

    Sunday, April 1, 2012 5:40 PM
  • Hi Tech-fun,

    Could you explain your FS4SP farm topology. How many servers, what performance characteristics do they have, and which servers run which FS4SP component?

    8gb ram sounds a bit on the short side, as the recommended is 16gb.

    Indexing is both CPU bound (and disk bound), and as you are maxing out the CPU you either have to cut down on the number of document processors in your FS4SP farm or add more servers to distribute the load. The ideal CPU load is to have the system be around 90%, to not saturate it. Saturation always leads to degradation.

    By cutting down the number of doc procs you may increase the crawl time, so it's better to add more servers or beef up the hardware on your current server (as additional servers have a premium cost).

    Regards,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/WCF4/ASP.Net4
    http://techmikael.blogspot.com/

    Monday, April 2, 2012 2:12 PM

All replies

  • We also are crawling a huge data set ~25TB...We started with the recommended and then bumped things up.

    Look at the following link, you may be low on memory and and cores as you do not mention number of cores.

    http://technet.microsoft.com/en-us/library/ff381239.aspx

    Ultimately our single fast server is 3G Hz @ 16 cores and 24G of memory with a local disk of 12 disk in a raid 10 configuration for the local fast database.

    This configuration during crawling averages out at 75% cpu utlization with spike over 85 and lulls of 20%.

    Our crawls of 25TB have been running for over 900 hours so this is not the speediest process...but it is impacted by degree of customization, we run a almost vanilla setup.

    Wednesday, March 28, 2012 10:22 PM

  • Hello,

    Did you get a chance to review Performance and capacity management (FAST Search Server 2010 for SharePoint)

    http://technet.microsoft.com/en-us/library/gg604778.aspx

    In particular the testing and tuning sections.  Did you run a SQLio test?  if so did you meet or exceed the minimum results?

    The Fast Search indexer in general requires significant Disk i/o.  I have seen cases where both memory and CPU spike when when the system is waiting for disk i/o.  Other possibile issues are software that could be monitoring the Fast Search files and folders (examples antivirus; backup; monitoring).

    Perfmon Counters Low % Idle Time and High CurrentDisk Queue Length can help you diagnos Disk i/o issue.

    Hope this helps,

    Best Wishes,

    Michael Puangco | Senior Support Escalation Engineer | US Customer Service & Support

    Customer Service & Support                         Microsoft| Services

    Sunday, April 1, 2012 5:40 PM
  • Hi Tech-fun,

    Could you explain your FS4SP farm topology. How many servers, what performance characteristics do they have, and which servers run which FS4SP component?

    8gb ram sounds a bit on the short side, as the recommended is 16gb.

    Indexing is both CPU bound (and disk bound), and as you are maxing out the CPU you either have to cut down on the number of document processors in your FS4SP farm or add more servers to distribute the load. The ideal CPU load is to have the system be around 90%, to not saturate it. Saturation always leads to degradation.

    By cutting down the number of doc procs you may increase the crawl time, so it's better to add more servers or beef up the hardware on your current server (as additional servers have a premium cost).

    Regards,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/WCF4/ASP.Net4
    http://techmikael.blogspot.com/

    Monday, April 2, 2012 2:12 PM