none
Issue with sending query packet using web-service for more than 2000 results. RRS feed

  • Question

  • Hi all,

      I am getting error  (generally if total number of rows  returned are more than 2000 by querex() web-method) while sending query packet using web-service, and adding specific range to the result set in query packet,

    I am using range section of query packet which fetches only 50 results at a time.

    For example :- 

    if total rows returned are 400

    to extract last 50 results i use <start>451</start><count>50</count> which works fine. But if total rows returned by queryex() web-method is 2000, and if pass <start>1951</start><count>50</count> in range section , it throws error "The search request was unable to execute on FAST Search Server". this happens only if total rows returned by queryex() web-method is more than 2000 and while fetching last 50-70 results. Please let me know how do i resolve this.

    Thanks,

    Nikhil



    Nikhil

    Friday, April 20, 2012 12:40 PM

All replies

  • Hi All,

       Looks like the problem not only exists for last 70 to 50 results, it exists for last 15% to 10% of results for total rows returned by queryex() web-method is more than 2000.

    Anybody have or been through such problem ?

    Thanks,

    Nikhil S


    Nikhil

    Monday, April 23, 2012 5:29 AM
  • Hi,

    I'm having a similar problem. In fact I think the total rows is being wrongly calculated when I remove duplicate results.

    Therefore, the last results (might be 10%, 15% or more) don't exist but my paging controls assume that they exist, according to the total rows.

    How did you overcome this issue?

    Thanks,

    Pedro V

    Thursday, May 3, 2012 2:52 PM
  • Hi

    How complex is your query? And do you use wildcards? It could be that you encounter a bug in FS4SP. Read this thread for more information: http://social.technet.microsoft.com/Forums/en-US/fastsharepoint/thread/3fb37385-73ef-4282-b1f6-a3d2b2c251c9

    If not, then it could be related to that there is a maximum offset. This means that no matter what you will not get search results past a certain number. In FAST ESP it was default set to 4000: http://support.microsoft.com/kb/2529087, and believe it is set to this number in FS4SP as well. I think you can manually increase this number in FS4SP as well, but this will break your support. And if you set it too high it will impact the search engine's performance. Regardless, there will allways be limit to how many search results you can actually read back.

    Why do you want that many results in the first place? It is a bit unusual for search engines to deliver data for that many results. It is not what a search engine does best. Maybe a different tool is better suited for what you are trying to accomplish? And if it is a person performing the search you would think that if any interresting results was not found among the first few hundred he or she would try a different set of keywords.

    Regards Gunnar

    Thursday, May 3, 2012 3:32 PM
  • The query is very simple, only for testing purposes.

    If I don't remove duplicates I get even more items and it works fine, so its not a limit problem.

    When I trim my duplicates according to a certain property,  the "total rows" extended property being returned by the Relevant Results data set is wrong (the count has more items than it should). 

    Is it possible that the "total rows" is a estimated value? Maybe it gets a sample of results, counts the duplicates and then extrapolates the value to give the total count of results. I'm just guessing because I haven't seen this anywhere. 

    Any help is welcome.

    Thanks

    Pedro V

    Thursday, May 3, 2012 5:19 PM
  • Ok, so even without removing duplicates the count is not correct?

    The value is not an estimated value when using FS4SP. I know that the other search engines by MS does this for some counts, but FS4SP does not do this.

    I am suspecting that the number returned by the QueryEx method is not the right one.

    What is the value of the extended property "IsTotalRowsExact"?

    Can you also check the latest querylogs on the FS4SP server (Fast installation dir\var\etc\logs\querylogs\) for your queries? At the end of the log entry for each query a number indicating number of hits is present. "STATS(0.0160, 0.0150, 21)" indicates 21 hits.

    Have you tried running the same query not through the QueryEx method, but directly on the Sharepoint Server? What is the output of this:

    Add-PSSnapin Microsoft.Sharepoint.Powershell -ErrorAction SilentlyContinue;
    
    $siteurl = "http://your site url/";
    $site = New-Object Microsoft.SharePoint.SPSite $siteurl;
    $query = New-Object Microsoft.Office.Server.Search.Query.KeywordQuery $site;
    $query.ResultTypes = [Microsoft.Office.Server.Search.Query.ResultType]::RelevantResults;
    $query.ResultsProvider = [Microsoft.Office.Server.Search.Query.SearchProvider]::FASTSearch;
    $query.EnableFQL = $true;
    $query.QueryText = "your query";
    $query.TrimDuplicates = $true;
    $counter = $query.SelectProperties.Add("Title");
    
    try {
    	$result = $query.Execute();
    	$resulttable = $result.Item([Microsoft.Office.Server.Search.Query.ResultType]::RelevantResults);
    	$rows = $resulttable.Table.Rows;
    	foreach ($row in $rows) {
    		"Title					: " + $row.Title;
    	}
    	"Total number of rows: " + $resulttable.TotalRows;
    	"Total number of rows including duplicates: " + $resulttable.TotalRowsIncludingDuplicates;
    	"Is total rows exact : " + $resulttable.IsTotalRowsExact;
    } catch {
    	$_.Exception;
    	Write-Host Could not Execute Search;
    }

    Can you also post your QueryPacket xml?

    Regards Gunnar

    Friday, May 4, 2012 9:12 AM
  • By the way, this issue seems related (I believe they are using the Query method):

    http://social.technet.microsoft.com/Forums/en-US/fastsharepoint/thread/74e683c7-226b-4a40-ba5e-471aae719c45

    Friday, May 4, 2012 10:03 AM
  • Hi,

    In order to get the right count for total rows you have to include either "write" or "size" as refiners in your query. Those two properties will exist for all items, and that count is used for giving the total rows back. So in order to get a correct count for total rows you have to include a refiner for a property which exists on every single item in the index.

    It could of course also be an issue with duplicate removal not affecting the refinement count.

    Thanks,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/WCF4/ASP.NET4
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Friday, May 4, 2012 8:06 PM