Asked by:
SharePoint 2013 Crawl complex URLs

Question
-
Hello folks, thank you for taking a look.
I am wondering if anyone has successfully been able to crawl complex URLs in SharePoint 2013. I have a farm where I have set up a crawl rule to include these links:
I have a content editor web part set up on the root page of SharePoint 2013 (i.e. /pages/default.aspx). In this CEWP, I have added a few links to SharePoint content with query string variables at the end (i.e. pages/videos.aspx?vid=123456)
The crawl log shows that default.aspx is being crawled just fine, but crawler is not picking up the links in the CEWP.
As a side note, I have set up the site to crawl all webpart content regardless of permissions:
What am I missing here?
Thanks in advance,
Phil
EDIT 03/10/2014
There seems to be some confusion so I want to be clear. Content from my CEWP is being crawled successfully by SharePoint. When I search for content on the CEWP, it is returned on the search results page. The problem is that I expect the crawler to follow links on the CEWP which it is not doing.
As mentioned above, the use case is a dynamic page in which video content is loaded based on an ID value in the query string.
- Edited by Phillip Demro Monday, March 10, 2014 3:41 PM Clarification
Friday, March 7, 2014 8:34 PM
All replies
-
Sunday, March 9, 2014 2:45 AM
-
Thanks for your response Brandon. In my second screenshot I've shown the setting you reference as "index all web parts on this site" (regardless of permissions).
Regardless of what I set this to, I still get the same results. SharePoint will not crawl dynamic content. Indexing dynamic content seems like a pretty common task so I would be surprised if there isn't a supported path.
Monday, March 10, 2014 12:50 PM -
Please check below articles:
http://www.bluedoglimited.com/SharePointThoughts/Lists/Posts/Post.aspx?ID=150
Or
You can use below code to check if Index instead of noIndex can help you solve the issue:
Monday, March 10, 2014 1:09 PM -
can you tell me whether this not crawling links within CEWP or any links placed on your page. if none of the links get crawled then go to your respective content source and make the required changes in HOP/ URL depth settings which will help you crawl links placed in the sites(any sites)Monday, March 10, 2014 1:23 PM
-
1. Crawl behavior is the same for both the home page and other pages in a document library. Probably due to updates since SharePoint 2003
2. Per my edit on the original post, the CEWP is being crawled
Monday, March 10, 2014 3:42 PM -
This is a SharePoint content source and link depth is not enabled in the Content Source settings.Monday, March 10, 2014 3:43 PM
-
Hi Phillip,
I know this is an old question, but I am struggling with this exact same problem for one of my customers.
Did you manage to have the complex URLs crawled in your site ?
I have tried everything you mentionned and no luck, always the same result, search doesn't show dynamic content...
Thanks in advance for your answer.
Best regards,
Gilles Kern
- Edited by GillesK Wednesday, March 1, 2017 4:10 AM
Wednesday, March 1, 2017 4:08 AM -
Make sure you're using a web content source rather than a SharePoint content source. Looking at my notes I think this was a requirement for crawling complex URLs
- Proposed as answer by GillesK Thursday, March 2, 2017 7:16 AM
Wednesday, March 1, 2017 12:50 PM