none
Crawl Log error: Access Denied

    Question

  • Hi,

    I've been using MOSS for a few months. Previously, I've concentrating on developing some specific webpart, so I didn't really test the standard features like search, etc.

    Just yesterday, I do a search which should return some results, but it shows none. So I checked the Crawl Logs and it shows this 1 error:

    ====
    https://intranet.companyname.com
    Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content. (The item was deleted because it was either not found or the crawler was denied access to it.)
    ====

    I only have 1 Content Source, which is the default "Local Office SharePoint Server sites". The Defaul Content Access Account is a special account created for the crawer. I've tried remote desktop to the MOSS server, opening IE, and try to access that URL using the crawl account, and I can successfully logged in.

    Any hints on where to look for the problem?

    Regards.
    Wednesday, October 03, 2007 3:35 AM

Answers

  • Well, like I said in my post, I already set the Default Content Access Account, and I already tried to use that account (using a browser) to access the site, and it works. But, the crawl log still say Access is Denied.

    After some digging, I finally found the problem. Apparently, the default access for the crawler is to use NTLM authentication to the site. If we need to use Basic Authentication, we need to create a Crawl Rule and uncheck the "Do not allow Basic Authentication" checkbox. We still can use the default crawling account, because the account is not an issue here.

    Why I need to do this is because the setting for my sharepoint is to use Basic Authentication (network requirement, because a lot of my users use non-Microsoft proxies that doesn't support NTLM).

    Hope this helps for others who encounter the same problem. I still have a problem with the crawl result though, but I will post it on another topic.

    Regards.
    Wednesday, October 03, 2007 9:21 AM

All replies

  • I found this entry in this forum:
    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1626315&SiteID=1
    which eventually lead to this blog:
    http://www.keirgordon.com/2007/04/bdc-crawl-error-parameter-is-incorrect.html

    Unfortunately, it doesn't help me. My "Office Sharepoint Server Search" service is running on a specific account, and not using Local Service.

    Anyway, I wonder how he can see the "login failed" message in the event viewer, because in my event viewer, I can only found a warning entry with this content:

    =======
    The start address <https://intranet.jatis.com> cannot be crawled.

    Context: Application 'SharedServices1', Catalog 'Portal_Content'

    Details:
        Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content.   (0x80041205)

    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
    =======

    So, I can not confirm which account the crawler use.

    On the other hand, I see a discrepancy here. The "Office Sharepoint Server Search" service is using "sps.admin" account, which is the Farm Administrator account, while the SSP setting is using "appl.sps-crawler" as the Default Content Access Account. Originally, I use the admin account (sps.admin) for the Default Content Access Acount, but afterwards I modified it to use the new account (appl.sps-crawler). I wonder why the service account is not changed. Should I change it manually? Or is there a way to modify it from MOSS? How can I know which account is now used by the crawler?
    Wednesday, October 03, 2007 4:23 AM
  • You can see/change the account that is used from SharePoint Central Administration site

     

    Goto:

    Shared Service Provider ->  Search Settings ->  Default Content Access Account.

     

    You need to check that this account has Read Access to the web application you are crawling,  if you have changed this you could provide this via the Central Admin ->  Application Management

     

    Choose Policy for Web Application under Application security, select the required web application and choose Add Users,  normally choose All Zones and then add you crawl account and give it full read permission.

     

    Rerun the crawl and you should see items being index.

     

    HTH

     

    Andrew

     

    Wednesday, October 03, 2007 8:12 AM
  • Well, like I said in my post, I already set the Default Content Access Account, and I already tried to use that account (using a browser) to access the site, and it works. But, the crawl log still say Access is Denied.

    After some digging, I finally found the problem. Apparently, the default access for the crawler is to use NTLM authentication to the site. If we need to use Basic Authentication, we need to create a Crawl Rule and uncheck the "Do not allow Basic Authentication" checkbox. We still can use the default crawling account, because the account is not an issue here.

    Why I need to do this is because the setting for my sharepoint is to use Basic Authentication (network requirement, because a lot of my users use non-Microsoft proxies that doesn't support NTLM).

    Hope this helps for others who encounter the same problem. I still have a problem with the crawl result though, but I will post it on another topic.

    Regards.
    Wednesday, October 03, 2007 9:21 AM
  • Hi.

    I faced the same problem on MOSS 2007 Server running on Windows Server 2008 OS. I managed to fix this problem by disabling look backup check, which is enabled by default on Windows Server 2008 Operating Systems.

    Complete detail can be found at:http://www.mossgurus.com/adnan/Lists/Posts/Post.aspx?ID=22

    Thanks
    Adnan Ahmed
    http://www.mossgurus.com

    Saturday, May 02, 2009 4:09 PM
  • Awesome!  I know this post is pretty old, but this fixed my issue as well.  We were running NTLM initially and crawling was failing so I followed the advice of Sandeep in this posting http://social.technet.microsoft.com/Forums/en-US/sharepointsearch/thread/84f93fbe-f4a6-4683-b25b-b595b9006ad7?prof=required and then search was working.  Then recently we switched over to Basic Authentication and searching stopped working again with the same 'Access Denied' error.  Creating crawl rules and checking "Specify a different content access account", even though i used the same default crawling account, and then unchecking "Do not allow Basic Authentication" like Irving said, got rid of the errors.  Thanks for your post.
    Thursday, May 28, 2009 6:02 PM
  • Why I need to do this is because the setting for my sharepoint is to use Basic Authentication (network requirement, because a lot of my users use non-Microsoft proxies that doesn't support NTLM).

    Hope this helps for others who encounter the same problem. I still have a problem with the crawl result though, but I will post it on another topic.

    Regards.

    Hi Erving or MOSS team

    I'm facing a similar issue.Well my MOSS 2007 webapplication is using basic authentication. An Access denied error comes up when I'm crawling the content.

    I'm able crawl when the web applicaiton is using windows authenticaiton but once it is changed to basic authenticaiton the crawl fails.So its not an issue with the lookback check.

    The webhosting team insists us to use basic authentication for the website to work on internet.

    The below technet link explains about unchecking the don't use basic authenticaiton check box but i'm still getting the access denied error.

    http://technet.microsoft.com/en-us/library/cc262934.aspx

    Any help would be greatly appreciated

    Thanks in advance
    Murali Raju

    Saturday, November 21, 2009 5:02 PM
  • Hi All,

    I got out of this issue finally. Reason for error: I didn't gave the domain\<username>

    Quick summary for solving the access denied error in MOSS 2007 during crawling

    (Check 1) Crawl account has read access to the contetn being crawled


     You need to check that this account has Read Access to the web application you are crawling,  if you have changed this you could provide this via the Central Admin ->  Application Management 

    Choose Policy for Web Application under Application security, select the required web application and choose Add Users,  normally choose All Zones and then add you crawl account and give it full read permission.

    (Check 2) Regedit posted sandeep at http://social.technet.microsoft.com/Forums/en-US/sharepointsearch/thread/84f93fbe-f4a6-4683-b25b-b595b9006ad7?prof=required 

    (Check 3) When u change NTLM to basic authentication your crawl stops working.

                  Create a crawl   rule http://technet.microsoft.com/en-us/library/cc262934.aspx

    Thanks in advance
    Murali Raju


    • Proposed as answer by Murali Alluri Wednesday, August 10, 2011 11:35 AM
    Saturday, November 21, 2009 5:30 PM
  • Awesome!  I know this post is pretty old, but this fixed my issue as well.  We were running NTLM initially and crawling was failing so I followed the advice of Sandeep in this posting http://social.technet.microsoft.com/Forums/en-US/sharepointsearch/thread/84f93fbe-f4a6-4683-b25b-b595b9006ad7?prof=required and then search was working.  Then recently we switched over to Basic Authentication and searching stopped working again with the same 'Access Denied' error.  Creating crawl rules and checking "Specify a different content access account", even though i used the same default crawling account, and then unchecking "Do not allow Basic Authentication" like Irving said, got rid of the errors.  Thanks for your post.
    Initially my problem was, like u guys, the Search functionality on a portal wasn't working. I followed the error messages in the Crawl Log and found out the Portal wasn't being crawled (it had crawled 0 sites in the crawl log) .
    This lead me to re-entering the credentials for the Default Content Access account, but it still didn't work (to my suprise.. I was like.. "this should be a piece of cake, but what the heck is this..?")

    The "Basic Authentication" part in the reply I quoted above, actually triggered me to double check with IIS if this authentication method (Basic) is being used. This was the case!. From there on I followed the tips in this post and this led me to the final solution:
     
    - Created an extra crawl rule for the site, marked "Specify a different content access account" and unchecked "Do not allow Basic Authentication"
    - Started the Full Crawl and voilà!

    Troubleshooting should be as simple as this.

    Thanx guys, sharing knowledge really works out perfectly. You rule!
    Monday, November 23, 2009 11:48 AM
  • Thanks Murali and Sandeep!  My issue was solved by Check 2 (Regedit - DisableLoopbackCheck).
    Wednesday, September 15, 2010 4:39 PM