none
FAST content API with docacl RRS feed

  • Question

  • Hello,

    How can we find out the ACL properties of a SharePoint listitem programmatically? is this the SPSecurableObject.AllRolesForCurrentUser property? can we use this property as the docacl property for FAST indexing using the FAST Content API for pushing content into the FAST Index?

    Regards

    Saturday, April 14, 2012 9:53 AM

Answers

  • Hi,

    I will answer the last question first. The problem is that when an item is added with the SharePoint crawler framework the item is assigned an id like "ssic:1234". You have no way to generate the same number yourself for new items as it's being auto incremented in the crawler database. So avoiding duplicates becomes hard.

    As for the first question I'm not aware of this, and I would try to stay away from push indexing with the Content API if I could. Seems the benefit is not worth the effort for most cases (including security). Reading up a bit it seems you might be able to send in the security in the standard SID format instead of the SDDL SID format, but not 100% sure. That will help you a bit on the way. (SDDL description). Also take a look at this blog post about injecting ACLs.

    You probably know the Content API was discontinued the day it was released, which leads me to think it's not very future proof to rely on using it. We can only hope we get push indexing in SharePoint one day with the built-in crawlers.

    Regards,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/WCF4/ASP.NET4
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    • Marked as answer by RoelBSS Monday, April 16, 2012 8:33 AM
    Monday, April 16, 2012 7:52 AM

All replies

  • Hi,

    Not really. The ACL's in the index as expanded as Active Directory ACL's. In order to provide the correct information you would per item have to get all SharePoint groups, AD groups and AD individuals who have read access to the item. For each SP role/group you have to expand it into whatever AD group/user is a member of that entity. Once you have complete list, you have to transform the SID list into SDDL format (if I'm not mistaken). The ACL's are also encoded using a custom base32 encodig inside the pipeline.

    So, a lot of work to get this going.

    I assume you are trying to do push based indexing on SP content? Or what is the usecase instead of doing rapid incremental crawls?

    Regards,
    Mikael Svenson 


    Search Enthusiast - SharePoint MVP/WCF4/ASP.Net4
    http://techmikael.blogspot.com/

    Sunday, April 15, 2012 10:49 AM
  • Hi Mikael,

    Thanks for the quick response. That is exactly the usecase: upon adding a sharepoint listitem, we want to immediately push the metadata for this listitem into the fast index, by firing an event receiver and using the fast content api. It sure sounds like a lot of work for construcing the docacl info. You don't know of any class that would do the work you described programmatically for us?

    second question (don#t know if I should open a new topic for this?): when using the content api to push content into the fast index, upon crawling this content later, we get to have duplicates in the fast index: 

    - the item that was pushed in using the content api

    - the item as found by the crawler

    Is the a way that we can avoid having duplicates in the index (and I understand that having rapid incremental crawling is one way to solve this issue, but that is not the usecase sadly :) )

    Thanks again!


    • Edited by RoelBSS Monday, April 16, 2012 7:23 AM added second question
    Monday, April 16, 2012 7:17 AM
  • Hi,

    I will answer the last question first. The problem is that when an item is added with the SharePoint crawler framework the item is assigned an id like "ssic:1234". You have no way to generate the same number yourself for new items as it's being auto incremented in the crawler database. So avoiding duplicates becomes hard.

    As for the first question I'm not aware of this, and I would try to stay away from push indexing with the Content API if I could. Seems the benefit is not worth the effort for most cases (including security). Reading up a bit it seems you might be able to send in the security in the standard SID format instead of the SDDL SID format, but not 100% sure. That will help you a bit on the way. (SDDL description). Also take a look at this blog post about injecting ACLs.

    You probably know the Content API was discontinued the day it was released, which leads me to think it's not very future proof to rely on using it. We can only hope we get push indexing in SharePoint one day with the built-in crawlers.

    Regards,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/WCF4/ASP.NET4
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    • Marked as answer by RoelBSS Monday, April 16, 2012 8:33 AM
    Monday, April 16, 2012 7:52 AM
  • Hi Mikael, 

    Thanks for your help so far! Because of some of the feedback you gave, we are now considering other options to implement the scenario needed, for instance by trying to implement some sort of priority crawling. 

    I've asked a question on the possibilities, here: http://social.technet.microsoft.com/Forums/en-US/fastsharepoint/thread/a68489e9-8547-4fb3-813b-02de3e33a240/#a68489e9-8547-4fb3-813b-02de3e33a240

    Regards,

    RoelBSS


    • Edited by RoelBSS Thursday, April 26, 2012 8:10 AM updated link
    Thursday, April 26, 2012 7:43 AM
  • Hi Mikael,

    Another question on this topic, as you suggested to use frequent incremental crawling as a solution: How does SharePoint/Fast know whether an item is to be crawled in the next incremental crawl run? Is there an entry made in one of the ContentDBs upon upload/change of a SharePoint document? If so, we could use an event receiver wired to adding/updating of SP ListItems, which would then directly change/make an entry in a ContentDB on the SharePoint SQL Server, which moves the item to the top of the incremental crawl queue, so that it would be crawled as soon as possible. Is this a valid option?

    Thanks again!

    Regards,

    RoelBSS

    Thursday, May 3, 2012 12:24 PM
  • Hi,

    When SharePoint is doing incremental crawls of SP content it is checking an internal change log for items which have changed since the last crawl. So it is not iterating all content and checking the time stamp. Basically incremental crawls should work as expected already.

    But there is some overhead for a crawl to start and finish, which takes longer than the actual crawl when few items have changed. This you cannot get around.

    Regards,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/WCF4/ASP.NET4
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Thursday, May 3, 2012 12:45 PM
  • Hi Mikael,

    I am trying to create docacls for an item that I push into the FAST index using the contentAPI, but when I view the fixml for this document, I only seem to get <context name="bcondocacl"><![CDATA[ ǂ unknown ǂ ]]></context>. I am using the (commented out) section of the SetSecurityDescriptor method in the DocumentFeederExample class, and I use my username (domain\username) as a string input to test, like so:

    string userName = @"domain\myusername";
                NTAccount account = new NTAccount(userName);
                SecurityIdentifier sid = (SecurityIdentifier)account.Translate(typeof(SecurityIdentifier));
                DiscretionaryAcl dacl = new DiscretionaryAcl(false, false, 1);
                dacl.AddAccess(AccessControlType.Allow, sid, 0x7, InheritanceFlags.None, PropagationFlags.None);
                CommonSecurityDescriptor sd = new CommonSecurityDescriptor(false, false, ControlFlags.DiscretionaryAclPresent,
                                                                              sid, sid, null, dacl);
                doc.SecurityDescriptor = sd.GetSddlForm(AccessControlSections.All);

    Am I doing anything wrong here? The SecurityDescriptor property of the document is being filled with this string in my example: "O:S-1-5-21-1503372869-1711278804-1989271447-1705G:S-1-5-21-1503372869-1711278804-1989271447-1705D:(A;;CCDCLC;;;S-1-5-21-1503372869-1711278804-1989271447-1705)", but as mentioned, I only see unknown in the fixml for the doc..

    any ideas?

    Thanks again!

    Regards,

    RoelBSS

    Friday, May 11, 2012 9:42 AM
  • Hi,

    As I haven't tried this myself I'm not sure :) Have you tried adding a spy stage before and after the security stage in the pipeline to see what data is available?

    Thanks,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/WCF4/ASP.NET4
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Friday, May 11, 2012 10:40 AM
  • Hi,

    Thanks for the suggestion, I just tried that, I added spy stages between all of the SAM module processors.

    <processor name="Spy"/>
    <processor name="MSPermissionDecoder"/>
    <processor name="Spy0"/>
    <processor name="SharePointClaimConverter"/>
    <processor name="Spy1"/>
    <processor name="DocumentSecurityUnknown"/>
    <processor name="Spy2"/>

    Now what I found out is that the generated SDDL string ( "O:S-1-5-21-1503372869-1711278804-1989271447-1705G:S-1-5-21-1503372869-1711278804-1989271447-1705D:(A;;CCDCLC;;;S-1-5-21-1503372869-1711278804-1989271447-1705)") is set to the attribute 'docaclms', and is maintained throughout the stages. There is still no 'docacl' property present throughout the SAM stages, up until we are leaving the SharePointClaimConverter stage, so when the DocumentSecurityUnknown stage is run, the 'docacl' attribute is added with a value of 'unknown'... Maybe I should add a custom stage to copy the docaclms value to the docacl attribute? I am a bit confused here, because this should be possible using the contentAPI...

    Regards,

    RoelBSS

    PS: after checking the spy stages on a normal crawl, i saw that the docacl property is added after the MSPermissionDecoder stage. This does not happen however when working with the ContentAPI..

    • Edited by RoelBSS Friday, May 11, 2012 12:34 PM added testresult
    Friday, May 11, 2012 12:24 PM
  • Hi,

    Which field is the SDDL present in during a SP or file server crawl, before it's added to the docacl field? (I don't have a VM at hand to check right now).

    But I would assume docaclms would be copied over to docacl, in order for acl to work with the ContentAPI. Sorry I cannot be of ore help.

    Thanks,
    Mikael


    Search Enthusiast - SharePoint MVP/WCF4/ASP.NET4
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Friday, May 11, 2012 7:44 PM
  • Hi,

    The SDDL string is present in the docaclms field, and in the MSPermissionDecoder Stage, it is copied to the docacl field. I found out how to be able to correctly fill the field now. The problem was, that the built-up SDDL string wasn't valid, and was disregarded in the MSPermissionDecoder Stage. Afterwards, the DocumentSecurityUnknown took over (as there was no value in the docacl field) and wrote "unknown" as a result in the field. 

    I found a code snippet that provided me with a correct SDDL string for a single user here: http://msdn.microsoft.com/de-de/library/gg294169.aspx

    So now I up for the next challenge, how to find out which users/groups have access to the current list item (SharePoint side). Then I can add the users (as per code snippet from the mentioned msdn link) and create a correct SDDL string. If you have any pointers for me on doing that, that would be great. I am specifically interested in using groups for the SDDL descriptor of the SP list item. 

    Regards,

    RoelBSS

    Monday, May 14, 2012 7:52 AM
  • Hi RoelBSS,

    I have written code to expand SharePoint groups to AD users/groups in the past but that code is long gone. A quick search on google led me to this PowerShell script: Get-EffectiveSPPermissions on Codeplex (http://sharepointpsscripts.codeplex.com). You could probably look over the code and pull out the relevant parts and use them directly as it uses the object model internally.

    Once done, you should blog this :)

    Regards,
    Mikael Svenson


    Search Enthusiast - SharePoint MVP/MCT
    http://techmikael.blogspot.com/
    Author of Working with FAST Search Server 2010 for SharePoint

    Monday, May 14, 2012 5:37 PM
  • MSPermissionDecoder stage ignores  Full read permissions:  CCDCLC

    Change it to Full access on the addaccess call.

    Cheers.

    Thursday, May 31, 2012 7:35 PM