locked
Adding people to groups based on employeeType RRS feed

  • Question

  • Our scenario is that we have around 100 000 persons in our metaverse and we want to add them to mainly one of two groups in Active Directory based on employeeType; around 15000 in one group and the rest in the other.

     

    What is the best way of accomplishing this?

     

    We've tried creating group objects with multi-value fields but run into strange and severe performance problems when adding metaverse people to the groups (around 85000 SQL SPs executing per minute when synchronizing the incomping connector space with the metaverse and only a few objects proccess in the same time frame).

    Tuesday, April 3, 2007 6:48 AM

Answers

  • In the Group Populator MA, do you have any Reference (DN) attributes?

     

    It appears that you have a different approach to group population than what I assumed before.  Instead of using a multi-valued attribute in the Connector Space, you are importing a record for each member.  This would clearly take longer.  What I would suggest you do is to create an Extensible Connectivity Management Agent (XMA) that "pre-processes" the group membership.  This way, you would have a one-to-one relationship between groups in your Group Populator MA CS and groups in the AD MA CS.  The group would have a multi-valued member attribute that clusters all of your previous eduPersonPrimaryAffiliation object.  You would likely pull this off by creating a attribute-value pair (AVP) text file with the XMA.  I know this is alot to bite off, but it would certainly speed things up considerably.

     

    Why are you having the performance issues?  If I read this correctly, each eduPersonPrimaryAffiliation adds an additional join to a MetaVerse entry.  The MetaVerse entry could potentially have many, many connectors.  Each time the MetaVerse entry is "touched", it re-evaluates all of the rules for entry.  This likely includes reprocessing every import attribute flow for every connector!

    Wednesday, April 4, 2007 5:39 PM
  • The approach that I was talking about would require the use of the Old Group Populator MA that reads off of the metaverse. (This could be a SQL ma also)

     

    This would require you to import all of your person objects to the metaverse from a seperate ma(maybe your sun one ma that has all of the users).

     

    Then you have your AD accounts join up to these metaverse objects to flow the DN of the AD object to the metaverse.

     

    Now when you do the Group Populator ma you configure it to read the DN of the user objects and build the member string with this.

     

    That way when you do the Import it has the full object with all of the members in the string member attribute.

     

    Then you could just do a direct import flow into the group metaverse object.

     

    Now for the export flow, I don't remember if you have to use an Advanced Export flow rule or if you can use a direct flow...

     

    Either way that part wouldn't be too difficult.

     

    HTH,

     

    Joe

    Wednesday, April 4, 2007 5:44 PM

All replies

  • One solution is to use a standard multi-valued string attribute in the MetaVerse as opposed to a multi-valued reference attribute.  This eliminates all of the SQL Queries related to the joins and the maintenence of the reference attribute.  For large deployments such as yours, it reduces the chatter with SQL Server substantially.  The downside is that it put's the onus on you to programmatically build the export attribute flow to the AD Connector Space.  One of my deployments provisions over 440,000 security groups ranging in size up to as many members.  I am finishing up an article on this very topic as we speak.  I will post a link this evening when it is ready for prying eyes.
    Tuesday, April 3, 2007 10:48 AM
  • Its actually quite a bit quicker when using the Multi-Valued string....

     

    I have also noticed when using the old group populate it's quite a bit faster than using the SQL ma approach, especially if you have a complex member view...

     

    I also figured out a way to initial groups with doing the multi-valued string approach and it works quite nicely...

     

    HTH,

     

    Joe

     

    Tuesday, April 3, 2007 7:54 PM
  • The funny thing is that we've used multi-value strings with uids +  (tried indexed and non-indexed without any performance difference). Just adding to and read from multi-value string-fields shouldn't be a problem, should it? We're thinking it's got to do with the joins we're doing. Tried applying SP2 Beta 2 but no difference.

     

     

    Wednesday, April 4, 2007 7:16 AM
  •  

    Just to make sure, you wouldn't want non-indexable strings, as these correlate to nText fields in the database, as opposed to nVarChar.  As for adding to and reading multi-valued string attributes, the performance will depend on your implementation.  Direct attribute flows will always be faster than advanced flows.  In this particular scenario, you would not define the member attribute as a reference attribute, neither in the Group Populator MA nor in the MetaVerse (requiring a custom attribute).  You would have a direct import attribute flow from the Group Populator MA into the MetaVerse.  This is acceptably fast.  You would then need an advanced export attribute flow from the MetaVerse into the AD MA.  I am still working on the article that will be posted ASAP.

     

    One other thought, to determine performance improvements on flows, you need to turn off MetaVerse provisioning during the initial full import.  If your provisioning logic is doing much work, it would manifest itself in the form of slower performance in the initial full import.  In addition, if the Group Populator MA is busy projecting groups, this will also slow down processing dramatically during the initial full import.

    Wednesday, April 4, 2007 12:48 PM
  • Ok; I've made a clean start in a new MIIS environment but still the same problem. The original code was run using MIIS 2003 SP1 and a local SQL 2000 SP4-database. This latest test was done using MIIS 2003 SP2 and a local SQL 2005 SP2-database.

    The problem lies with the MIIS-code that insert new values to a multi-value string (index)-attribute. In my clean environment I don't have any complex code (in fact the only code I have is in the group populator extension below to insert uid-strings into the multi-value attribute).

    Here's everything that I've done to set up the new clean environment:

    1) Added a new object class to the metaverse; called "peopleAffiliation" with two existing properties; cn and displayName and a new multi-valued string (indexable) called affiliationMembers that I've created.

    2) Created a AD MA to import some groups to use (called employee, student etc). No export rules are defined. Only two direct import flow rules for cn and displayName are defined. (I could have used any data source for the group names; the fact that it's an AD doesn't matter.)

    3) Created a group populator MA (Sun Directory server in our case; but again that's not important) to get two attributes from our data source; eduPersonPrimaryAffiliation (can be employee, student etc) and uid. In this MA I have a direct join between eduPersonPrimaryAffiliation and the cn of the group (which I've named the same in AD to avoid an advanced rule while testing) and a projection of peopleAffiliation. The code for the advanced attribute flow from uid to affiliationMembers is below (and is the only code that I've written):

    Public Sub MapAttributesForImport(ByVal FlowRuleName As String, ByVal csentry As CSEntry, ByVal mventry As MVEntry) Implements IMASynchronization.MapAttributesForImport

    ' TODO: write your import attribute flow code

    Select Case FlowRuleName

    Case "cd.luEduPerson:uid->mv.peopleAffiliation:affiliationMembers"

    If Not mventry("affiliationMembers").Values.Contains(csentry("uid").Value) Then

    mventry("affiliationMembers").Values.Add(csentry("uid").Value)

    End If

    Case Else

    ' TODO: remove the following statement and add your default script here

    Throw New EntryPointNotImplementedException()

     

    End Select

    End Sub

    Full imports from the group populator MA are extremely fast, that's not the problem. But when I try to do a full synchronization with the metaverse the first 500 objects take 4:30 minutes. If I import another 500 objects from the data source and do a delta sync with the new objects it takes 14 minutes. I've tried up 4000 objects and that takes all night so each new object in the connection space takes proportionally longer to sync. And I need 98000 for one type of group and 14000 for another...

    It's the same in reverse when I delete the group populator connector space; that also takes forever; but runs quicker and quicker for every object removed.

    When I view the metaverse groups I can list all the uid in the multi-value attribute straight away; they're all there; the problem is getting them in there.

    When I do a SQL Trace I get a lot of these:

    exec mms_getcsobjectwithguid_xlock @guid='002D564F-871B-48DE-9249-398AA5543B45'

    From what I can see and understand MIIS runs a check for every object in the connector space before it inserts the new value in the multi-value field. If it has to do 97999 checks before it can create the 98000th object... well that might take a while...

    So what do I do? Is there a way to disable the lock-check for the initial full synchronization?

    Thanks for reading all this!

    Wednesday, April 4, 2007 4:19 PM
  • In the Group Populator MA, do you have any Reference (DN) attributes?

     

    It appears that you have a different approach to group population than what I assumed before.  Instead of using a multi-valued attribute in the Connector Space, you are importing a record for each member.  This would clearly take longer.  What I would suggest you do is to create an Extensible Connectivity Management Agent (XMA) that "pre-processes" the group membership.  This way, you would have a one-to-one relationship between groups in your Group Populator MA CS and groups in the AD MA CS.  The group would have a multi-valued member attribute that clusters all of your previous eduPersonPrimaryAffiliation object.  You would likely pull this off by creating a attribute-value pair (AVP) text file with the XMA.  I know this is alot to bite off, but it would certainly speed things up considerably.

     

    Why are you having the performance issues?  If I read this correctly, each eduPersonPrimaryAffiliation adds an additional join to a MetaVerse entry.  The MetaVerse entry could potentially have many, many connectors.  Each time the MetaVerse entry is "touched", it re-evaluates all of the rules for entry.  This likely includes reprocessing every import attribute flow for every connector!

    Wednesday, April 4, 2007 5:39 PM
  • The approach that I was talking about would require the use of the Old Group Populator MA that reads off of the metaverse. (This could be a SQL ma also)

     

    This would require you to import all of your person objects to the metaverse from a seperate ma(maybe your sun one ma that has all of the users).

     

    Then you have your AD accounts join up to these metaverse objects to flow the DN of the AD object to the metaverse.

     

    Now when you do the Group Populator ma you configure it to read the DN of the user objects and build the member string with this.

     

    That way when you do the Import it has the full object with all of the members in the string member attribute.

     

    Then you could just do a direct import flow into the group metaverse object.

     

    Now for the export flow, I don't remember if you have to use an Advanced Export flow rule or if you can use a direct flow...

     

    Either way that part wouldn't be too difficult.

     

    HTH,

     

    Joe

    Wednesday, April 4, 2007 5:44 PM
  •  

    I agree with Richard on the XMA approach..

     

     with the Group Populator Code with what I said above would work nicely...

     

     

    HTH,

     

    Joe

    Wednesday, April 4, 2007 5:49 PM
  •  

    Nope, no Reference attributes.

     

    Thank you both very much for your informative answers. I'm not at all experienced with MIIS so it takes awhile for me to grasp the concepts you've described.

     

    But now I understand perfectly well why our approach is doomed for failure. It works, just barely, in small environments. After all 1000 users only takes 10 minutes to process. But 100 000 is a whole other ballgame. Like you say Richard, every time we touch the metaverse it has to check it's "integrity" and naturally that will require processing the mathematical 1000000! number of operations (or something like that).

     

    I'm just glad we decided against letting it run for a couple of days since it would get progressively and horribly slower with each new join and we would have ended up with a broken implementation of MIIS.

     

    So, the way I see it, and hopefully understand it; we have two choices:

    1. Create a new type of managment agent; a XMA. Does it work as an additional layer between the Sun MA and a connector space? Because as I understand it I don't have to create a new management agent from scratch I hope? Do you have good pointers to documentation on XMA that will get us up to speed quickly?
    2. Import the data from Sun One LDAP to a SQL database store and use the SQL management agent to select users based on the type-attribute and provision the metaverse groups that way.

    What we have to decide on is how to proceed from here... The problem is that we don't have any groups in our Sun LDAP; instead we use our existing LDAP more like a referential database than like an LDAP. The Sun MA naturally wants to work with groups whereas the SQL MA as I understand it has built-in logic for provisioning groups based on the columns in a database store?

    Wednesday, April 4, 2007 9:11 PM
  • As with most solutions, there are many different ways you could tackle this.  If it were me, I would likely create an XMA and use System.DirectoryServices to suck the data out of Sun One and dump it directly to an AVP Text file for direct import into MIIS.  This would be along the lines of your choice 1.  You could in fact leverage MIIS to pull the data out of Sun One and push it into SQL Server, but I think that would be overkill and would add that much more work for MIIS to do, not to mention the added complexity of the solution.  Besides, once the data is in SQL Server, you would need to turn around and pull it back into MIIS with some rather intricate SQL Views.  If you take the XMA approach, the code will not be too complex.  A good place to start for instruction on XMA’s is actually the Developer documentation that ships with the product.  You will be implementing a file-based, import-only Extensible Connectivity Management Agent.  You will only need to implement a single IMAExtensibleFileImport method called GenerateImportFile.  It’s almost self-explanatory.  There are also other Walkthroughs and samples out there if you Google “Extensible Management Agent".  I have not done the proper research to recommend one, but maybe someone else can pop in and let us know.
    Thursday, April 5, 2007 2:24 AM
  • Hey Richard,


    I was just wondering how you plan on getting the dn to export out to the AD groups.

    Im just trying to figure out what your thinking.... Smile

    Here's my idea...

    By importing the user from Sun to the metaverse and then joining the AD users, you can then import the AD DN into the metaverse and you have one place to query all of the information you need to create a mutli-value string attribute.

    Would that really be complex? I honestly think it would be pretty easy to setup.

    I would definitely build and XMA that reads from the SQL mms_metaverse table.(just like how the old group populator scenario worked)

    Then in the XMA you just write out to an AVP file with the group and all the members.

    This way you now have a a group with the AD User dn value.

    Then when you sync it back to the Metaverse group object, you don't need to configure it as reference attribute value, you could just use a direct flow to a multi-valued string attribute.

    This would eliminate all of the processing for the reference attribute.

    Then from the metaverse object you setup a flow to the AD object...

    I honestly forget if you have to use and advanced flow rule to access the member object.
    But either way the code wouldn't be that hard.... But I am pretty sure you could use a direct flow.

    The only other way that I could think of that would work would be to use the XMA and read info from both the Sun Directory and the AD directory which would work, but then you need to come up with how to join the two.

    Anyway I'm not trying to jump on any toes... I just like the whole though process behind this... It seems like a lot of the other posts really don't explore some the more advanced topics.

    Joe

    Thursday, April 5, 2007 3:52 AM
  • I understand. As always there is a time constraint and we really have to get going with this project. When we did our limited tests with 500 users things seemed ok; it wasn't until we did complete test of all users now at the end of our project that we hit the performance problems.

     

    We've also looked at doing direct imports using the Sun MA and using System.Directoryservices to programatically add users to Active Directory directly when synchronizing the metaverse with the AD CS and then for each user look at the user type attribute and add the user to one of our predefined groups. Then later we can import the groups from AD into the metaverse and have a join that way. Any thoughts on this approach?

    Thursday, April 5, 2007 4:21 AM
  • And I also want to add that the division by employee type is just a small part of what we need to do. We have other objects and objectclasses in our LDAP directory that contains information on departments, roles and other things.
    Thursday, April 5, 2007 12:57 PM
  • I wouldn't recommend using DirectoryServices to add users to groups.  This defeats the purpose of the state-based engine that MIIS provides you.  It will attempt to add the user to the group, even if the user is already a member of that group.  Provisioning code like that will slow down the system as well because it fires everytime to touch the MetaVerse entry.  The XMA is likely your fastest and easiest solution.

     

    Joe, I was going by the requirement that the logic for determining the group membership is coming from Sun One and cannot be derived from the MetaVerse alone.  Also, the XMA solution described would use standard reference attributes just like the Group Populator solutions.  Originally, I was under the assumption that he was using the Group Populator which is why I suggested going with standard multi-valued String attributes to speed things up a bit.  You are right though, we should tackle these more advanced topics more often.  Maybe we can take this over to a brainstorming thread in IDMCenter.com?

    Thursday, April 5, 2007 4:23 PM