locked
Idea to automate the clean-up of orphaned EREs RRS feed

  • Question

  • Hi again.  I'm after some anecdotal/theoretical assessment on an idea I have successfully implemented in a lab just now to clean up 13000 orphaned EREs.  I am considering implementing this idea in the Production environment, but I want to find out if there are any good reasons why NOT to do this ... particularly as it strikes me that if orphaned EREs were supposed to be deleted immediately they become orphaned then this would be OOTB FIM functionality!!!

    What I have done is created the following:

    • a set (Orphaned EREs) defined as "Select expected rule entry that match all of the following conditions:
      Resource Parent not in All Objects"
    • a workflow (Delete orphaned EREs - run on policy update = ON) invoking a custom activity to delete an /ExpectedRuleEntry[ObjectID='[//Target/ObjectID]']
    • a set transition MPR (Orphaned ERE objects are automatically deleted) to invoke the above workflow whenever an ERE falls into the above set

    The above worked like a charm, deleting all 13K orphaned EREs in an hour or two.

    Question is ... what are the implications (besides a little CPU overhead) on running such a "housekeeping" activity in a production environment?  I was thinking of making the set temporal based on one of the many DateTime attributes on an ERE, but it is not clear which one to use ... but leaving this to run at say 1 am every night might actually be more of an overhead than running it as and when an ERE is orphaned.

    Thoughts anyone?

    TIA


    Bob Bradley, www.unifysolutions.net (FIMBob?)
    Wednesday, December 15, 2010 5:45 AM

Answers

  • I have kind of protection from accident object deletion in external system. so MV object deletion rule is set up to fire only from FIM MA.

    once OSR is removed and object is deleted in external system is remains on FIM portal and in MV with all attributes for another 30 days (hand made recycle bin for an external MA :) ). 

    • Marked as answer by UNIFYBobMVP Friday, December 17, 2010 11:45 AM
    Friday, December 17, 2010 10:28 AM

All replies

  • Ignoring ERE cleanups for a moment, you hit the nail on its head - the main consideration in conjunction with these operations is the performance impact.
    As an alternative, you could also just schedule a script that runs during the night - comparable to a run profile automation.
    The tradeoff of an automated script is that a script can fail to do its job.
    In the case of a cleanup process, this is probably not too bad because the worst case is that you have unnecessary data; however, this kind of data doesn't prevent your system from working correctly.

    Regarding orphaned EREs, I belief that a solution designer should also look at how to avoid creating them.
    It means, you have deleted an object that has at least one active OSR - what is the impact of this on your environment?
    So, from that perspective, it might be better to prevent objects with active OSRs from being deleted.
    While you might be authoritative for FIM, you might not have the required permissions to do the required cleanups in the external systems.

    So, what I'm trying to get at is that there should be first a discussion on where the orphaned EREs came from and how to avoid them before looking into a cleanup solution - does this make sense?

    Cheers,
    Markus


     

     


    Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
    Wednesday, December 15, 2010 1:36 PM
  • Markus - thanks for the reply.

    Firstly, I am not a fan of an external scheduled task if I can avoid it as these tend to be "out of sight, out of mind" - however if this is considered "best practice" then maybe that's the way to go.  I would prefer to continue to drive this from a set transition MPR, but make it a temporal set, which means I would need to drive it from a date.  Is one of the date bindings for an ERE appropriate here?

    Secondly, the EREs I deleted were caused by a design change where object classes and their corresponding attribute flows were removed from the solution entirely ... but I'd read about the occurrance of these in a day-to-day context and figured this might be a good idea.  I understand this would be band-aiding, though, if what you are saying is treat the cause and not the symptom ... I guess I was wondering if an orphaned ERE could occur through no fault of the design ...


    Bob Bradley, www.unifysolutions.net (FIMBob?)
    Wednesday, December 15, 2010 1:43 PM
  • Bob, from my point of view - MPR with a transition set is an easy solution to remove orphaned EREs... but! I would never do this in my production.

    the reason for this is a question: why do I have these EREs at all? I know if I would delete a group object with an active OSR it would be deleted from AD, SQL MA, Sharepoint MA and so on by the object deletion rule in MIIS. I do not care about that ERE. in this particular case.

    but in another scenario I would think of placing custom WF action as a first one in AuthZ stage on MPR allowing to delete a group. having your MPR to automatically delete such EREs can hide a problem with existing design or failed workflows... who knows.

    it really depends on design.

    as for me - I do use cmdlet from Markus and run it manually once a week.

    Wednesday, December 15, 2010 7:34 PM
  • The simple scenario where you have an object deletion rule set to anything other than the FIM MA is going to cause these things to be orphaned. IMO it's not really a design flaw if you're using the product as intended but part of the product (EREs in this case) don't talk to the other part of the product.

    One of the MCS folks that posts here sometimes has a solution much like Evgeniy describes which uses an AuthZ WF that fires on delete.


    My Book - Active Directory, 4th Edition
    My Blog - www.briandesmond.com
    Thursday, December 16, 2010 3:14 PM
  • Thanks for your replies guys ... makes sense to me, and Brian/Evgeniy - your last comments about OSRs and the object deletion rule explains why I am continuing to get these EREs under some circumstances (but this being a development environment I am aware that a lot of them have not arisen from realistic production scenarios ... such as schema cleanup or use case replay).  I am curious to know more about the MCS idea, because the ERE in itself has very little useful info in it and wouldn't be much use to an admin trying to make a decision - so I presume you mean firing an AuthZ just before the ERE is orphaned?
    Bob Bradley, www.unifysolutions.net (FIMBob?)
    Thursday, December 16, 2010 11:21 PM
  • Ohhhh - please don't get me wrong, I didn't mean to indicate a best practice here.
    To me, this is a perfect example for a scenario that doesn't have a "silver bullet" solution.

    One can either use an external cleanup script / process or something that is integrated into the FIM processing model.
    Like you have indicated it - "I'm not a fan of..." - is perfectly acceptable as long as an approach doesn't violate best practices - something like cleaning up a connector space by whipping the related table in the database :o)

    Just making sure, I'm not trying to defend external cleanup processes; I'm just trying to make sure that using them is not interpreted by anyone as a "second best approach".

     Whether or not an orphaned ERE could occur through no fault of the design is a tough one to answer - I wonder if there is an answer because it depends...
    When you delete an object with a populated ERL, you get "one" - is this a design flaw?

    Should the system automatically delete them?
    It might be OK to delete them and it might also be a design flaw if the system would do this.

    This is to me debatable.

    Cheers,
    Markus

     

     

     

     

     


    Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
    Friday, December 17, 2010 1:00 AM
  • Thanks for your replies guys ... makes sense to me, and Brian/Evgeniy - your last comments about OSRs and the object deletion rule explains why I am continuing to get these EREs under some circumstances (but this being a development environment I am aware that a lot of them have not arisen from realistic production scenarios ... such as schema cleanup or use case replay).  I am curious to know more about the MCS idea, because the ERE in itself has very little useful info in it and wouldn't be much use to an admin trying to make a decision - so I presume you mean firing an AuthZ just before the ERE is orphaned?
    Bob Bradley, www.unifysolutions.net (FIMBob?)

    AuthZ WF will fire before the Delete occurs so at that point you can walk the ERE/DRE links and remove the associated objects before the delete is processed.
    My Book - Active Directory, 4th Edition
    My Blog - www.briandesmond.com
    Friday, December 17, 2010 2:20 AM
  • I've ended with prohibiting regular users to delete objects on a portal and removed these rights from all MPRs.

    instead I have a checkbox for lets say a 'role' named 'disable this role' that brings an object to 'To be deprovisioned' set and removes all OSRs from a role.

    and administrator has to manually delete objects once they appear in another set 'to be deleted' which is 30 days after all OSRs were removed.

    the second step was setting up MV object deletion rule by FIM MA only.


    Friday, December 17, 2010 7:43 AM
  • I've ended with prohibiting regular users to delete objects on a portal and removed these rights from all MPRs.

    instead I have a checkbox for lets say a 'role' named 'disable this role' that brings an object to 'To be deprovisioned' set and removes all OSRs from a role.

    and administrator has to manually delete objects once they appear in another set 'to be deleted' which is 30 days after all OSRs were removed.

     the second step was setting up MV object deletion rule by FIM MA only

    Wow, this is actually very cool!
    I like your approach with the disable checkbox a lot.

    Cheers,
    Markus


    Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
    Friday, December 17, 2010 7:58 AM
  • Markus,

     

    for the perfect solution this checkbox should trigger Transition Out MPR and take object off the 'Enabled Roles' set and start Remove Sync Rules WF.

    same with object deletion - there must be another MPR that should raise object delete action.

    just have no time to do this :)

    Friday, December 17, 2010 9:59 AM
  • It is possible that I have missed something - but why do you need a MPR for the object deletion?
    After removing an object from the scope of all OSRs, you just need to sync it.
    In other words, the deletion is imported from the synchronization service.

    Cheers,
    Markus


    Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
    Friday, December 17, 2010 10:24 AM
  • I have kind of protection from accident object deletion in external system. so MV object deletion rule is set up to fire only from FIM MA.

    once OSR is removed and object is deleted in external system is remains on FIM portal and in MV with all attributes for another 30 days (hand made recycle bin for an external MA :) ). 

    • Marked as answer by UNIFYBobMVP Friday, December 17, 2010 11:45 AM
    Friday, December 17, 2010 10:28 AM
  • Ahhhh - very smart!
    Have you ever thought about documenting your solution?
    This is a great topic for a wiki article.

    Cheers,
    Markus

     


    Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
    Friday, December 17, 2010 11:37 AM
  • never thought about such things.

    I'll have 10 days off in January, so may be...

    actually it was caused by real life's practice:

    1. imaging that you have an application role that grants access to something, e.g. 'HR manager' in Axapta.

    2. one day you've migrated from Axapta to new Dynamics AX with its own roles

    3. you don't need old 'HR manager' any more

    4. to remove a role from business application catalog you have a proper workflow and everyone has approved that role removal (actually disable checkbox)

    5. AD group is removed and remains in AD recycle bin.

    6. in a week or two you get an incident that some 'smart' people from another department were using this group for something else. and they need its membership back.

     

    what would you do? restore a group from the recycle bin? enable it back on the FIM portal so the new group with another objectSID will be created? do you have a MV extension rule to delete all unnecessary objects in this particular OU? do you have 'do not recall object attributes on deprovision' checkbox enabled? I can continue to count these questions for couple more hours :D

     

    7. finally, you're restoring that object from a recycle bin and enabling it back in FIM, join rule does its job and everyone is happy. for at least an hour :)

    Friday, December 17, 2010 11:50 AM
  • never thought about such things.

    I'll have 10 days off in January, so may be...

    10 days would do the trick :o)

    Well, let me know if you need help with this.
    This could become something like "Methods to automate the clean-up of orphaned EREs".

    So, your implementation would be covered next to others including the discussion we had on this thread about pros and cons.

    Cheers,
    Markus


    Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
    Friday, December 17, 2010 5:58 PM
  • Tuesday, December 28, 2010 2:37 PM
  • Wonderful, Evgeniy, thanks a lot!

    Could you please do one more thing: How do I post a Wiki article announcement

    I want to make sure that the community can find your article.
    Let me know if you have questions about the announcement article.

    Cheers + Thanks,
    Markus


    Markus Vilcinskas, Knowledge Engineer, Microsoft Corporation
    Tuesday, December 28, 2010 2:47 PM
  • Hi. I am aware that some time has passed since this post was written but I hope you can

    help me anyway.

    I state that I read the entire article and "How to avoid..." article. They are very

    interesting. But I am not able to implement some points.

    This is my situation: I have EREs for deleted users and deleted synchronization rules

    (both deleted directly from the portal). So I want to clean these ERE's (following the

    instructions written above) and try to avoid their creations (following "How to avoid..." article suggestions).

    For the first objective, I'm not able to create a custom activity to delete ERE.. How could I create it? Could you specify step by step this pont and, in general, the entire solution more deeply?

    For the second objective, I will study in deep you solution subsequently.

    Thank you for your kidness.

    Thursday, September 1, 2011 1:54 PM
  • This topic takes a new tack with the introduction of the latest FIM hotfix, and the introduction of a built-in stored procedure to deal with these.  Hopefully someone who has first-hand experience with this new stored procedure can report back to this thread as to the success or otherwise of the new approach ... if successful it would appear to make the above options redundant (if run by say the SQL agent on a regular basis).
    Bob Bradley (FIMBob @ http://thefimteam.com/) ... now using Event Broker 3.0 @ http://www.fimeventbroker.com/ for just-in-time delivery of FIM 2010 policy via the sync engine
    Monday, October 24, 2011 11:01 AM
  • Looks like this solution is back in play, Bob.

    As I've just found out, [debug].[DeleteOrphanedRulesByType] doesn't appear in FIM 2010 R2 SP1. Having a look around, it looks like it may have vanished as early as R2.

    Really wishing I'd cleared out my development environment before I upgraded, now!

    For the record, the DeleteOrphanedRulesByType stored proc worked wonderfully up to that point.

    I've created another Technet post to see if anyone knows what happened to it.

    - Ross Currie


    FIMSpecialist.com | MCTS: FIM 2010 | Now Offering ECMA1->ECMA2 Upgrade Services

    Wednesday, March 20, 2013 2:27 PM
  • I might be mistaken, but doens't FIM 2010 R2 handles this for you? So perhaps the stored procedure doesn't exists, but FIM does handle this internally in some way.

    http://setspn.blogspot.com

    Wednesday, March 20, 2013 2:36 PM
  • Well, I thought that might be the case - and would be a logical explanation for why the function was missing, but I definitely had a heap of orphaned EREs sitting in the FIMService database,

    If you look at this post, you'll see Markus recommend using that function for FIM R2 RC. I can't find any documentation saying it was removed.

    - Ross Currie


    FIMSpecialist.com | MCTS: FIM 2010 | Now Offering ECMA1->ECMA2 Upgrade Services

    Wednesday, March 20, 2013 11:14 PM
  • Just following up on this post - so does FIM 2010 R2 deal with orphaned EREs and DREs?

    If not, what is the Microsoft recommended approach?

    Thank you.

    Thursday, July 3, 2014 12:10 AM
  • Yes Shim, I am interested in this as well. Is there an answer to this. I returned to look at an environment recently and there were loads of Orphaned Entries. This is FIM 2010 R2 and its been in action happily for a few months now.

    Would like to hear what Microsoft and Others have to think about this ?

    Rob


    Rob

    Thursday, August 28, 2014 2:17 PM
  • I too am interested having just looked at a customer's system that has accumulated thousands of the damn things since implementation.

    SV

    Friday, November 21, 2014 11:39 AM