Sign in
 
Home2012Previous VersionsLibraryForumsGallery
System Center TechCenter >
> 
> 
Config and AD connectors causing slow GUI response and multiple emails in SCSM 2010 SP1 CU3

Unanswered Config and AD connectors causing slow GUI response and multiple emails in SCSM 2010 SP1 CU3

  • Thursday, February 14, 2013 9:59 PM
     
     
    Sign In to Vote
    0

    We have a persistent, intermittent problem with connectors in our large Service Manager 2010 environment.  I have recently taken it over, and I understand the problem has been occurring since installation.  Here are the symptoms:

        The config connector, scheduled to run at 2AM, hangs somewhere around 25-30%.  The connector pane shows "Finished Success" with no ending time and a small percentage.

        When incident tickets begin to come in, response time is very slow in the GUI.  A restart of services will improve performance.

        Sometimes, the Exchange connector will hang at the same time and prevent email messages from being sent.  Sometimes, multiple emails will be sent.

        Re-syncing the config connector will break the log jam and the connector will eventually reach 100%.  However, it may take 2 or more hours to finish, and so it must be done very early in the morning in order to prevent excruciating GUI slowness when Help Desk folks get to work.

        There are multiple event log 33610 errors associated with the AD and config connector sync times.  When I say "multiple", last night it was 6,695 of them.  These indicate problems with query length ("The database subscription query is longer than expected.  Check the database or simplify the database subscription criteria."  Associated with these are info events referring to the config and AD connectors.

    To try and fix this problem, we applied CU3.  I found a reference to a bug in SQL Server 2008 R2 having to do with splitting queries over multiple cores, and I limited "max degree of parallelism" to 2 on the DB server.  We have split our AD connector into 12 separate connectors and spaced out the schedule of these from 11PM to 2AM, once a day.  Our config connector kicks off at 2AM and usually runs about 3 hours.  This problem happens an average of 2-3 times a week, and it is very disruptive to the Help Desk.  I have a development environment also, and it exhibits the same behavior. 

    We will be moving to SCSM 2012 in the next few months, but it would be a big help (to my career, anyway) to stabilize this installation.  Any ideas?

     

All Replies

  • Thursday, February 14, 2013 10:55 PM
     
     
    Sign In to Vote
    0

    Hello,

    The problem you are describing is a Workflow issue.  What is happening is since the connectors are taking so long to complete (probably due to the amount of data), the watermarks on the workflows are getting behind. 

    http://blogs.technet.com/b/servicemanager/archive/2013/01/14/troubleshooting-workflow-performance-and-delays.aspx

    What we have done in the past, in very large, global environments is create light custom AD/SCCM connectors that only brings over the necessary data.  These connectors run and compares the changes made from the previous day and only imports the deltas.  These connectors run much faster (CSV imports via Orchestrator).  Then, we run the full out-of-box connectors over the weekend when there is a bigger maintenance window.  That allows all of the data to come over.

    With all of that said, you will also want to make sure the server hosting the operational CMDB has a good amount of resources depending on your company's size.

    - Christopher

    Until the Whole World Hears


     
  • Friday, February 15, 2013 2:59 PM
     
     
    Sign In to Vote
    0

    It turns out I have seen that blog post, but since it specifically didn't talk about connectors, I didn't follow it through closely.  The "light custom AD/SCCM connectors" you reference I assume are implemented by modifying the management packs involved, right?  Are there examples available of the modifications?  I have a DEV environment in place where I can test, in case I have to come up with the modifications myself, but some direction would be appreciated.

    Thanks again.

     
  • Friday, February 15, 2013 7:02 PM
     
     
    Sign In to Vote
    0

    How long do the out-of-box connectors take to run from start to finish?  What you could do is schedule them at different times to account for the heavy load on the DB.  As Travis suggests in his post, start by looking at resources on the servers and then troubleshoot the individual workflows that my be behind.  You may have to modify the Batch Size or one of the other settings he suggests.

    - Christopher

    Until the Whole World Hears

     
  • Friday, February 15, 2013 8:17 PM
     
     
    Sign In to Vote
    0

    The out-of-box config connector takes about 2 1/2 to 3 hours, when it finishes.  We have AD connectors chopped up into 14 regions, each getting a separate OU, and they take between 2 minutes and 2 hours or more, depending on the region.  It can vary a lot within a given region.  One region has taken anywhere from 42 min. to over 3 hours.  The schedules on these run from 11PM - 2AM and are overlapped so that 2 begin every half hour.  The config connector begins at 2AM.  So, I think we've about exhausted the scenarios for splitting and scheduling connectors.  We need to do what you suggest and customize them to retrieve only deltas and maybe only certain classes of  items.

    This problem has been going on for quite a long time, and we've tried a whole lot of things.  For months, we only had the Exchange connector enabled, since the others caused so many problems.  I have tried all kinds of scheduling, performed cumulative updates, and investigated possible SQL Server bugs.  I'm intrigued by your suggestion of customized connectors, if I can figure out the correct method to implement them.  I have downloaded and looked at the Authoring Tool and have read a bit about importing Management Packs.  As of now, I haven't located the System Center Configuration Manager Connector Configuration MP that apparently defines the config connector.  I have an extensive programming background, including lots of markup language work, so I feel confident in my ability to do the work, if I can get enough info.

    Any information you have on building a custom MP for the config connector would be appreciated.

     
  • Saturday, February 16, 2013 5:46 AM
     
     
    Sign In to Vote
    0

    You wouldn't want to customize the out of box MPs that hold the configuration of the connectors.  My suggestion at this point would be for you to take a look at Travis's blog post here that shows you how to create an AD connector using PowerShell and CSV import.  This is a great practical way to help you with your scheduling issues.  Let me know if this helps.

    http://blogs.technet.com/b/servicemanager/archive/2009/12/21/creating-an-ad-connector-using-powershell-and-csv-import.aspx

    Great post on how the CSV import works:

    http://blogs.technet.com/b/servicemanager/archive/2009/05/26/using-the-csv-import-feature.aspx

    - Christopher

    Until the Whole World Hears