none
Failed to process CI Assignment changes - SQL ERROR, change will be retried on next processing cycle

    Question

  • Good Morning!

    Current Configuration:

    1. 1 Central, Primary Site; 1 Child Primary Site; 4 Child Secondary Sites (children of the Child Primary Site)

    2. The Primary Site servers have the SQL server databases roles as well, so there are no connectivity issues between SQL and SCCM per site.

    3. The Central Site (CEN), was recently upgraded from Server 2003 to Server 2008 R2 and the SCCM environment was migrated using a side-by-side migration.

    4. We ran into several issues with the server migration, all of which have been remedied, EXCEPT for this particular issue.

    5.  I am not sure if this issue is a by-product of a larger replication issue, but I wouldn't put it past it.

    Here is what we have done to troubleshoot the issue:

    1. Ran a site reset (reset permissions to ensure the accounts and servers have the adequate permissions to the SQL server database.)

    2. Examined the CI_CIAssignments table on the SQL database on CEN (where the error is occuring) to ensure all CI Assignments exist. 

    3.  For purely informative purposes:  The CI_CIAssignments table is directly-related to the Software Updates --> Deployment Management SCCM node. The Deployment ID attribute are equvialent to the rows from the Assignment_ID column in the CI_CIAssignments table within SQL.  I have verified between the two Primary Site databases that the entries are in sync; HOWEVER,  there are some stale records that do not exist on the Primary Child Site.  My assumption is that this would be the case if, say, there were only one Primary Site server and Deployment Management objects were created/removed and then later a second Primary Site were brought online.  The objects that were created/removed prior to the second primary site server being brought online would never replicate from the parent primary site server as they are no longer relevant or in use.

    Long story short, we are receiving the following error CIAMGR.LOG:

    Wait timed out after 11 minutes while waiting for at least one trigger event.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:54 AM	8312 (0x2078)
    ProcessReplicatedFiles.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    ProcessReplicatedFilesInRetry.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    ProcessChangedObjects.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    Failed to process CI Assignment changes, Status = SQL ERROR, changes will be retried on next processing cycle.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    STATMSG: ID=5804 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_CI_ASSIGNMENT_MANAGER" SYS=<CENTRAL SITE SERVER NAME> SITE=<CENTRAL SITE CODE> PID=476 TID=8312 GMTDATE=Mon Sep 16 14:00:59.320 2013 ISTR0="" ISTR1="" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    ProcessObjectsInRetry.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    ProcessAffectedObjects.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    HandleSiteAttach.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    Process CI Assignment Delete notification files.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    WaitForEvent.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)
    Waiting for changes for 29 minutes	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 10:00:59 AM	8312 (0x2078)

    This occurs when there are *.CIA objects in the ciamgr.box inbox.  All other objects, such as *.CIN files, process normally.  For example, the below snippet is from the CIAMGR.LOG as well after a single attribute (such as custom severity) is altered on a Deployment Management object. 

    Processing CI Change Notification file %SCCM INSTALL%\inboxes\ciamgr.box\98624.CIN.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 9:25:16 AM	8312 (0x2078)
    ************Processing CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F}************	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 9:25:16 AM	8312 (0x2078)
    Successfully updated Policy for CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F}	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 9:25:16 AM	8312 (0x2078)
    Successfully updated Policy Targeting for CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F}	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 9:25:16 AM	8312 (0x2078)
    Successfully updated CRCs for CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F}	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 9:25:16 AM	8312 (0x2078)
    STATMSG: ID=5801 SEV=I LEV=M SOURCE="SMS Server" COMP="SMS_CI_ASSIGNMENT_MANAGER" SYS=<CENTRAL SITE SERVER> SITE=<CEN> PID=476 TID=8312 GMTDATE=Mon Sep 16 13:25:16.699 2013 ISTR0="MS10 - Servers" ISTR1="" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=1 AID0=414 AVAL0="{B6BCD488-56E4-4E7D-A438-15E7C00FE81F}"	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 9:25:16 AM	8312 (0x2078)
    ************Successfully processed CI Assignment {B6BCD488-56E4-4E7D-A438-15E7C00FE81F}************	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 9:25:16 AM	8312 (0x2078)
    Successfully processed CI change file %SCCM INSTALL%\inboxes\ciamgr.box\98624.CIN. Deleting it.	SMS_CI_ASSIGNMENT_MANAGER	9/16/2013 9:25:16 AM	8312 (0x2078)

    Once those *.CIN files are processed and the CIAMGR.LOG goes back to standard processing changes every 30 minutes, we receive the error posted above (Failed to process CI Assignement Changes - SQL ERROR).  This occurs regardless of the existence of *.CIA manager files in the CIAMGR.BOX inbox.  You can force changes by dropping files in the box but the same result occurs (SQL ERROR).

    Does anyone have any insight into what the generic "SQL ERROR" error can indicate.  My assumption is this is a SQL backend issue, but I don't know what to look for as far as corrupt rows or columns are concerned. 

    Ultimately this appears to be impacting the Software Update synchronizations between the two Primary Site servers.  When a synchronization is run on the Primary Site server, the child site server Deployment Management objects are not being sent the updates for the various objects.  If we manually change something (such as the Custom Severity attribute) on an update that is part of the Deployment Management object, this can force a *.CIN change, but not the *.CIA object change. 

    Any insight or assistance is greatly appreciated!

    Thank you!



    • Edited by pribnuts Thursday, September 19, 2013 3:45 PM
    Monday, September 16, 2013 2:09 PM

All replies

  • Hi You can refer to my post as below on my blog http://blog.theinfraguys.com/2013/06/object-replication-manager-objreplmgr.html See if it helps

    Http://blog.theinfraguys.com | Please remember to click “Mark as Answer” on my answer if it helps. :)

    Tuesday, September 17, 2013 11:58 AM
  • Hi Vincez,

    I would like to say we had something similar done a while back but it doesn't hurt to try again.  Unfortunately, due to the HA nature of the environment, we won't be able to run a full sync at least until the weekend.  I will work with my team in regards to arranging such a down time and get back to you with my results on Monday.  Thank you for the heads up!

    Wednesday, September 18, 2013 2:54 PM
  • Lastly,

    These errors are occuring on the CENTRAL SITE (CEN).  I know your blog post below indicates a problem with replication to a child site, so I wanted to see if wiping all CI items from both the CEN and PRI sites would guarantee a clean slate.

    The direct relationship I see is this:

    The %SCCM INSTALL%\inboxes\ciamgr.box on CEN contains a backlog of *.CIA objects.  These have not been processed at all and, from my understanding, when the CI_ASSIGNMENT_MANAGER component runs (every 30 minutes by default or a change is detected) it is supposed to process these objects.  What happens at run time is that the SQL ERROR is thrown (see original post above for specific error). 

    For example, in the Configuration Manager Console on the CENTRAL SITE server (CEN), below Computer Management --> Software Updates --> Deployment Management, there are several Deployment Management objects, each with a corresponding Deployment ID.  Let's say one called MS - 10 Servers, with a Deployment ID of 200.  In the CIAMGR.BOX, there is an object that cannot be processed called 200.CIA. 

    If I modify the actual Deployment Management object, for example, changing the severity of the object from Low to Important, a *.CIN object is created and processed fine, which, I assume directly affects the *.CIA object.  There is a successful processing message (see second log snippet, above) that the *.CIN change was processed successfully.  The 200.CIA object is created (which I assume is the broader object item), but it is not processed, generates and backlog and the resultant error (see first log snippet, above) occurs.

    I don't know if that better explains the process, but that is what we see on a step by step basis.

    Thursday, September 19, 2013 3:43 PM
  • Just wanted to provide an update to this.  We have escalated this issue to Microsoft PSS as no viable solution has been raised.  We were initially put in touch with a SQL Specialist but he is not fluent in the configuration of SCCM so the issue has been pushed over to the SCCM specialists.  I have not received feedback at this time. 

    Wednesday, October 02, 2013 1:29 PM