none
Queue errors with "An item with the same key has already been added" RRS feed

  • Question

  • Hi

    in a Project Server 2013 environment suddenly the queue stops working. Jobs are "processing" but after some minutes they fail with the following message:

    • An item with the same key has already been added.:
      • ProjectCheckinFailure (23008) - System.ArgumentException: An item with the same key has already been added. at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer) at Microsoft.Office.Project.Server.Events.ServerEventManager.ConfiguredServerEvents.EventHandlerConfiguration.GetInstance(ProjectSite projectSite) at Microsoft.Office.Project.Server.Events.ServerEventManager.ConfiguredServerEvents..ctor(ProjectSite projectSite) at Microsoft.Office.Project.Server.Events.ServerEventManager.ConfiguredServerEvents.<>c__DisplayClass8.<GetInstance>b__7() at Microsoft.Office.Project.Server.Utility.LazySet`2.GetValue(TKey key, Func`1 instanceFactory) at Microsoft.Office.Project.Server.Events.ServerEventManager.ConfiguredServerEvents.GetInstance(Guid siteUid) at Microsoft.Office.Project.Server.Events.ServerEventManager.<GetServerEventManager>b__0(IPlatformContext context) at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory) at Microsoft.Office.Project.Server.BusinessLayer.Queue.ProcessProjectCheckinMessage.ProcessProjectCheckinFireCheckinPostEventMessage(Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext) at Microsoft.Office.Project.Server.BusinessLayer.Queue.ProcessProjectCheckinMessage.HandleMessage(Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext). Details: id='23008' name='ProjectCheckinFailure' uid='2e4a35e1-0602-e411-93f3-00155d322e13' ProjectUID='68336cb2-5bf1-e311-93f1-00155d322e13' JobUID='16fa8edc-0602-e411-93f3-00155d322e13' MessageID='8' Error='System.ArgumentException: An item with the same key has already been added. at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer) at Microsoft.Office.Project.Server.Events.ServerEventManager.ConfiguredServerEvents.EventHandlerConfiguration.GetInstance(ProjectSite projectSite) at Microsoft.Office.Project.Server.Events.ServerEventManager.ConfiguredServerEvents..ctor(ProjectSite projectSite) at Microsoft.Office.Project.Server.Events.ServerEventManager.ConfiguredServerEvents.<>c__DisplayClass8.<GetInstance>b__7() at Microsoft.Office.Project.Server.Utility.LazySet`2.GetValue(TKey key, Func`1 instanceFactory) at Microsoft.Office.Project.Server.Events.ServerEventManager.ConfiguredServerEvents.GetInstance(Guid siteUid) at Microsoft.Office.Project.Server.Events.ServerEventManager.<GetServerEventManager>b__0(IPlatformContext context) at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory) at Microsoft.Office.Project.Server.BusinessLayer.Queue.ProcessProjectCheckinMessage.ProcessProjectCheckinFireCheckinPostEventMessage(Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext) at Microsoft.Office.Project.Server.BusinessLayer.Queue.ProcessProjectCheckinMessage.HandleMessage(Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext)'.

    Queue:

    • GeneralQueueJobFailed (26000) - ProjectCheckIn.ProjectCheckinFireCheckinPostEventMessage. Details: id='26000' name='GeneralQueueJobFailed' uid='324a35e1-0602-e411-93f3-00155d322e13' JobUID='16fa8edc-0602-e411-93f3-00155d322e13' ComputerName='8d0fce6c-a4c4-4dc4-a68c-160993778a98' GroupType='ProjectCheckIn' MessageType='ProjectCheckinFireCheckinPostEventMessage' MessageId='8' Stage='' CorrelationUID='b2a1a09c-85d7-60ca-78a9-811868fc5361'. Überprüfen Sie die ULS-Protokolle auf dem Computer 8d0fce6c-a4c4-4dc4-a68c-160993778a98 auf Einträge mit JobUID 16fa8edc-0602-e411-93f3-00155d322e13, um weitere Details anzuzeigen..

    I tried without success:

    • stop / restart queue and event service
    • edit  / reprovision pwa site
    • psconfig
    • reboot

    Any more ideas?

    Thanks for help

    Christoph


    Christoph Muelder | Senior Consultant, MCTS, MCSE, MCT | SOLVIN information management GmbH, Germany

    Wednesday, July 2, 2014 4:47 PM

Answers

All replies

  • Do you, too, have more than one event handler registered for the same event?

    That is - if you run this query in the projectserver database:

    SELECT * FROM [ProjectWebApp].[pub].[MSP_EVENT_HANDLERS]

    do you have more than one row where the field EVENT_ID contains the same value?

    If so, it could be related to

    http://social.technet.microsoft.com/Forums/projectserver/en-US/83921bcd-5f51-46dc-89d2-31bd22b96c5a/multiple-event-handlers-registered-for-the-same-event-and-error-when-adding-users-to-sec-groups?forum=project2010custprog

    /Lars Hammarberg


    //Lars Hammarberg www.connecta.se

    Friday, July 4, 2014 10:47 AM
  • Hi Lars,

    what a nice coincidence, that you just had the same error :-)

    Thanks a lot. You are my hero of the week.

    Yes, I had two events on Project created - a custom development plus a handler associated to Nintex Workflow. Means for us that we may not use the same event for Nintex as we use for our own environment.

    Tack se mycket !

    Christoph


    Christoph Muelder | Senior Consultant, MCTS, MCSE, MCT | SOLVIN information management GmbH, Germany

    Friday, July 4, 2014 11:57 AM
  • Hi

    Update to the issue:
    The issue is reproduced by Microsoft Support and a hotfix was requested. Should take 15days to get a decision from the product group.

    Regards

    Christoph


    Christoph Muelder | Senior Consultant, MCTS, MCSE, MCT | SOLVIN information management GmbH, Germany

    Tuesday, August 12, 2014 9:50 AM
  • Hi Christoph,

    Any updates on this case?

    Thank you,
    Ivan


    Softlanding.ca
    AutoSPInstallerGUI

    Wednesday, September 24, 2014 6:53 PM
  • Hi

    yes. Update request accepted.

    Shall make it into the November 2014 Cumulative Update.

    Kind regards

    Christoph


    Christoph Muelder | Senior Consultant, MCTS, MCSE | SOLVIN information management GmbH, Germany

    Wednesday, September 24, 2014 7:19 PM
  • Hi guys,

    Good news is that this issue is fixed in the November CU.

    However!!!, If you tested the private Hotfix for this issue, your farm is left in a broken state if you apply the November CU.

    The Projects Queue Service throws the following error:

    [bucketHash:7DC81C51] SqlException occurred in DAL (PS2013_STG_CGY_Content_Projects_PWA@SPSQL): Class 16, state 1, line 41, number 308, procedure MSP_ProjQ_Lock_Next_Available_Group, error code -2146232060, exception: System.Data.SqlClient.SqlException (0x80131904): Index 'IX_MSP_QUEUE_PROJECT_FILTER_FILTER_MSG_ACTIVE' on table 'draft.MSP_QUEUE_PROJECT_FILTER' (specified in the FROM clause) does not exist.   

     at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)   

     at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)   

     at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)   

     at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()   

     at System.Data.SqlClient.SqlDataReader.get_MetaData() 

    .... Omitted the rest....

    My investigation:

    1. We Applied the November CU on the same farms which had the Hotfix applied
    2. We see the following error from the Queue Service and it does not process any items in the Queue
      1. 'IX_MSP_QUEUE_PROJECT_FILTER_FILTER_MSG_ACTIVE' Index does not exist on table 'draft.MSP_QUEUE_PROJECT_FILTER'
    3. The missing Index was supposed added by Schema Update 15.1.601.0 which is visible on the November CU
    4. The Hotfix which was provided to us had a Schema Update for 15.1.601.0. However, the Schema update here had no reference to the 'IX_MSP_QUEUE_PROJECT_FILTER_FILTER_MSG_ACTIVE' Index
    5. Now after applying the November CU, the scheme will not be fixed as it is already at the “correct” version number 15.1.601.0

    At this point, the only option I see to fix this is to take the Project Server Databases from Production and move them to our Test Environments. This should correctly update the database.

    However, my concern is that the Hotfix may have broken other components in the farm.


    Softlanding.ca
    AutoSPInstallerGUI

    Thursday, November 13, 2014 8:34 PM
  • Hi Ivan,

    that reads not very good...

    But I understand you correct that you aleady had a private hotfix for that specific issue before you now updated to November CU? Until now my environment (just a testing one) does not show any problems.

    Kind regards

    Christoph


    Christoph Muelder | Senior Consultant, MCTS, MCSE | SOLVIN information management GmbH, Germany

    Friday, November 14, 2014 6:45 AM
  • Hi Chris,

    In my case the issue was caused by the Private Hotfix. We only applied it to our pre-production environments, and we were told that it shouldn't break anything once the final CU is out.

    Luckily, the Schema issue described above only affected the PWA databases so a sync from production resolved the issue. I posted this in case other people tested the hotfix so they don't have to rebuild their environments, sorry about the confusion.

    Cheers,
    Ivan


    Softlanding.ca
    AutoSPInstallerGUI

    Friday, November 14, 2014 9:57 PM