none
Project server: Suddenly all projects get stuck in queue RRS feed

  • Question

  • Project server has been working for years here. Yesterday at 11:08 AM a user tried to create a new project and it got stuck in the queue. Now everyone who tries to creat a project gets an error message and they get more jobs stuck in the queue. I have cleaned out the queue, cancelled all of the jobs, but it keeps happening.

    Every user that tries to create a new project receives a similar message. No one create projects, the product is essentially dead completely for us.

    The users receive the error message "Your new Enterprise Project has been created successfully, but failed to publish and will not be listed in the Project Center.", or a similar mesage. The message vary slightly depending on which project template they choose.

    I checked the queue and it was backed up with one Failed But Not Blocking Correlation followed by 13 job groups, one for each project that users had tried to create since the first instance (52 jobs in the queue total). I cancelled all jobs (except for the Failed But Not Blocking Correlation job, which can't be cancelled because the Cancel button is greyed out).

    I have the ULS logs from the machine and I can see them with ULS Viewer but I don't know how to interpret what it is telling me,. Each of these incidents has over 100 log entries associated with the Correlation ID so I do not know which ones to post.

    Can anyone help, or point me in the right direction?

    Here is the detailed message from the first failed job, this is the Failed But Not Blocking Correlation job:

    General Project Site creation it is already in use by another project:WSSSpWebAlreadyLinkedToProject (16406).

    Details: id='16406'

    name='WSSSpWebAlreadyLinkedToProject'

    uid='b03bf689-d333-e611-8106-00505685719b'

    projectUid='002bc489-d333-e611-8106-00505685719b'

    wssFullUrl='http://aaofsharv:22222/PWA/Sharepoint 2013 Migration'.

    Queue:?GeneralQueueJobFailed (26000) - CreateWssSite.CreateWssSiteMessage.

    Details: id='26000'

    name='GeneralQueueJobFailed'

    uid='b13bf689-d333-e611-8106-00505685719b'

    JobUID='3c2bc489-d333-e611-8106-00505685719b'

    ComputerName='248e6198-cb33-407c-a081-482f2878ce50'

    GroupType='CreateWssSite' MessageType='CreateWssSiteMessage'

    MessageId='1' Stage=''

    CorrelationUID='a4be869d-6912-6078-8ce5-06730f817d82'.

    For more details, check the ULS logs on machine 248e6198-cb33-407c-a081-482f2878ce50 for entries with JobUID 3c2bc489-d333-e611-8106-00505685719b.

    Here is the detailed message from the first Failed and Blocking Correlation job in the queue (the second overall):

     General Project: ProjectNameAlreadyExists (1034).

    Details: id='1034' name='ProjectNameAlreadyExists'

    uid='38bb83f6-9734-e611-8106-00505685719b'

    projName=''.

    GeneralQueueException:?GeneralQueueException (9131).

    Details: id='9131' name='GeneralQueueException'

    uid='3abb83f6-9734-e611-8106-00505685719b'

    messageID='1' exception='System.Data.SqlClient.SqlException (0x80131904): Violation of UNIQUE KEY constraint 'UQ_MSP_PROJECTS_SHADOW'. Cannot insert duplicate key in object 'draft.MSP_PROJECTS_SHADOW'. The duplicate key value is (57cd421d-d433-e611-8106-00505685719b, 0). The statement has been terminated. at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.Office.Project.Server.Data.SqlSession.ExecuteNonQuery(SqlCommand command) at Microsoft.Office.Project.Server.DataAccessLayer.SqlSessionAdapter.ExecuteStoredProcedureNonQuery(String schemaQualifiedStoredProcedureName, SqlParameter[] parameters) at Microsoft.Office.Project.Server.DataAccessLayer.DAL.SubDal.<>c__DisplayClass18.<ExecuteStoredProcedureNoResult>b__16(IProjectServerSqlSession session) at Microsoft.Office.Project.Server.Security.SecurityContext.RunAsProcess(CodeToRunElevated secureCode) at Microsoft.Office.Project.Server.DataAccessLayer.DAL.RetryableExecuteHandlerWithSession(IProjectServerSqlSession session, Action`1 actionHandler, Boolean allowRetry) at Microsoft.Office.Project.Server.DataAccessLayer.DAL.RunInActiveSession(DataStoreEnum store, Action`1 actionHandler) at Microsoft.Office.Project.Server.DataAccessLayer.DAL.SubDal.ExecuteStoredProcedureNoResult(String storedProcedureName, SqlParameter[] parameters) at Microsoft.Office.Project.Server.DataAccessLayer.ProjectDal.DeleteProject(Guid projectUid) at Microsoft.Office.Project.Server.BusinessLayer.ProjectMain.QuickDeleteProject(IPlatformContext context, Guid projectUid) at Microsoft.Office.Project.Server.BusinessLayer.ProjectQueueDispatch.HandleProjectAddToMessage(MessageContext mContext, ProjectMain project, ServerEventManager sem, ProjectAddToMessage projectAddToMessage) at Microsoft.Office.Project.Server.BusinessLayer.ProjectQueueDispatch.InternalDispatchMessage(IPlatformContext context, Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext) ClientConnectionId:69c03dd0-13f1-4b35-8426-e0356f17e0f9 Error Number:2627,State:1,Class:14'.

    •Queue:?GeneralQueueJobFailed (26000) - ProjectCreate.ProjectAddToMessage.

    Details: id='26000' name='GeneralQueueJobFailed'

    uid='3bbb83f6-9734-e611-8106-00505685719b' JobUID='5bcd421d-d433-e611-8106-00505685719b' ComputerName='248e6198-cb33-407c-a081-482f2878ce50'

    GroupType='ProjectCreate' MessageType='ProjectAddToMessage'

    MessageId='1' Stage=''

    CorrelationUID='e0be869d-597c-6078-8ce5-054517014ee4'.

    For more details, check the ULS logs on machine 248e6198-cb33-407c-a081-482f2878ce50 for entries with JobUID 5bcd421d-d433-e611-8106-00505685719b.

    Friday, June 17, 2016 4:15 PM

Answers

  • Just to follow up on this in case anyone else has a similar issue: I found and fixed three issues, after which this problem was no longer happening. I am not 100% sure which was the true culprit, or if it was a combination. The three things I fixed are:

    - There was a project with a blank project name. I deleted that project. I couldn't delete it the normal way, I had to do it via Central Administration.

    - A user, a Sharepoint Admin actually, left the company and his ID was deleted from Active Directory but he was still primary administrator for some site collections. This of course isn't good but the confusing part is that when I went to make the change, the address resolution field was filled with garbage data that could not resolve to any address.

    - Several Enterprise Resource Objects had been checked out but Project Web App would not perform a “check in”. There were 3 Enterprise Resources (i.e. humans), 3 Calendar Resources, and 6-8 Project Resources reserved in an exclusive state. To fix, I used Central Admin tool to force objects into “checked in” status

    • Marked as answer by Da J Zunitch Tuesday, July 5, 2016 8:47 PM
    Tuesday, July 5, 2016 8:47 PM

All replies

  • Based on the ULS log, I see multiple issues. One project already exists and another one is share already in use.

    What is the Cu version?



    Hrishi Deshpande, Project MVP

        

    Saturday, June 18, 2016 2:54 AM
    Moderator
  • Just to follow up on this in case anyone else has a similar issue: I found and fixed three issues, after which this problem was no longer happening. I am not 100% sure which was the true culprit, or if it was a combination. The three things I fixed are:

    - There was a project with a blank project name. I deleted that project. I couldn't delete it the normal way, I had to do it via Central Administration.

    - A user, a Sharepoint Admin actually, left the company and his ID was deleted from Active Directory but he was still primary administrator for some site collections. This of course isn't good but the confusing part is that when I went to make the change, the address resolution field was filled with garbage data that could not resolve to any address.

    - Several Enterprise Resource Objects had been checked out but Project Web App would not perform a “check in”. There were 3 Enterprise Resources (i.e. humans), 3 Calendar Resources, and 6-8 Project Resources reserved in an exclusive state. To fix, I used Central Admin tool to force objects into “checked in” status

    • Marked as answer by Da J Zunitch Tuesday, July 5, 2016 8:47 PM
    Tuesday, July 5, 2016 8:47 PM