none
Project server queue is failing and Publishing time has been increased

    السؤال

  •  

    Hi,

    I am running Project Server using Project server 2007 (SP3) (64 bit) with SQL Server 2005 on Windows Server 2003.

    During last one week, many of the jobs have started failing. The job usually process successfully but sometimes fails randomly with state ‘Failed But Not Blocking Correlation’.

    Whenever the job fails, it exactly fails at following percentage:

    OLAP Cube Build  --> 50%
    Project Checkin  --> 12%
    Project Publish  --> 96%
    Project Publish Notifications  --> 0%
    WSS Workspace Create  --> 0%

    Secondly the workspace creation has started taking a lot of time to complete whenever we publish the project which is causing a huge problem for the users.

    Can you please inform me that what can be the reason of failing these jobs?

    Thanks!!

    22/ربيع الثاني/1433 04:37 م

الإجابات

جميع الردود

  • hi there,

    Check SQL logs, event logs on SQL server.


    Hrushikesh Deshpande – Senior EPM Infrastructure Consultant, www.DeltaBahn.com Please Mark As Answer if my post solves your problem or Vote As Helpful if a post has been helpful for you

    22/ربيع الثاني/1433 05:46 م
  • Hi Hrishi,

    Do you have any solution to my problem? when the project workspace creation job fails, it takes a lot of time to publish the project from project professional. i need to resolve this issue asap as users are spending 5-10 eherytime they publish that project.

    Could you please help me?

    thanks!

    22/ربيع الثاني/1433 08:00 م
  • What do the logs at 14 say? You need to understand what is causing this error? Is this happening only with a particular project publish or every project publish?
    22/ربيع الثاني/1433 08:09 م
  • The project publish problem is happening with all those projects whose workspace creation job was failed earlier. and WSS Workspace Create and other jobs are failing randomly with state ‘Failed But Not Blocking Correlation’.

    Following is the error detail of one of the failed workspace job:

    Standard Information:PSI Entry Point:   Project User: US1\vpmi_dba  Correlation Id: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy PWA Site URL: http://epmxxxx/PWA  SSP Name: SharedServicesxx  PSError: GeneralQueueJobFailed (26000) A queue job has failed. This is a general error logged by the Project Server Queue everytime a job fails - for effective troubleshooting use this error message with other more specific error messages (if any), the Operations guide (which documents more details about queued jobs) and the trace log (which could provide more detailed context). More information about the failed job follows. GUID of the failed job: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. Name of the computer that processed this job: EPMxxxx (to debug further, you need to look at the trace log from this computer). Failed job type: CreateWssSite. Failed sub-job type: CreateWssSiteMessage. Failed sub-job ID: 1. Stage where sub-job failed:  (this is useful when one sub-job has more than one logical processing stages).

    Error summary/areas:

    Queue

    GeneralQueueJobFailed

    Error details:

    <?xml version="1.0" encoding="utf-16"?>

    <errinfo>

      <general>

        <class name="Queue">

          <error id="26000" name="GeneralQueueJobFailed" uid="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" JobUID=" xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx " ComputerName="EPMxxxx" GroupType="CreateWssSite" MessageType="CreateWssSiteMessage" MessageId="1" Stage="" />

        </class>

      </general>

    </errinfo>

    Please help..


    22/ربيع الثاني/1433 08:55 م
  • Is this the error from 14 logs folder on server or is it from the PWA error description? I'm not able to read much from these. So if this is not from 14 logs, may be there could be some more info there 
    22/ربيع الثاني/1433 09:20 م
  • The above paragraph is copied from the Diagnostics Logs which exists in 12 Log Folder.

    The XML Error detail is copied from PWA error description.


    has

    22/ربيع الثاني/1433 09:46 م
  • You should review log file with servername and date time on it

    Ignore file with title as Diagnostics


    Hrushikesh Deshpande – Senior EPM Infrastructure Consultant, www.DeltaBahn.com Please Mark As Answer if my post solves your problem or Vote As Helpful if a post has been helpful for you

    22/ربيع الثاني/1433 09:58 م
  • Yes Hirshi.


    Following error detail is copied from the log of the job that was failed while creating workspace:

    (The log data is copied from Message column of log.)


    Standard Information:PSI Entry Point:   Project User: US1\vpmi_dba  Correlation Id: yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy PWA Site URL: http://epmxxxx/PWA  SSP Name: SharedServicesxx  PSError: GeneralQueueJobFailed (26000) A queue job has failed. This is a general error logged by the Project Server Queue everytime a job fails - for effective troubleshooting use this error message with other more specific error messages (if any), the Operations guide (which documents more details about queued jobs) and the trace log (which could provide more detailed context). More information about the failed job follows. GUID of the failed job: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. Name of the computer that processed this job: EPMxxxx (to debug further, you need to look at the trace log from this computer). Failed job type: CreateWssSite. Failed sub-job type: CreateWssSiteMessage. Failed sub-job ID: 1. Stage where sub-job failed:  (this is useful when one sub-job has more than one logical processing stages).


    22/ربيع الثاني/1433 10:07 م
  •  

    There could be various reasons for workspace site creation failure like customize template, disk space issue on content database, maximum number of sites setting etc.

    I think this error message is copied from the Application Event log . We are looking for more detailed log.

    Navigate to PWA>>>Server Settings>>Manage Queue

    Go to failed job and from the last column click on the hyper link to view more detailed log

    ULS log is more helpful

    You need to navigate to log folder on Application Server and click on date modified column (so that latest date and time will be displayed at the top)

    if you see the log file with server name and date time open that log . e,g PSVR-20120315-1511.log

     


    Hrushikesh Deshpande – Senior EPM Infrastructure Consultant, www.DeltaBahn.com Please Mark As Answer if my post solves your problem or Vote As Helpful if a post has been helpful for you

    22/ربيع الثاني/1433 10:19 م
  • Hi,

    Thanks for your response. I had copied only the message column of ULS log. now I am copying the whole row.

    As I have informed earlier that it is not only Workspace Creation job that is failing. Other jobs have also started failing. So there is a problem that is effecting other jobs as well.

    Secondly it is not necessary that all the jobs of same project fails together.

    Log from EPMxxxx-20120316-0538.log

    Log of Project Publish Faliure:

    03/16/2012 05:53:40.46  Microsoft.Office.Project.Server (0x15E8) 0x16E8 Project Server                 Project Server Queue           7h60 Critical Standard Information:PSI Entry Point:   Project User: a_dom\PS_Usr  Correlation Id: 5fcf733d-2228-4faa-a04d-f77b7ac40dc5  PWA Site URL: http://EPMxxxx/PWA  SSP Name: SharedServices1  PSError: GeneralQueueJobFailed (26000) A queue job has failed. This is a general error logged by the Project Server Queue everytime a job fails - for effective troubleshooting use this error message with other more specific error messages (if any), the Operations guide (which documents more details about queued jobs) and the trace log (which could provide more detailed context). More information about the failed job follows. GUID of the failed job: cd816999-d1d6-4191-84c4-503d08a70cb9. Name of the computer that processed this job: EPMxxxx (to debug further, you need to look at the trace log from this computer). Fail... 0f7dd607-f8b7-4447-b903-7f061b6e0018
    03/16/2012 05:53:40.46* Microsoft.Office.Project.Server (0x15E8) 0x16E8 Project Server                 Project Server Queue           7h60 Critical ...ed job type: ProjectPublish. Failed sub-job type: UpdateWSSCountersMessage. Failed sub-job ID: 182. Stage where sub-job failed:  (this is useful when one sub-job has more than one logical processing stages). 0f7dd607-f8b7-4447-b903-7f061b6e0018


    Log of Create Workspace:

    03/16/2012 05:57:56.85  Microsoft.Office.Project.Server (0x15E8) 0x16E8 Project Server                 Project Server Queue           7h60 Critical Standard Information:PSI Entry Point:   Project User: a_dom\PS_Usr  Correlation Id: 349d992a-fcbc-47b4-8770-ed2e962fe2a4  PWA Site URL: http://EPMxxxx/PWA  SSP Name: SharedServices1  PSError: GeneralQueueJobFailed (26000) A queue job has failed. This is a general error logged by the Project Server Queue everytime a job fails - for effective troubleshooting use this error message with other more specific error messages (if any), the Operations guide (which documents more details about queued jobs) and the trace log (which could provide more detailed context). More information about the failed job follows. GUID of the failed job: db841235-1bb5-458b-92de-478a52f9cad0. Name of the computer that processed this job: EPMxxxx (to debug further, you need to look at the trace log from this computer). Fail... 0f7dd607-f8b7-4447-b903-7f061b6e0018
    03/16/2012 05:57:56.85* Microsoft.Office.Project.Server (0x15E8) 0x16E8 Project Server                 Project Server Queue           7h60 Critical ...ed job type: CreateWssSite. Failed sub-job type: CreateWssSiteMessage. Failed sub-job ID: 1. Stage where sub-job failed:  (this is useful when one sub-job has more than one logical processing stages). 0f7dd607-f8b7-4447-b903-7f061b6e0018

    Error detail from Navigate to PWA>>>Server Settings>>Manage Queue:

    Error summary/areas:
    Queue
    GeneralQueueJobFailed
    Error details:

    <?xml version="1.0" encoding="utf-16"?>
    <errinfo>
      <general>
        <class name="Queue">
          <error id="26000" name="GeneralQueueJobFailed" uid="ff90fe08-f395-48f1-8627-fb661ca8e4fb" JobUID="db841235-1bb5-458b-92de-478a52f9cad0" ComputerName="EPMxxxx" GroupType="CreateWssSite" MessageType="CreateWssSiteMessage" MessageId="1" Stage="" />
        </class>
      </general>
    </errinfo>

    Error summary/areas:

    ProjectPublishFailure
    Queue
    GeneralQueueJobFailed
    Error details:

    <?xml version="1.0" encoding="utf-16"?>
    <errinfo>
      <general>
        <class name="">
          <error id="23000" name="ProjectPublishFailure" uid="498da489-24b0-4e0a-8a96-e6baacc77c75" projectuid="5ce98cd8-1c08-4016-90c0-447179430456" messagetype="Microsoft.Office.Project.Server.BusinessLayer.QueueMsg.UpdateWSSCountersMessage" messageID="182" stage="" blocking="DontBlock" />
        </class>
        <class name="Queue">
          <error id="26000" name="GeneralQueueJobFailed" uid="379b9490-1314-4930-bad0-0f6ea1f8ad86" JobUID="cd816999-d1d6-4191-84c4-503d08a70cb9" ComputerName="EPMxxxx" GroupType="ProjectPublish" MessageType="UpdateWSSCountersMessage" MessageId="182" Stage="" />
        </class>
      </general>
    </errinfo>

    Thanks!!


    23/ربيع الثاني/1433 04:13 م
  • Thanks for the log.

    Are you using default Project Workspace template or it has been customized?


    Hrushikesh Deshpande – Senior EPM Infrastructure Consultant, www.DeltaBahn.com Please Mark As Answer if my post solves your problem or Vote As Helpful if a post has been helpful for you

    23/ربيع الثاني/1433 04:24 م
  •  

    We were using customized workspace for a long time but today morning we have turned it back to default just to check if it has any problem.

    But let me specify one thing that it is not only Create workspace job that is failing. If project A is failing in creating workspace then Project X may fail in publish etc.

    Whenever the job fails, it exactly fails at following percentage:

    OLAP Cube Build  --> 50%
    Project Checkin  --> 12%
    Project Publish  --> 96%
    Project Publish Notifications  --> 0%
    WSS Workspace Create  --> 0%

    Thanks for your efforts Hrishi!

    23/ربيع الثاني/1433 05:06 م
  • I understand your efforts on getting us above logs, but it’s not helping to narrow down the issue.

    You may need to turn on verbose logging for specific events

    Also did you try with new project?


    Hrushikesh Deshpande – Senior EPM Infrastructure Consultant, www.DeltaBahn.com Please Mark As Answer if my post solves your problem or Vote As Helpful if a post has been helpful for you

    23/ربيع الثاني/1433 05:37 م
  • how should I send you the file?

    23/ربيع الثاني/1433 09:09 م
  • I am not sure.

    Well try to set default template as original and try to reproduce the issue


    Hrushikesh Deshpande – Senior EPM Infrastructure Consultant, www.DeltaBahn.com Please Mark As Answer if my post solves your problem or Vote As Helpful if a post has been helpful for you

    23/ربيع الثاني/1433 09:33 م
  • I have set the Default template now but queue is still failing. I have seen in the lpg that it fails after a deadlock problem. The log is given below.

     


    03/22/2012 03:12:57.21 Microsoft.Office.Project.Server (0x0F90) 0x0AD4 Windows SharePoint Services Database 6f8g Unexpected Unexpected query execution failure, error code 1205. Additional error information from SQL Server is included below. "Transaction (Process ID 89) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction." Query text (if available): "SET NOCOUNT ON; DECLARE @DN nvarchar(256),@LN nvarchar(128),@@DocUIVersion int,@@Level tinyint,@@S uniqueidentifier; DECLARE @ItemId int; DECLARE @@iRet int; DECLARE @ExtraItemSize int; DECLARE @Login nvarchar(255); DECLARE @Title nvarchar(255); DECLARE @Email nvarchar(255); DECLARE @Notes nvarchar(1023); DECLARE @AddedToTable bit; DECLARE @UserIdOut int; DECLARE @LoginOut nvarchar(255); DECLARE @TitleOut nvarchar(255); DECLARE @EmailOut nvarchar(255); DECLARE @NotesOut nvarchar(1023); DECLARE @DeletedOut bit; DECLARE @IsGroup bit; SET @IsGroup = 0; SET @@Level = 1; SELECT @Login = ?; SELECT @Title = ?; SELECT @Email = ?; SELECT @Notes = ?; SELECT @@S=?; BEGIN TRAN; EXEC @@iRet = proc_SecAddUser @@S, ?, ?, ?, ?, ?, @Login, @Title, @Email, @Notes, ?, ?, ?, ?, @AddedToTable OUTPUT, @UserIdOut OUTPUT, @LoginOut OUTPUT, @TitleOut OUTPUT, @EmailOut OUTPUT, @NotesOut OUTPUT, @DeletedOut; IF @@iRet <> 0 BEGIN ROLLBACK TRAN; GOTO DONE; END ; SELECT @ItemId = @UserIdOut IF NOT EXISTS( SELECT tp_ID FROM UserData WHERE tp_ListId = '2D928546-1C84-473D-A975-38597284106E' AND tp_ID = @ItemId  AND tp_Level = 1 AND tp_RowOrdinal =0) BEGIN SELECT @ExtraItemSize = 0  EXEC @@iRet = proc_AddListItem @SiteId = '05DD5CAD-9FEF-412F-91F5-BB1603F6E9B0',@WebId='50C0050F-B13C-4783-882D-16007B599432',@ListID = '2D928546-1C84-473D-A975-38597284106E',@RowOrdinal = 0,@ItemId = @ItemId OUTPUT,@ItemDirName=@DN OUTPUT,@ItemLeafName=@LN OUTPUT,@UserID = 1073741823,@TimeNow = '20120322 08:12:38',@ServerTemplate = 112,@Basetype= 0,@Level= 1,@tp_GUID =NULL,@AddNamespace=1,@CheckDiskQuota=1, @tp_ContentTypeId = ?, @tp_ContentType = ?, @nvarchar1 = ?, @tp_ModerationStatus = ?, @nvarchar3 = ?, @nvarchar4 = ?, @nvarchar5 = ?, @bit2 = ?, @bit3 = ?, @nvarchar8 = ?, @nvarchar9 = ?, @bit4 = ?, @Size = 144, @ExtraItemSize = @ExtraItemSize ,@acl=0xF3FE0000010000000000000000000000; IF @@iRet <> 0 BEGIN ROLLBACK TRAN; GOTO DONE; END  END ; COMMIT TRAN; DONE: SELECT @@iRet, @ItemId, @AddedToTable, @LoginOut, @TitleOut, @EmailOut, @NotesOut, @DeletedOut " 

    03/22/2012 03:12:57.21 Microsoft.Office.Project.Server (0x0F90) 0x0BEC Project Server Project Server Queue 7h60 Critical Standard Information:PSI Entry Point:   Project User: DOM1\user1 Correlation Id: dff571a0-917e-4ae5-abd0-8804827e83f8  PWA Site URL: http://epmxx/PWA  SSP Name: SharedServices1  PSError: GeneralQueueJobFailed (26000) A queue job has failed. This is a general error logged by the Project Server Queue everytime a job fails - for effective troubleshooting use this error message with other more specific error messages (if any), the Operations guide (which documents more details about queued jobs) and the trace log (which could provide more detailed context). More information about the failed job follows. GUID of the failed job: a77a8c6c-6e40-4c05-a59f-4fab328a4cec. Name of the computer that processed this job: EPMXX (to debug further, you need to look at the trace log from this computer). Failed job type: SynchronizeSingleUserMembershipInWss. Failed sub-job type: SynchronizeSingleUserMembershipInWssMessage2. Failed sub-job ID: 2. Stage where sub-job failed:  (this is useful when one sub-job has more than one logi 79911f41-4ca3-41e8-af14-2eaf7f2036f8


    Now can you please inform me that what would be the reason of this deadlock and how can I avoid it?
    According to the script which is creating deadlock, I think it is not written by the developer and is a part of Project Server.

    Thanks!




    29/ربيع الثاني/1433 06:48 م
  • Restarting SQL server will help you to address deadlock issue

    Here is the more information about SQL deadlock and Project Server

    http://blogs.msdn.com/b/brismith/archive/2007/04/17/sql-deadlocks-and-the-project-server-2007-queues.aspx


    Hrushikesh Deshpande – Senior EPM Infrastructure Consultant, www.DeltaBahn.com Please Mark As Answer if my post solves your problem or Vote As Helpful if a post has been helpful for you

    • تم وضع علامة كإجابة بواسطة DBA MS Project Server 26/جمادى الأولى/1433 04:33 م
    29/ربيع الثاني/1433 06:55 م