none
Project Server 2007 - Reporting (Resource Sync) Failed and Blocking Correlation RRS feed

  • Question

  • Dear All

    I happen to see a lot of jobs in the queue with the JobType "Reporting (Resource Sync)" and Job State "Failed and Blocking Correlation". Error details are as follows:

    Error summary/areas:
    Reporting message processor failed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    ReportingResourceChangeMessageFailed
    Queue
    GeneralQueueJobFailed
    Error details:
    
    <?xml version="1.0" encoding="utf-16"?>
    <errinfo>
      <general>
        <class name="Reporting message processor failed">
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="432e8d2a-e15c-470e-9803-fcc7401d47d5" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="b70358e1-7b1d-4edd-af1f-f26ba7cb0912" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="a825e88a-3d1a-48c1-8c54-90f63177cb98" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="7bb928ff-0335-439e-93fd-2f650b458a86" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="ccf0cb52-36cb-4576-a1c6-dbee8c712b87" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="e39f39b2-0427-4be0-a730-2d2a0200c851" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="4d277352-5846-4efb-bd49-02084c60c46c" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="76a3a0e7-085f-45d0-aafe-4dda812da428" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="3065edab-1e39-42b0-a03b-b579ab6944cc" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="cc2709bd-d185-4bbd-a446-ad27bfd33b5b" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="2f24ecb3-b63a-4dc5-8c43-2eeffe6e7161" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="e217024f-ea97-492d-b069-6bd3498afccc" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="b5711c08-0dbb-422c-8eb6-869a069429b9" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="e1a338cb-88f9-44ef-a8d8-ef3072ccae0e" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="f040fab7-96e5-4186-9ac0-7b4a92ef95ba" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="3a12b8ea-5eec-4377-8dc1-897e3cee74d6" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="897d4de7-2e75-4ca1-b224-f74b23a94269" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="5fc73c6a-12b5-46b7-a3e9-d210f9903508" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="823ccbdd-b776-4c11-a228-87f66982650d" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="46f7130b-64c5-4c7b-8967-5eb20e8e01f8" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="1aee2c09-f241-4896-941f-5ded1325f22c" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="95314a5b-7ce3-4578-b478-74feffad4b68" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="6cd0f03a-c18c-44f9-8156-48451c49bf16" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="2e746ac6-906b-4a67-a917-ce0cc0720190" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="2ec6ae95-2648-407b-bdc4-5cb1cacca5e6" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
          <error id="24008" name="ReportingResourceChangeMessageFailed" uid="1d27a19a-6cb0-4431-8e85-47ab97449db4" QueueMessageBody="Resource UID: '5962409c-bce7-483e-bc9d-8559afb56cc3'. ChangeType='Alter'. ResourceChangeType='All'" Error="Failed to read the resource data from the published database. Resource UID='5962409c-bce7-483e-bc9d-8559afb56cc3" />
        </class>
        <class name="Queue">
          <error id="26000" name="GeneralQueueJobFailed" uid="42bf76aa-6ec1-4439-9b81-a5c7a627cca4" JobUID="42e68ecd-c53c-4ef0-a51c-7482a45f398d" ComputerName="W3SRV-FS-01" GroupType="ReportingResourceSync" MessageType="ReportResourceSyncMessageEx" MessageId="1" Stage="" />
        </class>
      </general>
    </errinfo>

    Due to this problem timesheet hours are not transferred to the Reporting database and my cubes are displaying wrong reports. 

    I tried to rebuild the reporting database using "Backup-Restore Enterprise Custom Fields" method, but reporting rebuild job completes very quickly without any effects (I think it should at least take a few minutes).

    Tuesday, July 17, 2012 6:17 AM

All replies

  • Hi Bijan,

    Yes, you are right, we need to fix reporting database.

    We can try RDB refresh but before we need to identify if there is any issue Enterprise Custom field, Look up table and Resources.

    Since RDB refresh is failing, open each Enterprise Custom field one by one and without making any changes click on save. Do validate queue job status once in a while. Similarly perform save operation of Lookup tables and resources.

    If everything goes well, check SQL server logs and if possible restart Project Queue service and initiate RDB refresh by performing administrative backup and restore of Enterprise Custom fields


    Hrishi Deshpande – Senior Consultant DeltaBahn
    Blog | < | LinkedIn

    Tuesday, July 17, 2012 3:02 PM
    Moderator
  • Hi

    The logs state that there are errors reading the data from the published database - why should the RDB refresh solve the issue (apart from that it might be, that the error message is misleading.... :-)

    Christoph

    Tuesday, July 17, 2012 3:57 PM
  • Hi Bijan

    did you check out if you receive any ULS error logs?


    Marc Soester [MVP] http://marcsoester.blogspot.com

    Wednesday, July 18, 2012 4:16 AM
    Moderator
  • Hi there--

    Error ID 24008 is related to RDS change message failed for a resource. Based on the error logs, It looks like there is a resource with RES_UID=5962409c-bce7-483e-bc9d-8559afb56cc3, having issue with account or other data setup.  You can run the below query in Reporting database to find the Resource name using Resource user view & get the Resource Name.

    select ResourceName, ResourceNTAccount from MSP_EpmResource_UserView where ResourceUID = '5962409c-bce7-483e-bc9d-8559afb56cc3'

    SELECT 
    ResourceName, 
    ResourceNTAccount 
    FROM MSP_EpmResource_UserView 
    WHERE ResourceUID = '5962409c-bce7-483e-bc9d-8559afb56cc3'

    Would suggest to open the Resource account & If you are using Resource level custom fields , try changing the values (text or lookup) & re-save it.

    Please let us know the result.


    If you found this post helpful, please “Vote as Helpful”. If it answered your question, please “Mark as Answer”. Thanks, Amit Khare |EPM Consultant| Blog: http://amitkhare82.blogspot.com http://www.linkedin.com/in/amitkhare82


    Wednesday, July 18, 2012 5:20 AM
  • Dear Hrishi

    I opened each one of the enterprise custom fields and clicked the 'Save' button. This operation creates a 'Reporting (Custom Field Metadata Sync)' job which fails every time with the following error:

    Error summary/areas:
    Reporting message processor failed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    ReportingCustomFieldMetadataChangeMessageFailed
    Queue
    GeneralQueueJobFailed
    Error details:
    
    <?xml version="1.0" encoding="utf-16"?>
    <errinfo>
      <general>
        <class name="Reporting message processor failed">
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="f06c332e-68fc-4c2d-b2f9-498a7101c5d3" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="6757d905-7254-4c28-8fdc-5cd5b5a617f7" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="0c570762-9a82-4799-afd5-83aa30e97f79" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="affadc11-a81a-46f9-b058-73ee6a38500e" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="1b8c592f-6a32-4f7e-92a7-5a658e233087" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="43b57e97-50d1-415d-b241-79631cf9469e" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="06cdcb36-9343-456f-91aa-6dd3c2df47eb" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="c44e4e40-bc4d-4241-8d1e-236720e86f77" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="4c996c90-33e2-41c0-bb35-0715c80c2973" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="88ba14c3-0fbc-47f5-bc4d-a7245e6b7860" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="b8ce8c26-e15d-4a55-a5af-edceff3fb7e4" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="84fa5635-2d13-4e11-b75f-8d9734412e58" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="7e0bd6f4-72cd-4731-9fcf-409b9ab0df12" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="ea4eaab4-1997-465b-a735-af96232194c2" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="77602fca-7633-4bb9-9228-59b430fd23d1" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="9dba130e-1ec4-4dcb-9b67-4a2fd9d322bb" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="fca4a5f7-7017-460f-9079-a7dc2064fe8a" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="e5e6ca1f-08cd-4fa1-994e-befbe2145074" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="6db51b7a-4512-4adb-83fc-baf37555b695" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="e09ebe92-a529-4806-a489-527547260697" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="cd02cdd0-c93a-4b27-ab21-d08f3581c24a" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="ce5d1a9f-aa18-4bdb-bb28-758a844a4293" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="0cb31947-ca90-458e-9233-3f1c3ce8cec5" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="df8de7c9-5e26-4f38-9c2f-78f4f5e0d145" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="086d4ee2-a3b6-4c61-9a49-bb0162eb98f2" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
          <error id="24002" name="ReportingCustomFieldMetadataChangeMessageFailed" uid="c9b3cdf5-1005-42e6-8a78-4ff801603caa" QueueMessageBody="CustomFieldTypeUID='0000783f-de84-434b-9564-284e5b7b3f49'. ChangeType='Alter'" Error="Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed." />
        </class>
        <class name="Queue">
          <error id="26000" name="GeneralQueueJobFailed" uid="d05bf98f-059f-44f9-8356-e1b7a96284cf" JobUID="330c9430-996a-4bb1-b43d-5b24d3d03f14" ComputerName="W3SRV-FS-01" GroupType="ReportingCustomFieldMetadataSync" MessageType="ReportCustomFieldMetadataSyncMessageEx" MessageId="1" Stage="" />
        </class>
      </general>
    </errinfo>

    Then I checked the ULC log and found this error:

    PWA:http://www.****.com/ProjectsServer, SSP:SharedServices, User:***, PSI:   SqlException occurred in DAL:  
    <Error>
    	<Class>16</Class>
    	<LineNumber>39</LineNumber>
    	<Number>50000</Number>
    	<Procedure>MSP_Epm_GenerateOlapEntityView</Procedure>  
    	<Message>  
    		System.Data.SqlClient.SqlError: Invalid parameter @EntityTypeUID when call MSP_Epm_GenerateOlapEntityView  
    	</Message>  
    	<CallStack>
    		at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)     
    		at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)     
    		at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)     
    		at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)     
    		at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)     
    		at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)     
    		at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)     
    		at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)     
    		at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()     
    		at Microsoft.Office.Project.Server.DataAccessLayer.DAL.SubDal.ExecuteStoredProcedureReturnVal(String storedProcedureName, SqlParameter[] parameters)  
    	</CallStack>  
    </Error>
    and a few number of these errors
    PWA:http://www.***.com/ProjectsServer, SSP:SharedServices, User:***, 
    PSI:   [RDS] ReportCustomFieldMetadataSyncMessage (add) failed. The operation will be rolled back. 
    Error: Microsoft.Office.Project.Reporting.ProjectReportingPublic.ReportException: Unable to add the custom field: 'Cost Type'. MSP_Epm_AddCustomField failed. ---> 
    System.Data.SqlClient.SqlException: Invalid parameter @EntityTypeUID when call MSP_Epm_GenerateOlapEntityView     
    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)     
    at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)     
    at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)     
    at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)     
    at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)     
    at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)     
    at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)     
    at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)     
    at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()    
    at Microsoft.Office.Project.Server.DataAccessLayer.DAL.SubDal.ExecuteStoredProcedureReturnVal(String storedProcedureName, SqlParameter[] parameters)     
    at Microsoft.Office.Project.Server.DataAccessLayer.ReportingMetadataDal.AddCustomField(CustomField customField)     
    --- End of inner exception stack trace ---     
    at Microsoft.Office.Project.Server.DataAccessLayer.ReportingMetadataDal.AddCustomField(CustomField customField)     
    at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RDSCustomFieldMetadataMessageProcessor.CustomFieldAdd(Guid customFieldTypeGuid, Boolean isRefreshJob, Guid jobInfoUid)

    saving look up tables did not lead to any errors though.

    Wednesday, July 18, 2012 6:15 AM
  • Hi there--

    Cost Type is custom field in a default installation of Project Server. Do you see the field in Enterprise custom fields & lookup table in PWA server settings? Is Cost Type deleted in your PWA instance?

    If yes, You will need to restore Enterprise custom fields using the Administrative restore from PWA> Server Settings>Administrative restore.


    If you found this post helpful, please “Vote as Helpful”. If it answered your question, please “Mark as Answer”. Thanks, Amit Khare |EPM Consultant| Blog: http://amitkhare82.blogspot.com http://www.linkedin.com/in/amitkhare82


    Wednesday, July 18, 2012 6:32 AM
  • Hi Bijan

    did you check out if you receive any ULS error logs?


    Marc Soester [MVP] http://marcsoester.blogspot.com

    Dear Marc

    I checked UCL logs after starting to refresh RDB and found this one:

     
    PWA:http://www.***.com/ProjectsServer, SSP:SharedServices, User:***, 
    PSI:   SqlException occurred in DAL:  
    <Error>
    	<Class>16</Class>
    	<LineNumber>24</LineNumber>
    	<Number>3728</Number>
    	<Procedure>MSP_Epm_DeleteForeignKeysOnTables</Procedure>  
    	<Message>  System.Data.SqlClient.SqlError: 'FK_MSP_EpmAssignment_ProjectUID' is not a constraint.  </Message>  
    	<CallStack>     
    		at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)     
    		at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)     
    		at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)     
    		at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream,BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)     
    		at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)     
    		at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)     
    		at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)    
    		at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)     
    		at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()    
    		at Microsoft.Office.Project.Server.DataAccessLayer.DAL.SubDal.ExecuteStoredProcedureReturnVal(String storedProcedureName, SqlParameter[] parameters)  
    	</CallStack>  
    </Error>

    and this one

    PWA:http://www.***.com/ProjectsServer, SSP:SharedServices, User:***, 
    PSI:   SqlException occurred in DAL:  
    <Error>
    	<Class>16</Class>
    	<LineNumber>24</LineNumber>
    	<Number>3727</Number>
    	<Procedure>MSP_Epm_DeleteForeignKeysOnTables</Procedure>  
    	<Message>  System.Data.SqlClient.SqlError: Could not drop constraint. See previous errors.  </Message>  
    	<CallStack>     
    		at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)     
    		at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)     
    		at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)     
    		at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)     
    		at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)     
    		at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)     
    		at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)     
    		at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)     
    		at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()     
    		at Microsoft.Office.Project.Server.DataAccessLayer.DAL.SubDal.ExecuteStoredProcedureReturnVal(String storedProcedureName, SqlParameter[] parameters)  
    	</CallStack>  
    </Error>

    and this one too

    PWA:http://www.***.com/ProjectsServer, SSP:SharedServices, User:***, 
    PSI:   [RDS] Delete of the EPM area of RDB failed. Error: Microsoft.Office.Project.Reporting.ProjectReportingPublic.ReportException: Unable to delete RDB-Epm data. 
    MSP_Epm_DeleteAllData failed with the error: 'FK_MSP_EpmAssignment_ProjectUID' is not a constraint.  Could not drop constraint. See previous errors. ---> 
    System.Data.SqlClient.SqlException: 'FK_MSP_EpmAssignment_ProjectUID' is not a constraint.  Could not drop constraint. See previous errors.     
    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)     
    at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)     
    at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)    
    at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)     
    at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)     
    at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)     
    at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)     
    at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)     
    at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()    
    at Microsoft.Office.Project.Server.DataAccessLayer.DAL.SubDal.ExecuteStoredProcedureReturnVal(String storedProcedureName, SqlParameter[] parameters)     
    at Microsoft.Office.Project.Server.DataAccessLayer.ReportingMetadataDal.DeleteAllEpmData()     
    --- End of inner exception stack trace ---     
    at Microsoft.Office.Project.Server.DataAccessLayer.ReportingMetadataDal.DeleteAllEpmData()     
    at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsRefreshMessageProcessor.PurgeEpmArea()

    Wednesday, July 18, 2012 6:49 AM
  • Hi there--

    Error ID 24008 is related to RDS change message failed for a resource. Based on the error logs, It looks like there is a resource with RES_UID=5962409c-bce7-483e-bc9d-8559afb56cc3, having issue with account or other data setup.  You can run the below query in Reporting database to find the Resource name using Resource user view & get the Resource Name.

    select ResourceName, ResourceNTAccount from MSP_EpmResource_UserView where ResourceUID = '5962409c-bce7-483e-bc9d-8559afb56cc3'

    SELECT 
    ResourceName, 
    ResourceNTAccount 
    FROM MSP_EpmResource_UserView 
    WHERE ResourceUID = '5962409c-bce7-483e-bc9d-8559afb56cc3'

    Would suggest to open the Resource account & If you are using Resource level custom fields , try changing the values (text or lookup) & re-save it.

    Please let us know the result.


    If you found this post helpful, please “Vote as Helpful”. If it answered your question, please “Mark as Answer”. Thanks, Amit Khare |EPM Consultant| Blog: http://amitkhare82.blogspot.com http://www.linkedin.com/in/amitkhare82


    Dear Amit

    Actually all my resources have this problem. I tried to open a number of the resources in Resource Center and save them. All save operations leads to this error.

    I tried to run you query on one of the resources and got this error from SQL:

    Msg 208, Level 16, State 1, Procedure MSP_EpmResource_UserView, Line 1
    Invalid object name 'dbo.MSP_EpmCPResVar0'.
    Msg 4413, Level 16, State 1, Line 39
    Could not use view or function 'ProjectServer_Reporting.dbo.MSP_EpmResource_UserView' because of binding errors.

    apparently there is something seriously wrong with my database!

    Wednesday, July 18, 2012 7:01 AM
  • Hi there--

    Cost Type is custom field in a default installation of Project Server. Do you see the field in Enterprise custom fields & lookup table in PWA server settings? Is Cost Type deleted in your PWA instance?

    If yes, You will need to restore Enterprise custom fields using the Administrative restore from PWA> Server Settings>Administrative restore.


    If you found this post helpful, please “Vote as Helpful”. If it answered your question, please “Mark as Answer”. Thanks, Amit Khare |EPM Consultant| Blog: http://amitkhare82.blogspot.com http://www.linkedin.com/in/amitkhare82


    This field exists and is not deleted.
    Wednesday, July 18, 2012 7:02 AM
  • I did not find out what was the cause of the problem but I found a way to rebuild my reporting database. This is what I did:

    1. Installed a fresh SharePoint 2007 and Project Server 2007 on a test server.

    2. Installed all Service Packs and CUs so far.

    3. Provisioned a project web access on this test server.

    4. Did a backup of reporting database of the fresh server.

    5. Restored the database on my corrupted reporting database.

    6. Rebuilt the reporting database by backup-restore of custom fields.

    It was not quick and painless but at least I do not have errors in my queue now.

    Monday, July 23, 2012 7:29 AM
  • Glad you got it resolved Bijan - sounds like the schema of your reporting DB was somehow not as expected.  I'm not sure you are out of the wood yet, as there are a couple of things that may come back and bite you later.  When you add a CU for example, unless the Collection GUID of your new reporting DB matches the 3 other project DBs, you may get errors and I'm not sure the GUID will get matched up just with a restore.  Check in SQL Management Studio by right clicking the DB then look at Extended Properties - each of tthe 4 Project DBs should contain the same GUID against the ProjectCollectionGuid property.  Update the Reporting to match the others if it does not already match.  Also you should check if your timesheet data is ok in the reporting DB, as this workaround is not normal and having started with a new clean reporting DB you may find you are missing some data.  I'd suggest opening a case if you need help getting things all straight again.

    Best regards,

    Brian.


    Blog | Facebook | Twitter | Posting is provided "AS IS" with no warranties, and confers no rights.
    Project Server TechCenter | Project Developer Center | Project Server Help | Project Product Page

    Monday, July 23, 2012 8:26 PM
    Owner