none
DPM2012 SP1 - Cleaning job running for 20hrs+ - can't cancel RRS feed

  • Question

  • During the middle of our end of month tap backup run a cleaning job has been started, but has been running for over 20hrs.  I have ended up clicking Cancel Cleaning under Management -> Libraries but the job has still not cancelled 2hrs later.

    Anybody know how I can cancel this job so I can continue to backup the last few remaining protection groups?

    Sunday, June 2, 2013 9:19 PM

Answers

  • Hi,

    OK - thanks for that - it seems the library is in good working order, which means most likely this is a DPMDB inconsistency issue.  Unfortunately, it will be hard to troubleshoot in a forum setting.  Please open a support incident for additional troubleshooting, or restore DPMDB from known working state / time.  


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    • Marked as answer by adamf83 Wednesday, July 3, 2013 7:00 PM
    Tuesday, June 4, 2013 3:28 PM
    Moderator

All replies

  • Hi Adam,

    It seems DPM got stuck.

    Please run following queries

    use DPMDB

    Select * from tbl_JM_JobTrail

    WHERE JobId IN (SELECT DISTINCT JobId FROM tbl_TE_TaskTrail WHERE ExecutionState NOT IN (2,3))

    Select * from tbl_TE_TaskTrail

    WHERE ExecutionState NOT IN (2,3)

    Select * from tbl_PRM_LogicalReplica where OwnerTaskIdLock is not null and OwnerTaskIdLock not in ( Select TaskId from tbl_TE_TaskTrail where ExecutionState not in (2,3)) 

    Please post the result and let me know which job that you want to cancel by bolding the record.

    Monday, June 3, 2013 3:16 AM
  • Hery,

    This is what I want to cancel:

    4EE1771E-811D-4556-8D75-A960A60CCDDC 9D6A815D-835C-4E67-A3E1-5976E9DED0C6 NULL Cancel 2013-06-01 22:03:50.367 NULL <Job JobDefID="9d6a815d-835c-4e67-a3e1-5976e9ded0c6" JobID="4ee1771e-811d-4556-8d75-a960a60ccddc" ContinueOnTaskFailure="true" PreviousState="Execute" JobState="Cancel" MaxDuration="86400" StartDateTime="2013-06-01T22:03:50.3669235Z" ScheduleID="00000000-0000-0000-0000-000000000000" CancelInfo="UserRequest" xmlns="<Task">http://schemas.microsoft.com/2003/dls/JobManager.xsd"><Task TaskDefID="91429468-b46b-4806-9dc0-d3b9e8b82dfc" TaskID="3e18b625-b58f-45d8-9f3a-5a3d0dda18b9" VerbID="0317143e-4e2b-45f5-97b0-e5aefe449ef2" IsSerial="true" TaskState="GenerateWorkplan" IsExecutionSuccessful="false" HasErrors="false" /></Job>

    I have only pasted that record since, hope that is ok.

     

    Monday, June 3, 2013 6:56 AM
  • Hi Adam 

     

    As you provided, the jobid is 9D6A815D-835C-4E67-A3E1-5976E9DED0C6.

    Please backup your DPM DB first then, you may run queries below. 

    use DPMDB

    UPDATE tbl_JM_JobTrail 
    SET JobState = 'Succeeded', EndDateTime = GetDate() 
    WHERE JobId IN (SELECT DISTINCT JobId FROM tbl_TE_TaskTrail WHERE ExecutionState NOT IN (2,3))  and JobId ='9D6A815D-835C-4E67-A3E1-5976E9DED0C6'


    UPDATE tbl_TE_TaskTrail 
    SET ExecutionState = 2, ErrorCode = 0, LastStateName = N'Success', StoppeddateTime = GetDate() 
    WHERE ExecutionState NOT IN (2,3) and JobId='9D6A815D-835C-4E67-A3E1-5976E9DED0C6'

    update tbl_PRM_LogicalReplica 
    set OwnerTaskIdLock = null, Status = 8 
    where OwnerTaskIdLock is not null and OwnerTaskIdLock not in ( Select TaskId from tbl_TE_TaskTrail where ExecutionState not in (2,3) and JobId='9D6A815D-835C-4E67-A3E1-5976E9DED0C6'

    Hope it helps.

    Hery He

    Monday, June 3, 2013 7:58 AM
  • I ran that query and it came back with 0 rows affected.

    Any ideas?

    Cheers

    Adam.


    • Edited by adamf83 Monday, June 3, 2013 12:33 PM
    Monday, June 3, 2013 12:33 PM
  • Finally got that sorted, but I am now having problems with re-running jobs.  I am getting the following error:

    DPM could not reserve the drive resource because one of required drive resources is not online or it needs cleaning or servicing. (ID 24052)

    I've started it cleaning, but then looking in the webui of the library nothing is happening, no activity on the library and it fails with the same error above.  The webui is also reporting the drive doesn't need cleaning.

    Any suggestions how I can get past this?

    Monday, June 3, 2013 7:51 PM
  • Hi,

    Please refer to the following blog to help get the tape drive functional again.

    http://blogs.technet.com/b/dpm/archive/2012/02/22/dpm-tape-jobs-take-longer-than-expected-or-fail-with-id-24052.aspx


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, June 4, 2013 12:09 AM
    Moderator
  • Mike,

    The script says that no drive needs cleaning, yet jobs are still failing with the same error.

    Tuesday, June 4, 2013 7:37 AM
  • Please  download and run the DPMtapeLibraryCompatibilityChecker tool from the following link. To check physical state of the library and drives. 


    For tape library
    ================

    Run the following command to list the libraries attached to the system:

    DPMLibraryTest.exe /Certify /LL

    Run the following command, where \\.\Changer# is the symbolic name of the tape library to test from the output of the /LL command.

    DPMLibraryTest.exe /CERTIFY /TL \\.\Changer0 /ST 1 2 4 5

     

    Description of the tests.

    Test 1: Basic Configuration
    This test will scan your system for attached devices. It will identify the libraries and the stand-alone drives attached to your system and give you a summary of what it finds. For each device, you will see a Device Name, Serial Number, Vendor Name, Product Name, Firmware Revision and SCSI properties.
    At the end of this test, the user must verify whether all the information as seen by the tool. If the information is incorrect, check the following:
    • Check if all the devices are listed in Device Manager.
    • Ensure that the drivers for the devices are up-to-date.
    • If the drive mappings are incorrect, use the DPMDriveMapping.exe to correct the mappings. If you do not have DPM installed on the computer, copy the DPMLA.xml that DPMDriveMapping.exe creates to the folder to which you extracted the Tape Library Certification tool.

    Test 2: Mount/Dismount
    This test will pick a tape from the first available slot and perform a mount/dismount of that tape to/from a drive.

    Test3: Drive Cleaning
    A cleaner cartridge in needed for this test to run. Please read the pre-requisites section for this test. If you are using Firestreamer or a VTL where you cannot remove/change tapes, please use the /ST flag of the tool to skip the cleaner test.

    Test4: I/E Media
    This test will pick up the first available tape and move it to the I/E port and back. If your library/VTL does not have I/E ports the tool will automatically skip the test.

    Test5: I/O
    This test will pick up the first available writable tape and write a few buffers to it and try reading what is written. The purpose of this test is only to check for reading/writing ability.  Any specific/transient errors in any drive have to be inspected using the Advance Mode.
     Warning
    This test will overwrite information in the tape. Ensure that the first available tape does not have important data on it.

    The log file will be in the same location as the test tool was ran from.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, June 4, 2013 2:45 PM
    Moderator
  • Mike,

    I've run the tests.  The files have been uploaded to our SkyDrive: https://skydrive.live.com/#cid=2CAC77A324AACA98&id=2CAC77A324AACA98%21124

    Cheers

    Adam.

    Tuesday, June 4, 2013 3:06 PM
  • Hi,

    OK - thanks for that - it seems the library is in good working order, which means most likely this is a DPMDB inconsistency issue.  Unfortunately, it will be hard to troubleshoot in a forum setting.  Please open a support incident for additional troubleshooting, or restore DPMDB from known working state / time.  


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    • Marked as answer by adamf83 Wednesday, July 3, 2013 7:00 PM
    Tuesday, June 4, 2013 3:28 PM
    Moderator