none
DPM backups failing silently

    Question

  • Hi, I have a small DPM 2012 R2 (Update Rollup 14) server, running on Server 2012 R2.  For the past month, scheduled backups have not been running.  All protection group healths are green, and there are no errors under Monitoring.  The list of Failed Tasks is empty.  Scheduled tasks a present and appear to be as expected, but tasks appear not to execute. There are no errors in the Event logs in either the DPM server or on the protected servers.  Manual consistency checks and recovery point creation works fine, but expired recovery cleanup does not appear to be happening (I have successfully run the pruneshadowcopiesDpm.ps1 script).

    The server has 5 Protection Groups. 4 of them are set to backup to disk once a week with 2 weeks retention, volumes and bare metal recovery.  The 5th takes SQL backups twice a day every day, with 14 day retention. I thought i might be running out of disk space, but I've added an additional disk to the storage pool and migrated some of the data (again using supplied powershell scripts). Needless to say I have rebooted the DPM server as well as it's remote SQL database.

    Any suggestions?

    Tuesday, May 15, 2018 4:14 PM

Answers

  • Hi,

    Okay, you could try to manually start a DPM backup job from the SQL server to see if it works or not.
    Please refer to this blog.

    Did the DPM jobs stop working after you implemented the Group Managed Service Accounts?
    This should be quite easy to test aswell.

    Kind regards,
    Leon


    Blog: https://thesystemcenterblog.com  LinkedIn:   

    • Marked as answer by d chris hall Thursday, May 17, 2018 3:59 PM
    Tuesday, May 15, 2018 6:46 PM

All replies

  • Hi,

    If your scheduled Jobs doesn't run at scheduled time, then the problem can be on SQL side. You may even not receive an error from DPM side. Just proper records under Scheduled Job tree, but without adequate records under Complieted Jobs. No missing records under Failed Jobs.

    To confirm source of the problem, you can check if there is the Event under Event Viewer > Application Event Log:
    Event ID: 208
    Level: Warning
    Description: SQL Server Scheduled Job '<ID>' (<hex>) - Status: Failed - Invoked on: <Time>- Message: The job failed. The Job was invoked by User <UserName>. The last step to run was step 1 (Default JobStep).
    Or:
    Event ID: 976
    Level: Error
    Resolution
    • Go to the MS SQL that is connected to the DPM
    • Check if all the service is up and running, especially SQL Server Agent. Restart the service, check running account.
    SQL services in order to DPM now up and running correctly
    • Go to Management Studio
    • Go to SQL Server Agent
    • Check if icon is green and branch extendable
    SQL Server Agent is down. DPM Jobs won't be executed.
    • Check SQL Jobs list and Job Activity Monitor under Job branch. The Jobs are strictly related with DPM Jobs. Try to manually execute one of the SQL Jobs.  
    Agent Job Activity - for DPM Jobs
    • Additionally you can query the SQL for Protection Group and it's SQL agent Schedule Job 
    SQL DPM query for Protection Group and its SQL agent Schedule Job

    See if this helps.

    Kind regards,
    Leon


    Blog: https://thesystemcenterblog.com  LinkedIn:   

    Tuesday, May 15, 2018 4:45 PM
  • Hi Leon,

    Thanks, i checked my SQL server and I do indeed have a mass of failed scheduled jobs and 208 and 976 events. My SQL  Server Agent is running, but even after a restart I can't manually kick off a scheduled job.  One additional detail, we recently implemented Group Managed Service Accounts, and the Server Agent is running under a group managed service account. Could that have anything to do with this issue?

    Thanks,

    Chris

    Tuesday, May 15, 2018 6:38 PM
  • Hi,

    Okay, you could try to manually start a DPM backup job from the SQL server to see if it works or not.
    Please refer to this blog.

    Did the DPM jobs stop working after you implemented the Group Managed Service Accounts?
    This should be quite easy to test aswell.

    Kind regards,
    Leon


    Blog: https://thesystemcenterblog.com  LinkedIn:   

    • Marked as answer by d chris hall Thursday, May 17, 2018 3:59 PM
    Tuesday, May 15, 2018 6:46 PM
  • Hi Leon,

    That article was exactly what i needed. I wasn't able to get DPM working with the GMSA, but I was able to reset the Server Agent service account back to the original account.  My scheduled backups appear to be resuming successfully.  Thanks so much for your help!

    Chris

    Thursday, May 17, 2018 3:59 PM