System state and BMR backups are handled differently, so depending upon which of these is failing, we need to look in different places.
The system state backup is performed initially to the local disk (usually in c:\windowsimagebackup unless you've changed the location) and then transferred to DPM.
The BMR backup is created directly to a share on the DPM server.
It's worth selecting just the system state to backup initially and see if that works as that will help narrow down what's happening. Obviously if that fails, then look into the system state backup. If that works, reselect the BMR backup and see if that fails
again.
You can also see if the BMR backup works manually by creating a share on the DPM server, giving permissions to the machine account for the machine to be backed up, and then running the following command from the server that is to be protected:
wbadmin start backup -allcritical -backupTarget:\\<share created on the DPM server>
If that fails, try backing up to a local drive attached to the machine as well to see if that helps.
You can also test the system state backup manually by using the following command:
Wbadmin start systemstatebackup -backuptarget:e:
Where e: is the drive you want to use to store the system state backup.
If both of these succeed, then it seems more likely that it's a DPM issue.
Check the DPM logs on both the DPM and protected servers. On my DPM server, these are in %ProgramFiles%\Microsoft System Center 2012 R2\DPM\DPM\Temp named MSDPMCurr.errlog and on the protected server they're in %ProgramFiles%\Microsoft Data Protection
Manager\DPM\Temp and are named DPMRACurr.errlog.
Check also that things like anti-virus and real-time duplication of files to another server are off for the tests as well.
Andy