none
DPM issues on CSV (Compellent SAN) RRS feed

  • Question

  • We recently purchased a Compellent SAN.   Are there any known conflicts or issues we should watch out for?

    We are having strange issues when DPM backs up at night.  At times VMs seem to randomly lose connection.  Most nights at least 1 or 2 VMs get a disk error and have to be turned off the next morning and back on again.  Some nights this happens with many VMs.  One night we even had a VM get hung backing up and it froze the Hyper-V host it was on.

    Any ideas what could be causing these issues?  Does DPM conflict with replays (compellent snapshots)?  Or is it something else?

    One thing I did notice is that pretty much all the VMs are being backed up using Saved State rather than Online Backup.  I am not sure why this is.  They used to always use Online Backup.  Could that be causing issues?

    Any thoughts or advice would be helpful.

    Thank you.

    Monday, July 11, 2016 8:09 PM

Answers

  • Well, we think we have found the source of most of our issues. We were running Compellent Storage Center OS 6.7.5 and it has a bug with Windows Server and ODX which causes iSCSI targets to disappear. After upgrading to 6.7.11, we have not seen the issues again.

    Here is a link if anyone runs into the same thing:

    https://blog.workinghardinit.work/2016/02/22/dell-compellent-scos-6-7-odx-bug-heads-up/

    Anyway, we still haven't solved the backup issue, but at least our servers aren't going down anymore.


    Edit: Some people also say to turn off ODX as a precaution.
    • Edited by WVH-WA Wednesday, July 27, 2016 3:42 PM added last line
    • Marked as answer by WVH-WA Wednesday, July 27, 2016 3:42 PM
    Friday, July 22, 2016 7:59 PM

All replies

  • What versions of Hyper-V and DPM are you running?  If you are running current versions of Hyper-V and DPM, do the vm's that don't use Online Backup have SCSI adapters?  If you are not up to Hyper-V 2012 R2 and DPM 2012 R2 UR3, you may want to verify with your storage vendor that you are using their current VSS hardware provider.
    Monday, July 11, 2016 10:41 PM
  • We are running Hyper-V on Server 2012 R2 with the latest updates.   And DPM 2012 R2 UR10.

    I guess I will need to talk to the vendor about the storage.

    Tuesday, July 12, 2016 3:40 PM
  • At those version levels you would be using the VSS Null Provider, rather than a hardware provider from the vendor, to take the online snapshot.  But, this change does require the vm to have a SCSI adapter for the snapshot virtual hard disk.

    More info here: https://blogs.msdn.microsoft.com/virtual_pc_guy/2015/03/09/how-hyper-v-backup-got-better-in-2012-r2-but-now-requires-a-scsi-controller/

    Tuesday, July 12, 2016 4:40 PM
  • Just looked through a bunch of our VMs and all have a SCSI controller.  (I was assuming they did, since we've never gone through and removed them).

    As of weeks ago, all these VMs were backing up with an online snapshot when they were on local storage.  Once moved to the cluster, this changed.   Now only the few that are still on local storage use online snapshot.

    Tuesday, July 12, 2016 6:58 PM
  • Was the previous environment also Hyper-V 2012 R2? If not, you may need to update the integration services on the vm's. You might also double check that the Backup integration services is enabled for each vm.
    Wednesday, July 13, 2016 7:00 PM
  • Yes, the previous was HV 2012 R2 as well. Backup integration services is enabled. How do I check the integration services version?
    Wednesday, July 13, 2016 8:13 PM
  • You can use Get-VM to check the VMAdditions property of the virtual machines.  SCVMM will also show the version in the vm properties. 

    Really, this would only address the online vs offline backup.  The problems with vm's becoming unresponsive shouldn't happen with either backup option (except of course for the offline backup while the snapshot is taken). 

    Are you doing a separate SAN based backup of the LUN in addition to the DPM backups?


    Also, if you haven't seen it, this link details the requirements for online backups: https://technet.microsoft.com/en-us/library/hh757970(v=sc.12).aspx
    • Edited by EJ Henry Wednesday, July 13, 2016 10:19 PM add link
    Wednesday, July 13, 2016 10:10 PM
  • Well, we think we have found the source of most of our issues. We were running Compellent Storage Center OS 6.7.5 and it has a bug with Windows Server and ODX which causes iSCSI targets to disappear. After upgrading to 6.7.11, we have not seen the issues again.

    Here is a link if anyone runs into the same thing:

    https://blog.workinghardinit.work/2016/02/22/dell-compellent-scos-6-7-odx-bug-heads-up/

    Anyway, we still haven't solved the backup issue, but at least our servers aren't going down anymore.


    Edit: Some people also say to turn off ODX as a precaution.
    • Edited by WVH-WA Wednesday, July 27, 2016 3:42 PM added last line
    • Marked as answer by WVH-WA Wednesday, July 27, 2016 3:42 PM
    Friday, July 22, 2016 7:59 PM