none
DPM storage disk crashing. RRS feed

  • Question

  • 2 or 3 weeks ago Stood up a new DPM server 2012 R2. Created a single protection group with 28 databases on 4 servers and 2 folders. means 30 datasources.

    Storage pool is 60 TB, (30+30 TB) 2 disks.

    backups were frequently failing and while troubleshooting one 30TB disk went offline and lost all data. HP said and upgraded storage controller firmware and disk came online.

    I started backups again on this DPM 3 days back from scratch, today and yesterday randoms datasources failed with error : 

    Cannot create a recovery point because of a transient problem. (ID 215 Details: VssError:The shadow copy provider had an error. Check the System and Application event logs for more information.
     (0x80042306))

    To troubleshoot further more I ran command 'vssadmin list writers' and by pressing ENTER again one disk went offline which was having all data.

    -------------------------------------------------------------------------------------------------------------------------------------------------------------

    Since I created this group co-located > DPM creates 8 Volumes on this PG by its calculation

    Now when I checked Eventvwr one log is generating since morning when backups started failing :

    Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{37e5f2dd-5344-11e4-80c7-a0d3c1f8cd64} - 00000000000001EC,0x0053c008,0000008E0A42E5B0,0,0000008E0A42D5A0,4096,[0]).  hr = 0x80820001, The bootfile is too small to support persistent snapshots.


    Operation:
       Processing EndPrepareSnapshots

    Context:
       Execution Context: System Provider

    _________________________________________________________________________________

    Cant decide that this is hardware issue or I have to run chkdsk /r /f . I tried to run chkdsk /f on volume \\?\Volume{37e5f2dd-5344-11e4-80c7-a0d3c1f8cd64} by mounting it but it hangs system after no response with long time.

    Please help to troubleshoot this. Below is the volume lebels lins DPM created while creating PG

    ____________________________________________________________________________________________

    Possible values for VolumeName along with current mount points are:

        \\?\Volume{a8875d9d-92fb-11e3-80b3-806e6f6e6963}\
            *** NO MOUNT POINTS ***

        \\?\Volume{a8875d9e-92fb-11e3-80b3-806e6f6e6963}\
            C:\

        \\?\Volume{37e5f2ba-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\DiffArea\SqlServerWriter\vol_d0f6c0e7-9d20-4deb-ab91-7d61e68acde2\

        \\?\Volume{37e5f2c3-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\SqlServerWriter\vol_d0f6c0e7-9d20-4deb-ab91-7d61e68acde2\

        \\?\Volume{37e5f2cb-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\DiffArea\SqlServerWriter\vol_0550b7ac-7a8a-45bf-82ad-728369e5d5b2\

        \\?\Volume{37e5f2d1-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\SqlServerWriter\vol_0550b7ac-7a8a-45bf-82ad-728369e5d5b2\

        \\?\Volume{37e5f2d7-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\DiffArea\SqlServerWriter\vol_26cd530a-96c5-4b67-9d86-a380d07aff0d\

        \\?\Volume{37e5f2dd-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\SqlServerWriter\vol_26cd530a-96c5-4b67-9d86-a380d07aff0d\

        \\?\Volume{37e5f2e3-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\File System\vol_a2cd0b9d-709c-44e5-8e37-03464ac013a0\

        \\?\Volume{37e5f2e9-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\DiffArea\File System\vol_a2cd0b9d-709c-44e5-8e37-03464ac013a0\

        \\?\Volume{37e5f2ef-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\DiffArea\SqlServerWriter\vol_580e3d3e-e782-4331-b6e9-f6e2aeff7c19\

        \\?\Volume{37e5f2f5-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\SqlServerWriter\vol_580e3d3e-e782-4331-b6e9-f6e2aeff7c19\

        \\?\Volume{37e5f2fb-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\DiffArea\SqlServerWriter\vol_12f1ec92-5b65-4493-988b-e1da6d3f95f4\

        \\?\Volume{37e5f301-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\SqlServerWriter\vol_12f1ec92-5b65-4493-988b-e1da6d3f95f4\

        \\?\Volume{37e5f307-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\DiffArea\SqlServerWriter\vol_e07cbc10-116b-4eb1-95ae-c0618f4afc3b\

        \\?\Volume{37e5f30d-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\SqlServerWriter\vol_e07cbc10-116b-4eb1-95ae-c0618f4afc3b\

        \\?\Volume{37e5f313-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\DiffArea\SqlServerWriter\vol_65534060-53bc-4289-a11a-34c783a1629a\

        \\?\Volume{37e5f319-5344-11e4-80c7-a0d3c1f8cd64}\
            C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\SqlServerWriter\vol_65534060-53bc-4289-a11a-34c783a1629a\



    • Edited by V Jay Rana Thursday, October 16, 2014 12:36 AM
    Thursday, October 16, 2014 12:29 AM

All replies

  • What si teh version of DPM 2012 R2 which you were using? Did you install UR3?

    Were you having disk problems prior to installing DPM? Refer to this thread for more information:

    https://social.technet.microsoft.com/Forums/en-US/69ee6755-3f4e-46e5-8266-e8852b9ac0c7/backup-of-fileserver-fails-with-the-vss-application-writer-or-the-vss-provider-is-in-a-bad-state?forum=dataprotectionmanager


    Thursday, October 16, 2014 6:10 AM
    Moderator
  • Hello Trinadhk

    What si teh version of DPM 2012 R2 which you were using? Did you install UR3?

    >>The version of DPM we are using is SCDPM 2012 R2 4.2.1254.0 which is UR3. OS is Windows server 2012 R2 Standard 64 bit.

    Were you having disk problems prior to installing DPM? Refer to this thread for more information.

    >> This is new DPM server we build 3 weeks ago, everything is new, storage, hardware, OS, DPM. In three weeks this happens second time when disk went offline. Both time number of replica and RC jobs failed and both time repeated error log I found is :

    Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{37e5f2dd-5344-11e4-80c7-a0d3c1f8cd64} - 00000000000001EC,0x0053c008,0000008E0A42E5B0,0,0000008E0A42D5A0,4096,[0]).  hr = 0x80820001, The bootfile is too small to support persistent snapshots.

    Should I apply below fix ? 

    2929881 0x80070005 error when you run DPM jobs in Windows Server 2012 R2
    http://support.microsoft.com/kb/2929881/EN-US

    The actual error in my log is hr = 0x80820001, The bootfile is too small to support persistent snapshots. whereas this is different in fix. Not sure should apply this hotfix or not. 




    Thursday, October 16, 2014 9:43 AM
  • I am not able to bring the disk online now, A error log is being generated in Event viwer. May be this is the root cause.

    Event ID : 58

    Source : partmgr

    The disk signature of disk 1 is equal to the disk signature of disk 2.

    How to change signature of disk!

    Thursday, October 16, 2014 3:39 PM
  • Hi,

     Since the disks are GPT - it's not a straight forward operation to change the disk signature.  It may be quicker - easier to use diskpart.exe to "clean" disk1 - re-add it to the storage pool, then run dpmsync -reallocatereplica to remake the dpm volumes. .


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.



    Thursday, October 16, 2014 6:28 PM
    Moderator
  • Readded disks and disk signatures/IDs seems to be unique now.

    -----------------------------------------------------------------------------------------------------------------------------------------

    ISKPART> list disk

      Disk ###  Status         Size     Free     Dyn  Gpt
      --------  -------------  -------  -------  ---  ---
      Disk 0    Online          136 GB  6144 KB
      Disk 1    Online           29 TB    29 TB   *    *
      Disk 2    Online           29 TB    29 TB   *    *

    DISKPART> select disk 1

    Disk 1 is now the selected disk.

    DISKPART> uniqueid disk

    Disk ID: {84B74DED-5C0C-43D7-A05B-3013998DEF15}

    DISKPART> select disk 2

    Disk 2 is now the selected disk.

    DISKPART> uniqueid disk

    Disk ID: {BEABE0D2-6C03-40B2-9048-E43F32E0579A}

    ______________________________________________________________________________________

    Should I proceed with Testing protection groups now?

    Mike,

    Could you please check above error I posted  hr = 0x80820001, The bootfile is too small to support persistent snapshots. 



    • Edited by V Jay Rana Thursday, October 16, 2014 6:51 PM
    Thursday, October 16, 2014 6:45 PM
  • Hi,

    To be honest - I have never heard of that error before and even the description does not make sense.  Is this a Virtualized DPM Server ?

    Found this interesting post - please run chkdsk /f on effected volume(s)


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Thursday, October 16, 2014 7:28 PM
    Moderator
  • No, This is physical server. HP ProLiant DL360p Gen8  Server. Windows Server 2012 R2 Standard with DPM 2012 R2 with latest version 4.2.1254.0. Dont know what happend, Anyhow readded disk again and starting again from scratch, but worry if it happens again.

    I mounted that volume where getting this error and tried checkdisk. It lasts for hours without any response and finaly i closed.

    Since this is our second DPM server, but today my first DPM server (2012 SP1 UR7) also crashed before 1 hour before where more then 300 hundred jobs was in queue with idle timeout more than 7 or 10 hours. I cancelled jobs, restarted MSDPM service and then it crashed. All 30 TB data is on this. Still figuring Out.

    Its Black Thursday for me.


    • Edited by V Jay Rana Thursday, October 16, 2014 7:45 PM
    Thursday, October 16, 2014 7:43 PM