none
DPM 2016 RRS feed

  • Question

  • We recently migrated (fresh install of Windows Server 2016 and DPM 2016) and some of our backups started failing after a few days, more precisely SQL and Exchange databases.

    We' receiving the following error: The storage involving the current operation could not be read from or written to. (ID 40003)

    Additionaly we're having a hard time understanding how the new MBS works, it supposted to be more efficient, but we're seeing some very strange things:

    We're really not backing up a lot of data, 10GB should be more then enough.

    Also we can't seem to allocate disk space, during Protection Group creation or later. Why is that the case?

    Thursday, December 29, 2016 3:10 PM

All replies

  • I am getting the same error on a 2012 domain controller Hyper-V VM with the same dpm 2016 on server 2016.  vss writers seem to be waiting for completion or failed on the domain controller vm but windows server backup works without issue.  The only way we can backup this vm is to shut it down and then dpm will back it up.  Any thoughts?

    Mike

    Thursday, January 5, 2017 3:49 PM
  • Hi,

    Please make sure you install KB3190600 Update Rollup 1 for System Center 2016 Data Protection Manager as it had lots of fixes related to storage pool usage.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, January 6, 2017 7:00 PM
    Moderator
  • Hi Mike,

    I installed UR1 and ran the backup jobs again. All was fine for two days, now I'm yet again getting "The storage involving the current operation could not be read from or written to. (ID 40003)".

    With regards to storage, is it normal what I'm seeing. I created two 5TB virtual disks on the storage pool, but DPM goes from 25TB to 35TB of disk used, eventhought I don't have anywhere near that much data or that much space on the storage itself.

    Saturday, January 7, 2017 10:26 AM
  • Hello Mike,

    it appears not all the components were sucessfully updated to UR1. We ran the update again and all our issues are resolved right now. Hope it stays that way.

    Thursday, January 12, 2017 2:15 PM
  • I am getting the same error on a 2012 domain controller Hyper-V VM with the same dpm 2016 on server 2016.  vss writers seem to be waiting for completion or failed on the domain controller vm but windows server backup works without issue.  The only way we can backup this vm is to shut it down and then dpm will back it up.  Any thoughts?

    Mike

    After calling Microsoft support it turns out the following was our issue:

    Microsoft  set the Maximum Shadow Copy Storage  space to UNBOUNDED (vssadmin resize shadowstorage),  restarted the Com + Event System and the Volume Shadow Copy services. 

    We had another machine with Server 2016 & sccm 1610, that wouldn’t backup or checkpoint either.  We looked at the shadow copy storage, vssadmin list shadowstorage, and there was nothing listed.  So we created shadow copy storage on c: and d: drives, "vssadmin add shadowstorage /for=c: /on=c: /maxsize=unbound" (replaced c: with d: for the d drive) restarted the two services and HV checkpoint and DPM backups now works.  Hope this helps others.


    Mike

    Thursday, January 12, 2017 2:26 PM
  • I have another question that could be related to the issues we had. Can defragmentation cause these issues, or was it only a UR1 thing?

    Both drives that are used for DPM are currently 100% fragmented. Should I defrag them, leave them alone?

    Thursday, January 19, 2017 6:50 AM
  • We had the same problem. To solve it, i had to remove the last recovery point via Powershell like described here

    https://murison.wordpress.com/2015/01/12/removing-a-dpm-recovery-point/

    Tuesday, June 6, 2017 7:51 AM
  • Hi,

    I am facing similar issue. I deleted last recovery point just to check if it makes any difference but that didnt help.

    Any other suggestions?

    DPM Version : 5.0.322.0

    Regards,

    Pramodh.

    Friday, August 11, 2017 9:05 AM
  • Hi Everyone,

    Any solution on this?

    We started to face the same issue... 
    From time to time a backup will randomly fail with the following error:
    "The storage involving the current operation could not be read from or written to. (ID 40003)"

    Sometimes, by simply running the job again, it would start working... it does eventually work though, which is the strange thing. It's as if DPM or Windows gets in trouble trying to mount the volume, but then when you try again, it all works fine (???)

    Cheers,
    Stephane


    Stephane

    Monday, September 4, 2017 8:37 AM
  • I'm having this same issue.  Does this error indicate an issue on the DPM server itself or on the guests /servers you are trying to back up.

    I'm not seeing any errors on the target machines.

    I've tried the script to remove the last checkpoint and it didnt work.

    What does seem to work is removing a protected item (deleting from disk) and recreating.  This is really not desired though.

    I did read somewhere about defragging?  But defragging where, the dpm server area where these are stored?  I started a job and its only 2% done 2 hours later basically, so unsure this will work.


    Tech, the Universe, Everything: http://tech-stew.com Just Plane Crazy http://flight-stew.com

    Tuesday, November 5, 2019 9:03 PM
  • There is a known issue with Windows 2016 ReFs where the reference count table can get corrupt and lead to that error.  Basically we cannot create a new clone for the replica and mount it.

    DPMCurr.errlog shows this error.


    1AB8 2C84 02/08 17:49:49.084 17 intentservices.cpp(2920)   NORMAL CIntentServices::UnlockReplica(). datasourceId={00000000-0000-0000-0000-000000000000}, requestorId={5750004B-84BD-4EFC-B0F7-8849602FACDD}, deleteAllClones=1
    1AB8 44C0 02/08 17:49:49.100 11 StorageManager.cs(873)  B8BA9740-BCA2-4551-A148-44715F452028 NORMAL ==>CreatePIT(RequestId: b8ba9740-bca2-4551-a148-44715f452028 ContainerId: dfc09282-056c-405a-900d-febd934a3be6 StorageId: 9dedf0c3-9ca8-4ef7-91a4-560f542255e1 StorageType: ReFSVolume)
    1AB8 44C0 02/08 17:49:49.412 11 refsinstrumentation.cpp(866)   WARNING Failed: Hr: = [0x8007015b] : Encountered Failure: : lVal : DeviceIoControl( hDestFile, ( ((0x00000009) << 16) | ((( 0x0002 )) << 14) | ((209) << 2) | (0) ), &duplicateExtentsData, static_cast<DWORD>(sizeof(duplicateExtentsData)), 0, 0, &dwBytesReturned, 0 )
    1AB8 44C0 02/08 17:49:49.412 11 refsinstrumentation.cpp(835)   WARNING Failed: Hr: = [0x8007015b] : Encountered Failure: : lVal : DuplicateExtents(hSourceFile, hDestFile, llCurrentOffset, llCurrentOffset, llByteCount)
    1AB8 44C0 02/08 17:49:49.412 11 refsinstrumentation.cpp(174)   WARNING Failed: Hr: = [0x8007015b] : Encountered Failure: : lVal : DuplicateFile(hSourceFile, hDestFile, liFileSize.QuadPart)
    1AB8 44C0 02/08 17:49:49.412 11 StorageManager.cs(873)  B8BA9740-BCA2-4551-A148-44715F452028 NORMAL <--CreatePIT
    1AB8 1FC4 02/08 17:49:49.412 27 ShadowCopyBlock.cs(271)   WARNING Create SC failed with error code = SMIOError
    1AB8 1FC4 02/08 17:49:49.412 27 ShadowCopyBlock.cs(403)  B8BA9740-BCA2-4551-A148-44715F452028 NORMAL Aborted/Failed SC creation task
    1AB8 1FC4 02/08 17:49:49.412 27 ShadowCopyBlock.cs(450)  B8BA9740-BCA2-4551-A148-44715F452028 WARNING Create shadow copy failed with error code = SMIOError
    1AB8 1FC4 02/08 17:49:49.412 27 FsmBlock.cs(130)  B8BA9740-BCA2-4551-A148-44715F452028 WARNING Backup.ShadowCopyBlock : <-- Exited FSM block with FAILURE (errorCode = SMIOError) ErrorInfo = SMIOError
    1AB8 1FC4 02/08 17:49:49.412 27 RARequestor.cs(190)  B8BA9740-BCA2-4551-A148-44715F452028 NORMAL Sending Command RACancelAllSubTasks to RAForWrite on server LA-DPM.Contoso.com for workitem 6b51c37d-1767-491f-8260-ed02e0b7c3f2
    1AB8 54F8 02/08 17:49:49.428 27 RARequestor.cs(190)  B8BA9740-BCA2-4551-A148-44715F452028 NORMAL Sending Command RACancelAllSubTasks to RAForRead on server sv4-vhost2.Contoso.com for workitem 5d5c76e0-5d9e-4cdb-9983-edbb7711cead
    1AB8 3F64 02/08 17:49:49.834 27 BackupMachine.cs(2660)  B8BA9740-BCA2-4551-A148-44715F452028 NORMAL BackupMachine: Cleaning up ROlist for DatasetId [d203d4d7-f982-40af-95cd-d7a17f1380d4] in DBCleanup, will also mark Dataset as Invalid
    1AB8 5DD0 02/08 17:49:49.850 18 fsutils.cpp(4047)   WARNING Failed: Hr: = [0x80070002] : Encountered Failure: : lVal : HRESULT_FROM_WIN32(dwError)

    ErrorCode: 0x8007015B (-2147024549) = A file system block being referenced has already reached the maximum reference count and can't be referenced any further.

    WORKAROUND
    ==========

    1) Wait for all current backup jobs to complete or cancel them.
    2) Close the DPM Console.
    3) Stop and disable the DPM Service using service control manager.
    4) Open an administrative command prompt and run the below command.

    C:\>refsutil leak X: /x /a /t 6  (Where X: is the drive letter of the MBS volume hosting the problematic data source)

    5) After it completes Enable the DPM service - then open the DPM Console.
    6) Run consistency checks on the data sources that were failing backups.

    This issue if fixed in Windows 2019 - so upgrade to DPM 2019 then Windows 2019 to avoid future problems.

    Regards
    Mike Jacquet


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, November 6, 2019 4:34 PM
    Moderator