none
DPM encounterd error from VMware Disk SubSystem. ErrorCode - 3014 (ID 33620 Details: Internal error code: 0x80990EF1) RRS feed

  • Question

  • Hi,

    I am getting below message while taking backup of the VMware machine.

    DPM encounterd error from VMware Disk SubSystem. ErrorCode - 3014 (ID 33620 Details: Internal error code: 0x80990EF1)

    Machine looks fine and there are no issues with it. I am able to backup most of the machines, however getting error on 2 machines. Kindly suggest how to fix it.

    Monday, December 2, 2019 12:38 PM

Answers

  • I think, I now know the resolution for this along with changing the permission and adding Virtual Machine->Provisioning->Allow Virtual Machine Download. We need to reset the CBT(Change block tracking) per VM by using below script. After doing this all works fine.

    Connect-VIServer vCenter
    $vm="VMName" 
    $vmtest = Get-vm $vm| get-view
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
     
    #disable ctk
    $vmConfigSpec.changeTrackingEnabled = $false
    $vmtest.reconfigVM($vmConfigSpec)
    $snap=New-Snapshot $vm -Name "Disable CBT"
    $snap | Remove-Snapshot -confirm:$false
     
    # enable ctk
    $vmConfigSpec.changeTrackingEnabled = $true
    $vmtest.reconfigVM($vmConfigSpec)
    $snap=New-Snapshot $vm -Name "Enable CBT"
    $snap | Remove-Snapshot -confirm:$false

    • Marked as answer by SC2317 Wednesday, December 4, 2019 9:40 AM
    Tuesday, December 3, 2019 4:09 PM

All replies

  • Hi,

    I would suggest to check the DPM log on the DPM server for more clues, you'll find the log over here:

    • C:\Program Files\Microsoft System Center\DPM\DPM\Temp\DPMRACurr.errlog

    I would suggest you to move the log to a safe location (cut > paste), then retry the operation again to reproduce the error in a fresh DPMRACurr.errlog file, then upload the file to OneDrive and share the link here.

    This is to avoid having excess logs that we are not interested in, it makes troubleshooting easier.

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, December 2, 2019 12:55 PM
  • Hi Leon,

    I just noticed something that if I provide administrator permission to the service account that I use to take DPM backups then it works fine for the machine that was failing earlier. However, if I only use below permission on vCenter as per Microsoft documentation then its failing. It is just happening on 2 VMs and all other machines are getting backed up fine using only below permissions only.

    Name                        Description
    ----                        -----------
    Anonymous                   The only privilege held by sessions which have not logged in
    View                        Visibility without read access to an entity. This is assigned implicitly by the system, if read privileges are assigned at lower levels in the inventory
    Read                        Grants read access to an entity
    Manage custom attributes    Add, remove, and rename custom attribute definitions
    Allocate space              Allocate space on a datastore
    Assign network              Assign network to virtual machine, host service console, VMkernel virtual NIC or physical NIC
    Create new                  Create a new virtual machine or template
    Power Off                   Power Off a virtual machine
    Add new disk                Create and attach a new virtual disk
    Host USB device             Add, remove or edit a virtual USB device backed by a host USB device
    Advanced                    Make advanced configuration changes
    Swapfile placement          Set the placement policy for a single virtual machine's swapfile
    Disk change tracking        Enable or disable change tracking for the virtual machine's disks
    Create snapshot             Create a snapshot
    Remove Snapshot             Remove a snapshot
    Allow read-only disk access Allow read-only random access to disk files through a separate NFC connection

    Tuesday, December 3, 2019 9:58 AM
  • Yes DPM must at least have the permissions you've defined.

    What differences are there between the two virtual machines that are not working?

    Different disks/hardware? Does it have snapshots? Mounted ISOs?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, December 3, 2019 10:24 AM
  • There is no difference because I have other machines with same hardware version and running on same type of ESXi hosts and they are getting backed up.

    Not sure, what extra permission for these 2 machines is required that makes backup to succeed if role is administrator. 

    Tuesday, December 3, 2019 10:32 AM
  • The DPM log might give us a better clue.

    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, December 3, 2019 11:01 AM
  • Hi Leon,

    Please use below link and download the file.

    https://drive.google.com/file/d/1qkDBUJ2Ovr_pXQPFHqRUQd2rNIYmW2p6/view?usp=sharing

    Tuesday, December 3, 2019 12:16 PM
  • There are a lot of permission denied errors in the log, the VMWareErrorCode = 3014 also means that there are insufficient permissions:

    3014 – VIX_E_HOST_USER_PERMISSIONS
    Insufficient permissions in host operating system.

    -----------------------------------------------------------------------------------------

    Are these virtual machines located on a separate vCenter or VMware server?

    I suggest you to double check and read carefully the documentation below in case you missed something:
    Use DPM to back up and restore VMware virtual machines


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, December 3, 2019 12:34 PM
  • Hi Leone,

    This is already configured and on the same vCenter I am able to take backup of other machines. I had already mentioned the privileges that account has on vCenter. Below are the ones.

    Name                        Description
    ----                        -----------
    Anonymous                   The only privilege held by sessions which have not logged in
    View                        Visibility without read access to an entity. This is assigned implicitly by the system, if read privileges are assigned at lower levels in the inventory
    Read                        Grants read access to an entity
    Manage custom attributes    Add, remove, and rename custom attribute definitions
    Allocate space              Allocate space on a datastore
    Assign network              Assign network to virtual machine, host service console, VMkernel virtual NIC or physical NIC
    Create new                  Create a new virtual machine or template
    Power Off                   Power Off a virtual machine
    Add new disk                Create and attach a new virtual disk
    Host USB device             Add, remove or edit a virtual USB device backed by a host USB device
    Advanced                    Make advanced configuration changes
    Swapfile placement          Set the placement policy for a single virtual machine's swapfile
    Disk change tracking        Enable or disable change tracking for the virtual machine's disks
    Create snapshot             Create a snapshot
    Remove Snapshot             Remove a snapshot
    Allow read-only disk access Allow read-only random access to disk files through a separate NFC connection

    Tuesday, December 3, 2019 12:38 PM
  • Has these virtual machines that fail been backed up by any other software or backed up by using any other method before?

    Here's basically a summary of the errors you're receiving in the DPM log:

    WARNING	Error in exclusion of disk with exception System.IO.FileNotFoundException: Could not find file 'C:\Program Files\Microsoft System Center\DPM\DPM\bin\excludedisk.xml'.
    VMWareServer.cs(2174)			WARNING	File name: 'C:\Program Files\Microsoft System Center\DPM\DPM\bin\excludedisk.xml'
    VMWareServer.cs(2174)			WARNING	   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
    VMWareServer.cs(2174)			WARNING	   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
    VMWareServer.cs(2174)			WARNING	   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
    VMWareServer.cs(2174)			WARNING	   at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials, IWebProxy proxy, RequestCachePolicy cachePolicy)
    VMWareServer.cs(2174)			WARNING	   at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
    VMWareServer.cs(2174)			WARNING	   at System.Xml.XmlTextReaderImpl.FinishInitUriString()
    VMWareServer.cs(2174)			WARNING	   at System.Xml.XmlReaderSettings.CreateReader(String inputUri, XmlParserContext inputContext)
    VMWareServer.cs(2174)			WARNING	   at Microsoft.Internal.EnterpriseStorage.Dls.VMWareSDK.VMWareServer.GetExcludedDiskFileNames(String uuid)
    VMWareServer.cs(2174)			WARNING	Error in exclusion of disk with exception System.IO.FileNotFoundException: Could not find file 'C:\Program Files\Microsoft System Center\DPM\DPM\bin\excludedisk.xml'.
    VMWareServer.cs(2174)			WARNING	File name: 'C:\Program Files\Microsoft System Center\DPM\DPM\bin\excludedisk.xml'
    VMWareServer.cs(2174)			WARNING	   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
    VMWareServer.cs(2174)			WARNING	   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
    VMWareServer.cs(2174)			WARNING	   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
    VMWareServer.cs(2174)			WARNING	   at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials, IWebProxy proxy, RequestCachePolicy cachePolicy)
    VMWareServer.cs(2174)			WARNING	   at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
    VMWareServer.cs(2174)			WARNING	   at System.Xml.XmlTextReaderImpl.FinishInitUriString()
    VMWareServer.cs(2174)			WARNING	   at System.Xml.XmlReaderSettings.CreateReader(String inputUri, XmlParserContext inputContext)
    VMWareServer.cs(2174)			WARNING	   at Microsoft.Internal.EnterpriseStorage.Dls.VMWareSDK.VMWareServer.GetExcludedDiskFileNames(String uuid)
    dmsupport.cpp(586)	[0000025211E3BD90]	EBE558D9-6168-4C61-88F2-731D0098D2FE	WARNING	Failed: Hr: = [0x80070534] : Error trying to get the Sid of Machine XXXXvc02.XXXXsoftware.com
    dmsupport.cpp(514)	[0000025211E3BD90]	EBE558D9-6168-4C61-88F2-731D0098D2FE	WARNING	Failed to add Machine XXXXvc02.XXXXsoftware.com to group DPMRADmTrustedMachines, Ignoring the failure.
    securityutils.cpp(61)		EBE558D9-6168-4C61-88F2-731D0098D2FE	WARNING	GetAccountSid failed for XXXXvc02.XXXXsoftware.com$
    securityutils.cpp(49)		EBE558D9-6168-4C61-88F2-731D0098D2FE	WARNING	Failed: Hr: = [0x80070534] : Encountered Failure: : lVal : HRESULT_FROM_WIN32(dwErr)
    securityutils.cpp(216)		EBE558D9-6168-4C61-88F2-731D0098D2FE	WARNING	Failed: Hr: = [0x80070534] : Encountered Failure: : lVal : GetAccountSid(ssMachineAccount.PeekStr(), ppsid)
    vixdiskwrapper.cpp(316)	[0000025213D30D20]	3EF5E875-CABA-4CF3-8036-24F642C672D5	NORMAL	Calling VIX API:VixDiskLib_Open with arguments fForWrite:0, Diskpath:[XXX01-LUN08-GOLD-6TB] XXXXPRODRDS01/XXXXPRODRDS01-000003.vmdk.
    vixdiskutils.cpp(24)		3EF5E875-CABA-4CF3-8036-24F642C672D5	NORMAL	Message From VDDK Lib:VixDiskLibVim: Failed to open disk using NFC. VixError 3014 at 985.
    vixdiskwrapper.cpp(317)		3EF5E875-CABA-4CF3-8036-24F642C672D5	WARNING	VMWareErrorCode = 3014
    vixdiskwrapper.cpp(317)		3EF5E875-CABA-4CF3-8036-24F642C672D5	WARNING	VMWareErrorCode = 3014
    dsmsendersubtaskbase.cpp(588)	[000002522EA7C6E0]	3EF5E875-CABA-4CF3-8036-24F642C672D5	WARNING	Failed: Hr: = [0x80990ef1] SendData() failed

    Since most of your backups are working, I would simply create new virtual machines and attach the existing .VMDK files to the new virtual machines and check if the issue reappears.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, December 3, 2019 3:07 PM
  • Hi Leon,

    Today, I configured few more machines that were running in the same cluster where machine with failed backup was running. Surprisingly, only 1 machine out of 8 machines is running successful backup. However, after multiple hit & trial method I found that there is 1 extra permission that is required to DPM account for backup and that Virtual Machine->Provisioning->Allow Virtual Machine Download.

    After assigning this permission, issue with the failed backup on the machine is fixed. I'll see if all other backups also runs fine.

    Still not sure why other backups were running fine with those permissions. May be Microsoft needs to look into this, so that other users also do not face this issue.


    • Edited by SC2317 Tuesday, December 3, 2019 3:49 PM
    Tuesday, December 3, 2019 3:47 PM
  • What version of VMware are you running?

    It could be a new permission or a new required permission that simply hasen’t been documented yet.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, December 3, 2019 3:50 PM
  • It is vSphere 6.0 and just after running some other backups I still have issue on few machines. Its getting difficult to identify if its VM level or permission level issue.
    Tuesday, December 3, 2019 4:01 PM
  • I think, I now know the resolution for this along with changing the permission and adding Virtual Machine->Provisioning->Allow Virtual Machine Download. We need to reset the CBT(Change block tracking) per VM by using below script. After doing this all works fine.

    Connect-VIServer vCenter
    $vm="VMName" 
    $vmtest = Get-vm $vm| get-view
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
     
    #disable ctk
    $vmConfigSpec.changeTrackingEnabled = $false
    $vmtest.reconfigVM($vmConfigSpec)
    $snap=New-Snapshot $vm -Name "Disable CBT"
    $snap | Remove-Snapshot -confirm:$false
     
    # enable ctk
    $vmConfigSpec.changeTrackingEnabled = $true
    $vmtest.reconfigVM($vmConfigSpec)
    $snap=New-Snapshot $vm -Name "Enable CBT"
    $snap | Remove-Snapshot -confirm:$false

    • Marked as answer by SC2317 Wednesday, December 4, 2019 9:40 AM
    Tuesday, December 3, 2019 4:09 PM
  • Yes, its same. However, I just wanted to execute on machines where I was having issue so executed previous one.

    Good that, now I have fix for this one and maybe someone else can also advantage from this.Microsoft needs to improve the documentation or have more knowledge base according to issues faced by customers.

    Many thanks for your assistance once again. Will update you if I face any new issues.

    Wednesday, December 4, 2019 9:40 AM