none
MABS 13.0.415.0 New Install Fails to Backup Vmware (Vmware Disk SubSystem) ErrorCode 14009 RRS feed

  • Question

  • Hi,

    New to MABS. Which we have deployed to a server in Azure.

    We have configured MABS with access to our on-prem vCenter (6.5) and created protection groups for the vms.

    When running consistency check they all fail with this error at around 10 minutes.

    DPM encounterd error from VMware Disk SubSystem. Server - **MACHINENAME** ErrorCode - 14009 (ID 33620 Details: Internal error code: 0x80990EF1)

    I have been digging around online for hours trying all kind of solutions but nothing has changed. These machine were previously backed up fine with Veeam and CBT has been reset on all the machines.

    Anyone have a suggestion of where to look? No errors seen in event viewer and I can see MABS creating snapshots when jobs run.

    Thanks,

    Craig

    Thursday, June 13, 2019 4:27 PM

All replies

  • Hello Craig,

    First of all, did you follow the steps provided in the official documentation thoroughly?
    Back up VMware VMs with Azure Backup Server

    Here's a thread with the identical issue (the last reply), did you come up with this thread in your research? 
    VMWare backup with DPM 2012R2

    If everything is done according to the official documentation, and you're still having issues, I suggest checking the MABS logs for any more clues.

    You'll find the logs here:

    • %ProgramFiles%\Microsoft Azure Backup Server\DPM\DPM\Temp\MSDPMCurr.errlog

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, June 13, 2019 9:46 PM
  • Hi Leon,

    Thanks for coming back to me.

    First off yes, I have followed the guide to the letter for setup of MABS. Even going to the point of destroying the whole setup and redeploying on a new VM with everything brand new right down to clearing any tags in vmware and CBT.

    Right now I am just testing against one very small VM. Cleared its CBT and even tried keeping it powered off.

    Looking at the log file there are no errors I can see. Almost all entries have the status of NORMAL. The only WARNING I found says "AgentDeployment: bad column count in result set."

    I have the registry key enabled to disable SSL also and the certificates imported just in case.

    Account being used by MABS for vCenter has full admin rights to everything to see if that helped.

    Strange thing is, when I was doing this as a POC against a test machine before everything was rebuilt, I managed to successfully get the test machine backed up. Nothing has changed on-prem side, only a new box in Azure.

    Found that forum post you mentioned during my hours of digging and I have tried each of the solutions listed. Even tried back dating the VMDisk lib to an older version. Gave the same issue.

    Since then I have tried to backup another machine on a different host, that has ever been touched. Same results.

    Its got to be something stupid that I am just not seeing. Any thoughts? Can I provide anything that might help?

    Thanks

    Friday, June 14, 2019 8:35 AM
  • I'm not too familiar with MABS, but since it's more or less like a DPM under the hood it should work in a very similar way.

    Could you provide the latest MSDPMCurr.errlog by uploading it to OneDrive?

    I would suggest you "cut copy" all logs from the %ProgramFiles%\Microsoft Azure Backup Server\DPM\DPM\Temp folder to another location, this is to avoid many log files with hundreds/thousands of events.

    Then try performing a consistency check within MABS, this should create a new MSDPMCurr.errlog so we only have more or less the events from the failing consistency check, then upload the newly created log to a shared OneDrive, and I could take a look at it to see if I find something out of the ordinary.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Sunday, June 16, 2019 9:47 PM
  • Hi Lean,

    The log can be found here: https://bit.ly/2KSRmCF

    Looking through the log, the only real part jumping out is this..

    WARNING Backup.CommonLoop : RAReadDatasetFixup, StatusReason = Error (StatusCode = -2137452815, ErrorCode = VMWareVixDiskGenericError

    But everything I can find about this doesn't help.

    Thanks

    Monday, June 17, 2019 3:38 PM
  • When did you install your new MABS?

    I saw a lot of access denied errors on the 14th of June 2019 in your MSDPMCurr.errlog that you provided.

    There are a lot of these warnings:

    1238	0E14	06/17	08:13:53.165	27	BackupMachine.cs(2545)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	BackupMachine : FAILURE - BACKUP, errorCode=VMWareVixDiskGenericError
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	Task Diagnostic Information - <?xml version="1.0" encoding="utf-16"?>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	<TaskExecutionContext>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <PrmWriterId>18bee66c-826f-4499-a663-9805c8688ad3</PrmWriterId>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <PrmDatasourceId>fd7facc3-ad63-4ac9-a061-59a135c8c515</PrmDatasourceId>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <PrmPhysicalReplicaId>be155e57-67a9-4a65-916e-a28660b98290</PrmPhysicalReplicaId>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <PrmReplicaValidity>Invalid</PrmReplicaValidity>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <PrmReplicaStatus>Idle</PrmReplicaStatus>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <PrmOwnerLockId>00000000-0000-0000-0000-000000000000</PrmOwnerLockId>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <PrmLogicalReplicaId>d4892374-12cb-4c02-b119-099167b8a37c</PrmLogicalReplicaId>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <PrmDatasetId>c766604c-f024-423c-92cc-5bf221b44f53</PrmDatasetId>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <TEVerb>ValidateFixupReplica</TEVerb>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <TEErrorState>Backup.DataTransferLoopWaitState</TEErrorState>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  <TEErrorDetails>&lt;?xml version="1.0" encoding="utf-16"?&gt;
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	&lt;q1:ErrorInfo ErrorCode="134251348" DetailedCode="-2137452815" DetailedSource="2" ExceptionDetails="" xmlns:q1="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd"&gt;
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  &lt;q1:Parameter Name="protectedgroup" Value="Production Servers" /&gt;
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  &lt;q1:Parameter Name="servername" Value="hpvc01.p2udom.pharmacy2u.co.uk" /&gt;
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  &lt;q1:Parameter Name="agenttargetserver" Value="AZBKPRODSVER01.p2udom.pharmacy2u.co.uk" /&gt;
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  &lt;q1:Parameter Name="datasourceid" Value="fd7facc3-ad63-4ac9-a061-59a135c8c515" /&gt;
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  &lt;q1:Parameter Name="datasourcename" Value="HPCA01" /&gt;
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	  &lt;q1:Parameter Name="vmwareerrorcode" Value="14009" /&gt;
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	&lt;/q1:ErrorInfo&gt;</TEErrorDetails>
    1238	0E14	06/17	08:13:53.165	01	TaskInstance.cs(803)		72A1378E-5F5C-403C-A53E-A4280165326E	WARNING	</TaskExecutionContext>

    The VMware error code 14009 has to do with "the server refused the connection", this is normally caused by a firewall that's blocking the connection, it could also be some certificate or a secure protocol issue.

    For the rest I'm not too sure, but better do one thing at the time, to easier identify the root cause.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, June 17, 2019 10:03 PM
  • So.. Decided to pull down the whole thing and starting again. With all the changes that have been made it hard to tell whats helped and whats broken.

    I now have a brand new VM, with the latest version installed from scratch. I have imported the SSL certificates and everything is working as expected.

    I can connect to vCenter and everything seem OK.

    When doing the actual backup I am getting failure when MABS tries to connect to the ESX boxes.

    So, I think I am now down to a connectivity issue and we should be good.

    Thank you Leon for your help.

    Thursday, June 20, 2019 12:44 PM
  • I’m glad that I could be of any help.

    Let us know about your progress! 


    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, June 20, 2019 3:14 PM