none
VMWare backup with DPM 2012R2 RRS feed

  • Question

  • I'm trying to back up VMware VMs using DPM 2012R2. I've been through the pre-reqs here:

    https://technet.microsoft.com/en-us/library/mt742097(v=sc.12).aspx

    I've installed update rollup 11 for DPM and I'm using ESXi version 5.5. I've also followed this guide:

    https://blogs.technet.microsoft.com/dpm/2016/08/26/four-simple-steps-to-backup-vmware-vms-using-sc-dpm/

    I added the vCenter server into the DPM console fine and created a protection group which is pointed at 2 folders in vCenter containing 9 VMs in total. When I try to run the job it fails with the following description:

    DPM encounterd error from VMware while trying to get ChangeTracking information. Server - gat-vcentre-01 ErrorCode - FileFaultFault (ID 33621 Details: Internal error code: 0x80990EF3)

    I've tried searching for the error but found nothing, I guess this is quite new as UR11 was released in August. I wondered if it may be a permissions issue if it can't get change tracking information so made the account that DPM uses Administrator in vCenter but it still fails. Has anyone come across this before?

    Thursday, September 29, 2016 10:53 AM

Answers

  • I too am having the same issue and hopefully someone will know why this is happening. I've tried two different vm's and got the same thing. I did however try another vm and successfully backed it up. The two vm's that failed were both around 15TB in size. The vm that successfully backed up was around 100GB.

    I believe the issue is that an application previously backed up the vm using cbt and left behind .ctk files in the vm's directory. there is a kb article to make sure cbt is enabled on the vm and how to reset cbt on the vm and remove the .ctk files. I am going to try this and see if it works. I'm pretty sure it is the problem.

    • Proposed as answer by laurnwerner Thursday, September 29, 2016 6:52 PM
    • Unproposed as answer by laurnwerner Thursday, September 29, 2016 6:52 PM
    • Edited by laurnwerner Thursday, September 29, 2016 6:58 PM
    • Proposed as answer by laurnwerner Thursday, September 29, 2016 6:58 PM
    • Marked as answer by liamfit Friday, September 30, 2016 4:37 PM
    Thursday, September 29, 2016 6:13 PM

All replies

  • I too am having the same issue and hopefully someone will know why this is happening. I've tried two different vm's and got the same thing. I did however try another vm and successfully backed it up. The two vm's that failed were both around 15TB in size. The vm that successfully backed up was around 100GB.

    I believe the issue is that an application previously backed up the vm using cbt and left behind .ctk files in the vm's directory. there is a kb article to make sure cbt is enabled on the vm and how to reset cbt on the vm and remove the .ctk files. I am going to try this and see if it works. I'm pretty sure it is the problem.

    • Proposed as answer by laurnwerner Thursday, September 29, 2016 6:52 PM
    • Unproposed as answer by laurnwerner Thursday, September 29, 2016 6:52 PM
    • Edited by laurnwerner Thursday, September 29, 2016 6:58 PM
    • Proposed as answer by laurnwerner Thursday, September 29, 2016 6:58 PM
    • Marked as answer by liamfit Friday, September 30, 2016 4:37 PM
    Thursday, September 29, 2016 6:13 PM
  • That's a good idea. I was previously using Veeam.
    Friday, September 30, 2016 9:20 AM
  • Having reset cbt I now get this error:

    DPM encounterd error from VMware Disk SubSystem. Server - gat-vcentre-01 ErrorCode - 3014 (ID 33620 Details: Internal error code: 0x80990EF1)

    Friday, September 30, 2016 9:44 AM
  • Bit more info... I followed https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2139574 to reset CBT. I used the PowerCLI script method as I didn't want to power my VMs off. I am trying to back up 9 VMs, 8 of them still fail with the second error:

    DPM encounterd error from VMware Disk SubSystem. Server - gat-vcentre-01 ErrorCode - 3014 (ID 33620 Details: Internal error code: 0x80990EF1)

    1 VM however has now backed up successfully. This was previously failing with the first error but since resetting CBT is now working.

    • Marked as answer by liamfit Friday, September 30, 2016 4:37 PM
    • Unmarked as answer by liamfit Friday, September 30, 2016 4:37 PM
    Friday, September 30, 2016 2:53 PM
  • OK last update... I found https://www.veeam.com/kb1937 which suggested the 3014 error was due to permissions. I tried adding the permissions in the article, i.e. Global > DisableMethods and EnableMethods
    Global License however it still failed. I went back to using the Administrator role for my DPM backup account and it worked.

    So the initial question was about the "error from VMware while trying to get ChangeTracking information" and the proposed answer was correct - it was related to CBT and resetting CBT by following https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2139574 was the resolution to that problem.

    Friday, September 30, 2016 4:37 PM
  • Hello! 

    I had some problem, but after I reset Changed Block Tracking and used Administrator role I get new error: 

    The protection agent on backup3.test was temporarily unable to respond because it was in an unexpected state. (ID 60 Details: Internal error code: 0x809909B0)

    Any ideas? 


    Alex

    Friday, October 7, 2016 5:28 AM
  • Hi,

    It seems like you are encountering the FileFaultFault error. This occurs when the CBT is enabled in a manner that the DPM cannot do incremental backups.

    To overcome FileFaultFault error scenario, can you try the following procedure

    1.        Disable CBT (Change Block Tracking) on Virtual machine.
    2.        Power off the VM
    3.        Power on VM.

     

    For more information on resetting CBT, please follow below link.

    https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2139574

    Please email me: masaran@microsoft.com and VMWareVMBkp@microsoft.com for any further issues.

    Thanks

    Maanas Saran

     


    Thursday, October 13, 2016 10:40 AM
  • Hi ,

    I have responded to the above query. Please see if you can use the workaround and start backing up your VMware VMs again.

    Let me know if you face any issues. Please mail me: masaran@microsoft.com and VMWareVMBkp@microsoft.com for any issues.

    Thanks

    Maanas Saran

    Thursday, October 13, 2016 10:50 AM
  • Hello! 

    I reseted CBT. 

    The problem persist. 


    Alex

    Thursday, October 13, 2016 10:53 AM
  • Hello! 

    I explored logs and find warnings: 

    0DA4	0FE4	10/14	06:29:02.418	22	VMWareServer.cs(1690)			NORMAL	Snapshot created for VM vm-108 with snapshot ref snapshot-701
    0DA4	0FE4	10/14	06:29:02.531	22	VMWareServer.cs(1989)			WARNING	Error in exclusion of disk with exception System.IO.FileNotFoundException: Could not find file 'C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\bin\excludedisk.xml'.
    0DA4	0FE4	10/14	06:29:02.531	22	VMWareServer.cs(1989)			WARNING	File name: 'C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\bin\excludedisk.xml'

    and 

    149C	1500	10/14	06:30:28.525	03	runtime.cpp(1392)	[00000000003AD740]	3CC28BF4-AC76-4ED8-B69C-605F057ADD4E	FATAL	<Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2137454160" Reason="Error" CommandID="RAGetWorkItemInfo" CommandInstanceID="f2677c21-fe9a-4166-99ea-6fa0296efa44" GuidWorkItem="1063d1a6-df8c-4719-a473-aa0c524e42d4" TETaskInstanceID="3cc28bf4-ac76-4ed8-b69c-605f057add4e"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="536872913" DetailedCode="-2137454160" DetailedSource="2"><Parameter Name="AgentTargetServer" Value="backup3.test.local"/></ErrorInfo></Status>


    Alex



    • Edited by ArhangeL87 Friday, October 14, 2016 7:04 AM
    Friday, October 14, 2016 6:59 AM
  • Reason my problem: 

    "VMware backup is not supported on 2008 R2 operating system though rest of the workload protection will work.

    VMware backup will work on Windows server 2012 and above server OSes" 



    Alex

    Monday, October 17, 2016 4:17 AM
  • This problem occurs on VMs that previously protected by Veeam Backup software. Although I have done all of the procedures of the following article, the problem exists.

    https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2139574

    P.S. After restarting DPM server, the problem resolved.


    Please VOTE as HELPFUL if the post helps you and remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.



    • Edited by R.Alikhani Sunday, December 11, 2016 6:20 AM
    Wednesday, December 7, 2016 7:10 AM
  • Hello same issue for me for a Win 2003 R2 VM (3 vmdk) backup with DPM 2012 R2.

    VM Snapshot OK but backup never successful with DPM (same with HP dataprotector). (Backup OK with Vsphere Data Protection)

    for DPM log :

    VMWareServer.cs(2176)   WARNING FMBlock: Unknwon exception caught!! Calling exception transformer now OperationCode:QueryChangeDiskAreas,

    I clone the VM, not better.

    I try R.Alikhani solution ==> unsuccessfull...

    I detach all disk (3 disk + SCSI controller  LSI logic parallel) and then reattach them with new scsi controller (same type).

    Now Backup is OK for the VM.

    Tuesday, December 27, 2016 1:13 PM
  • Hi Alex,

    what does it mean ? From Microsoft documentation I can see that is it possibile to backup Windows Server 2008 R2 VM in Vmware.

    Where did you find this information ?

    Thank's a lot.

    Luca

    Thursday, March 8, 2018 3:13 PM
  • For reference, I had a similar error when attempting to backup VMware VMs using the Microsoft Azure Backup Server (MABS) and a VMware VSAN setup...

    DPM encountered error from VMware Disk SubSystem. Server - xxxx ErrorCode - 14009: (ID 33620 Details: Internal error code 0x80990EF1)

    My problem was the MABS server could not resolve the ESXi host names, so ensuring DNS/host entries were set up resolved for me.

    Took me a while to figure out so thought I'd put it here for anyone else with the same issue!


    • Edited by Basty_ss Wednesday, November 21, 2018 8:30 AM
    Wednesday, November 21, 2018 8:29 AM