none
HostAgentFail (2912); HR: 0x800706BE RRS feed

  • Question

  • Hi all

    I’ve been experiencing a strange issue with SCVMM and after countless attempts at fixing it, I'm coming here asking for help.

    I’ve recently deployed an install of SCVMM current branch 1807 in our environment (currently upgraded to 2019) and I’ve been attempting to configure a logical switch on a new temporary hypervisor based on Server 2016. Having encountered loads of SCVMM service crashes, I found that a host agent update was pending so I went ahead and installed it for that host which indeed allowed me to configure the logical switch and the virtual network adapter for management. Thinking all is well, I kicked off our migration script to continue migrating from our old ESXi environment however as soon as the V2V operation in SCVMM starts, the host agent service crashes with the following error:


    Faulting application name: vmmAgent.exe, version: 4.1.3403.0, time stamp: 0x5b33b02b
    Faulting module name: ntdll.dll, version: 10.0.14393.2608, time stamp: 0x5bd133d4
    Exception code: 0xc0000374
    Fault offset: 0x00000000000f7b43
    Faulting process id: 0x16cc
    Faulting application start time: 0x01d4bed3ed6bffc4
    Faulting application path: C:\Program Files\Microsoft System Center\Virtual Machine Manager\bin\vmmAgent.exe
    Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
    Report Id: 3ebd0a7c-46e8-4223-9249-0f8fad13103f
    Faulting package full name: 
    Faulting package-relative application ID:

    This didn’t happen before and I’ve been able to migrate a lot of VMs before, it then suddenly became an issue affecting all hypervisors. I initially thought it was affecting the new hypervisor but then realised that it is the same case in relation to the old hypervisor as well. In fact, it happens at any migration attempt. I’ve not been able to find much about this issue online, the error code is very specific. I’ve seen hints relating to bits and the bits ssl port but none of those tweaks help here.
    If nobody can shed any light at this, I believe the only way will be to raise a support case.

    Thanks for any ideas

    The SCVMM cmdlet returns (I've hidden some environment specific names):
    13-02-2019 14:08:26 vmname (3) Microsoft.VirtualManager.Utils.CarmineException: An internal error has occurred
    trying to contact the 'hv09.domain.local' server: : .


    WinRM: URL: [http://hv09.domain.local:5985], Verb: [INVOKE], Method: [CreateHttpPostClientJob], Resource: [http://sc
    hemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/HttpPostDeploymentJob]


    Check that WS-Management service is installed and running on server 'hv09.domain.local'. For more information use th
    e command "winrm helpmsg hresult". If 'hv09.domain.local' is a host/library/update server or a PXE server role then
    ensure that VMM agent is installed and running. Refer to http://support.microsoft.com/kb/2742275 for more details.
       at Microsoft.SystemCenter.VirtualMachineManager.TaskHelper.ThrowOnFailure()
       at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.TaskCmdlet`1.WaitForResult(Task task)
       at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.TaskCmdletBase`1.Process()
       at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.CmdletBase.ProcessRecord()
    *** Carmine error was: HostAgentFail (2912); HR: 0x800706BE
    *** xx-esxi-10.domain.local ** 443 ** /folder/vmname /vmname.vmdk?dsName=xx-esxi-10:localdisk1:10kS
    AS ** WinRM: URL: [http://hv09.domain.local:5985], Verb: [INVOKE], Method: [CreateHttpPostClientJob], Resource: [htt
    p://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/HttpPostDeploymentJob] ** hv09.domain.local **  **  **
    [s#1] DnsHelper.GetDomainName(dnsName: (string) "mgmt-scvmm.Domain.local", parseOption: DnsParseOption.RequireFQDN)
    in s:\btvsts\6723\private\product\common\Utils\DnsHelper_pp.cs:line 219
    Full call stack from when the exception was thrown:   at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.TaskCmdlet
    Base`1.Process()
       at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.CmdletBase.ProcessRecord()
       at System.Management.Automation.CommandProcessor.ProcessRecord()
       at System.Management.Automation.CommandProcessorBase.DoExecute()
       at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
       at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInterna
    l[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcCo
    ntext)
       at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
       at System.Management.Automation.PSScriptCmdlet.RunClause(Action`1 clause, Object dollarUnderbar, Object inputToProces
    s)
       at System.Management.Automation.PSScriptCmdlet.DoEndProcessing()
       at System.Management.Automation.CommandProcessorBase.Complete()
       at System.Management.Automation.CommandProcessorBase.DoComplete()
       at System.Management.Automation.Internal.PipelineProcessor.DoCompleteCore(CommandProcessorBase commandRequestingUpstr
    eamCommandsToStop)
       at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
       at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInterna
    l[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcCo
    ntext)
       at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
       at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
       at System.Management.Automation.DlrScriptCommandProcessor.RunClause(Action`1 clause, Object dollarUnderbar, Object in
    putToProcess)
       at System.Management.Automation.DlrScriptCommandProcessor.Complete()
       at System.Management.Automation.CommandProcessorBase.DoComplete()
       at System.Management.Automation.Internal.PipelineProcessor.DoCompleteCore(CommandProcessorBase commandRequestingUpstr
    eamCommandsToStop)
       at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
       at System.Management.Automation.Runspaces.LocalPipeline.InvokeHelper()
       at System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()
       at System.Management.Automation.Runspaces.LocalPipeline.StartPipelineExecution()
       at System.Management.Automation.Runspaces.PipelineBase.CoreInvoke(IEnumerable input, Boolean syncCall)
       at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)
       at System.Management.Automation.PowerShell.Worker.RunspaceAvailableCallback(IAsyncResult asyncResult)
       at System.Management.Automation.Runspaces.AsyncResult.SetAsCompleted(Exception exception)
       at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.ServicePendingRequests(Object useCallingThrea
    dState)
       at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.EnqueueCheckAndStartRequestServicingThread(Ge
    tRunspaceAsyncResult requestToEnqueue, Boolean useCallingThread)
       at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.BeginGetRunspace(AsyncCallback callback, Obje
    ct state)
       at System.Management.Automation.PowerShell.CoreInvokeAsync[TInput,TOutput](PSDataCollection`1 input, PSDataCollection
    `1 output, PSInvocationSettings settings, AsyncCallback callback, Object state, PSDataCollection`1 asyncResultOutput)
       at System.Management.Automation.PowerShell.BeginInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 o
    utput, PSInvocationSettings settings, AsyncCallback callback, Object state)
       at System.Management.Automation.ServerRunspacePoolDriver.HandleCreateAndInvokePowerShell(Object sender, RemoteDataEve
    ntArgs`1 eventArgs)
       at System.Management.Automation.ExtensionMethods.SafeInvoke[T](EventHandler`1 eventHandler, Object sender, T eventArg
    s)
       at System.Management.Automation.Remoting.ServerRemoteSessionDSHandlerStateMachine.DoMessageReceived(Object sender, Re
    moteSessionStateMachineEventArgs fsmEventArg)
       at System.Management.Automation.ExtensionMethods.SafeInvoke[T](EventHandler`1 eventHandler, Object sender, T eventArg
    s)
       at System.Management.Automation.Remoting.BaseTransportManager.OnDataAvailableCallback(RemoteDataObject`1 remoteObject
    )
       at System.Management.Automation.Remoting.ReceiveDataCollection.ProcessRawData(Byte[] data, OnDataAvailableCallback ca
    llback)
       at System.Management.Automation.Remoting.PriorityReceiveDataCollection.ProcessRawData(Byte[] data, DataPriorityType p
    riorityType, OnDataAvailableCallback callback)
       at System.Management.Automation.Remoting.BaseTransportManager.ProcessRawData(Byte[] data, String stream)
       at System.Management.Automation.Remoting.Server.OutOfProcessServerTransportManager.ProcessRawData(Byte[] data, String
    stream)
       at System.Management.Automation.Remoting.Server.OutOfProcessMediatorBase.OnDataPacketReceived(Byte[] rawData, String
    stream, Guid psGuid)
       at System.Management.Automation.Remoting.OutOfProcessUtils.ProcessElement(XmlReader xmlReader, DataProcessingDelegate
    s callbacks)
       at System.Management.Automation.Remoting.OutOfProcessUtils.ProcessData(String data, DataProcessingDelegates callbacks
    )
       at System.Management.Automation.Remoting.Server.OutOfProcessMediatorBase.ProcessingThreadStart(Object state)
       at System.Management.Automation.Utils.WorkItemCallback(Object callBackArgs)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object
    state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, B
    oolean preserveSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()

    Thursday, April 18, 2019 9:41 AM

All replies

  • Hello,

    Firstly, could you please check the status of the Hyper-V host and VMWare Server, and make sure the status is OK?

    Then, instead of using script, have you also tried to covert the VM using the wizard? You can refer to the following article for the steps.

    https://docs.microsoft.com/en-us/system-center/vmm/vm-convert-vmware?view=sc-vmm-2019

    Best regards,

    Andy Liu


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Friday, April 19, 2019 3:14 AM
  • Hi Andy

    Thanks for your response. I've checked the status of the 3 hyper-v hosts and also rebooted the ESXi hosts along with the vCenter to no avail - migrations still fail. It's annoying as I've already migrated around 80 vms and I have roughly the same left, leaving us stuck in this project.

    I'd like to get some more detailed logging from the scvmm agent because that's where the crash happens. The service just stops working. I upgraded scvmm from 1801 tho 1807 and now 2019 and the issue persists - this also happens when using the SCVMM conversion wizard.

    Looking forward to your assistance.

    Thanks

    Wednesday, April 24, 2019 8:05 AM