none
HyperV Cluster Replica Broker error - "object could not be found" from Enable-VMReplication Powershell RRS feed

  • Question

  • Ok I admit it. This one is driving me crazy ;)

    1) I have two Server2016 (Standard) HyperV Clusters - Source01 and DR01. Both part of same forest/domain.

    2) Both Source01 and DR01 clusters have a HyperV Replica Broker installed and replication settings configured for Kerberos, Port80 and allow from any server.

    3) I can initiate VM Replica successfully in both directions using the FailoverClusterManager UI. It creates the replica, transfers the initial data and everything appears to work.

    However....

    I need to create the replica using Powershell, but when I try to do it using the Enable-VMReplication powershell command it fails with:

    I must be doing something wrong as the UI works. Perhaps some remote permissions not enabled? Remote WMI? something? hopefully some rookie mistake...

    1) I've turned the firewall of DR01 off (just in case, just until figure out this issue)

    2) The hyperV virtual machine management service is running correctly on both Source01 and DR01

    3) the HyperV console on Source01 can correctly connect to DR01 and view/manage the VMs in the console - and vice versa.

    Anyone got any ideas? Or seen this behaviour before?

    thanks

    Joe.

    Friday, November 18, 2016 7:18 PM

All replies

  • the plot thickens...

    disabling the replica broker on DR01 (destination cluster), changing the port, or making some other breaking change results in exactly the same error! So it must not be even getting far enough to check its actually there??!

    I'm checking SPNs at this point and everything looks correct.

    As a sanity check I just pointed towards a standalone Server2016 host (so no cluster replica broker) and the enable-vmreplication etc worked first time. replication starts and works fine.

    repoint back to the destination cluster replica broker - back to the same error.

    I'm open to anything at this point :) No suggestion too stupid :)

    Cheers

    Friday, November 18, 2016 9:42 PM
  • the plot is now quicksand...

    Looks like a fairly massive bug in Server2016 atm. Anyone looking to utilise HyperV replica in Server2016 at the moment should probably hang tight for a little big. I've had a paid support case open with MS since Tuesday but all I'm getting is "we don't know... don't have documentation yet" :| Good job this isn't an RTM product that has been available for a couple of months already I guess.

    Basically none of the powershell commands related to interfacing with the a cluster replica broker seem to be able to talk to it. Remove the cluster replica broker and talk directly to a standalone host and everything works fine. I've also tested the same commands on a set of Server2012R2 hyperV cluster and that works fine as well.

    To double check everything I've now got three Server2016 environments, each with a source and destination cluster - all three are failing with the exact same error.

    long shot, but has *anyone* managed to get the cluster replica broker working with the powershell commands?

    Cheers

    J.

    Sunday, November 20, 2016 5:04 PM
  • Hi Joe,

    I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.

    Best Regards,
    Leo


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Monday, November 21, 2016 2:57 AM
    Moderator
  • Thanks Leo :)

    I can post/pm the support case number if that helps.

    Regards

    Joe

    Monday, November 21, 2016 11:19 AM
  • still no news from the product team. I'm just in limbo here :|
    Monday, November 28, 2016 9:58 AM
  • No, man. I am also by ur side in limbo...

    I am trying to invoke powershell command between clustered replica broker and Hyper-V server 2016, and get the same error. I even try replica broken and MS server 2016 too, and all the windows version is related to 14393, which I think is a very new one. Both not working. So do you have any update?

    Wednesday, May 17, 2017 2:29 AM
  • No Outcome?
    Monday, September 4, 2017 8:33 PM
  • I'm having the exact same problem, anyone have an update?
    Monday, October 30, 2017 2:18 PM
  • Having the exact same issue!

    trying to automate the initial replication (enable + start replication) on fresh VM's in our DC (using orchestrator)
    but this makes it impossible to do, so..enabling one by one manually atm :(

    hope this gets fixed soon! 

    Wednesday, November 29, 2017 12:59 PM
  • Bump - same problem here.  Anyone got a fix for this?

    I do it via the GUI, works fine.  Do it PS, fails.

    enable-VMReplication : Hyper-V encountered an error trying to access an object on computer 'replica_broker' because the object was not found. The object might have been deleted.
    Verify that the Virtual Machine Management service on the computer is running.

    I have 100's of VM's to replicate, doing by hand would be extremely ANNOYING!

    Thursday, February 22, 2018 8:11 PM
  • Nobody?
    Saturday, February 24, 2018 3:49 PM
  • Opened a case with Microsoft at the time of my previous post
    MS engineer confirmed our findings and that it is marked to be fixed in a future release...
    so no, the only way is to do every VM manually, one at a time :( which is the suggested so-called "workaround" told to us by the MS Engineer
    from my perspective this is an unacceptable solution/workaround, it is a hell of a job to enable replication for hundreds of VM's as you can imagine.

    :(

    Tuesday, February 27, 2018 2:23 PM
  • Same problem here. Infuriating, as that's the only way to enable Hyper-V Replica's with VHD Sets.
    Tuesday, February 27, 2018 11:45 PM
  • I have opened a ticket with Microsoft as manual replication is not an option.  Will update with findings.
    Wednesday, February 28, 2018 4:25 PM
  • Thank you, this has been driving me crazy (thought it was my scripting!)

    If shortcuts worked, then they wouldn't be shortcuts would they...

    Wednesday, February 28, 2018 7:27 PM
  • Thank you for sharing with us this information. If only microsoft would think to inform about known issues with it's products.

    And are they kidding me with this ?! 1,5 year from release and still not fixed... Just painfully learned about this limitation.
    Server 2012 r2 is end of mainstream support in 7 months and 2016 is still not ready for use...
    First you can't take a checkpoint of new VHD Sets (vhds). Still not fixed from release. So no checkpoints and no backups for vhds.
    Then promised VHD Set replication. Except it has a lot of asterisks and apparently can't be configured with gui (that's fine) but also not from powershell. And not documented at all, so not really working...
    And let's not forget S2D named confusingly "storage spaces" - a new thing completely separate from previous storage space, but let's confuse people and make them think that old storage space will also get S2D new shiny features...

    Anyway back to point.
    Any idea what they meant by "Future release" ? A hotfix or will we have to wait for server 1803, 1809 or whatever next long term release will be called ?

    edit: 2018-03-01

    found another "workaround". I was trying Enable-VMReplication from 2016 hyper-v standalone to a 2016 cluster and failed missurebly.

    However when I ran Enable-VMReplication on 2012 R2 host with " -ComputerName $srcserver  -ReplicaServerName $dstserver" for the same combination of 2016 hosts then it worked.

    So just configure replication of hyper-v 2016 hosts from 2012 R2 management machine.

    Hope you all still have 2012 r2 available somewhere :)


    • Edited by Andrzej__P Thursday, March 1, 2018 9:45 PM froud workaround
    Thursday, March 1, 2018 9:32 PM
  • Before invoking any replication commands, as a workaround on 2016 Server, you can use:

    Remove-Module Hyper-V

    Import-Module -Name Hyper-V -RequiredVersion 1.1

    Best Regards,

    Leon.

    • Proposed as answer by Andrzej__P Monday, March 12, 2018 12:12 PM
    Sunday, March 4, 2018 11:31 AM
  • I will have to try this from a 2012 server, I'm not prepared to run hyper-v module reinstall on a production cluster just yet.  

    Monday, March 12, 2018 11:51 AM
  • Hi, I can confirm that using Hyper-V module in version 1.1 works fine for replication - I set up replication from single host to cluster.

    You do not need to reinstall module. It's loaded only for current powershell session, and version 1.1 is apparently also avalible in 2016 as I had it already avalible.

    Check with:

    get-module -Name "hyper-v" -ListAvailable

    Monday, March 12, 2018 12:11 PM
  • I still have MS support working on this.  2 weeks now, they will still not accept its a bug.

    :-(

    When I load the PS module from 1.1, I get this.  This is something I AM NOT doing on a PRODUCTION cluster!

    PS C:\Windows\system32> install-module hyper-v -RequiredVersion 1.1
    NuGet provider is required to continue
    PowerShellGet requires NuGet provider version '2.8.5.201' or newer to interact with NuGet-based repositories. The NuGet
     provider must be available in 'C:\Program Files\PackageManagement\ProviderAssemblies' or
    'C:\Users\richard.smith\AppData\Local\PackageManagement\ProviderAssemblies'. You can also install the NuGet provider by
     running 'Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force'. Do you want PowerShellGet to install
    and import the NuGet provider now?
    [Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"):

    Tuesday, March 20, 2018 11:35 PM
  • no wonder, you are installing it, not loading to current session - there is a major difference.

    You should:

    Remove-Module Hyper-V
    Import-Module -Name Hyper-V -RequiredVersion 1.1

    Import - not install.

    Import is different then installing. If you have doubts read help for both to know what they do.

    Wednesday, March 21, 2018 10:50 AM
  • I have a MS premiere ticket open for this issue since March 20th.  Nearly 3 months of investigation and running around in circles with MS still no progress.

    They have confirmed its a bug though just trying to get it fixed.

    Tuesday, June 12, 2018 8:22 AM
  • Finally, MS <g class="gr_ gr_15 gr-alert gr_gramm gr_inline_cards gr_run_anim Grammar multiReplace" data-gr-id="15" id="15">have</g> been able to reproduce the issue internally.  Awaiting a fix!

    Yay!  Wonder which cumulative update it will appear in!  2018-07? 

    Monday, July 2, 2018 8:01 AM
  • Hello,

     I just want to add that we have same issue on Windows Server 2016 Standart in Hyper-v cluster.

    We need to use Resume-VMReplication cmdlet and it fails:

    resume-VMReplication : Hyper-V encountered an error trying to access an object on computer 'server01' because
    the object was not found. The object might have been deleted, or you might not have permission to perform the task.
    Verify that the Virtual Machine Management service on the computer is running. If the service is running, try to
    perform the task again by using Run as Administrator.
    At line:1 char:1
    + resume-VMReplication -VMName virtualMachine01
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : ObjectNotFound: (:) [Resume-VMReplication], VirtualizationException
        + FullyQualifiedErrorId : ObjectNotFound,Microsoft.HyperV.PowerShell.Commands.ResumeVMReplication
    

    Microsoft support suggested to rebuild WMI:

    Open CMD as administrator:

    • mofcomp  %systemroot%\system32\WindowsVirtualization.v2.mof
    • mofcomp %systemroot%\system32\en-US\WindowsVirtualization.v2.mfl

    But that do not worked for me.

    Thursday, July 19, 2018 8:14 AM
  • Hi, we are getting the same thing, no matter if we user PowerShell or the Cluster GUI. Nothing to add to help, just wanted to say the same is happening to us too.

    Phil


    Phil

    Tuesday, September 11, 2018 6:44 AM
  • I have the same issue at the moment but with WS 2019.  I have a ticket open with MS and (as some of you have also experienced) they have acknowledged that it's a bug.  They suggested using module version workaround but it made no difference and fails with the same error.

    Oddly, and I'm not sure if any of you guys have noticed the next bit.  Replication fails for all *new* VMs and VMs replicated from another server/cluster.  But if I replicate away from the new cluster, fail over and reverse replication, it works without issue.

    When replicating a VM for the first time, it creates the VM on the new cluster (so cert auth isn't to blame), but it's the initial seeding that fails.  That had me thinking it was an issue with storage or permissions, but it does create the new VHD also.

    Then I tried sending the initial replication of a brand new VM using external media, but it failed with the exact same error.  So it looks like the failure occurs at source.

    Wednesday, January 23, 2019 4:11 PM
  • Incidentally, I did get the above wizard to finish on a subsequent attempt and it exported the replication data to the D drive as show.  I copied that to the cluster and it was accepted on that side.  Great, thinks looked good and the 'last replication' time was up to date.

    Unfortunately, each subsequent replication attempt for this VM failed with the same error.

    > Hyper-V could not find the virtual machine 'TestVMFromRep' on the Hyper-V Replica Broker. (Virtual machine ID 4FEB777E-8D89-4DBC-B58C-ACDB77A5D3EF)

    Wednesday, January 23, 2019 4:32 PM
  • Hi all,

    same issues here with ws2019 17744 insider build.

    anybody know if this error occurs also in the stable version of ws2019?

    Workaround with the 1.1 module does not work.

    GUI does not work.

    regards

    Monday, February 4, 2019 9:44 AM
  • Mine was on the stable build.  I was asked to fill in a business impact questionnaire in order to help push for a fix from the development team.  The more users / bigger the impact, the sooner it will be resolved I suppose.
    Tuesday, February 5, 2019 2:15 PM
  • Hi JoeGough

    Please try the following procedure:

    On primary server run the following commands to reinstall the Hyper-V module.

    Remove-Module Hyper-V
    Import-Module -Name Hyper-V -RequiredVersion 1.1

    There is a error on Hyper-V module that prevents the command to work accordingly. I've tested the command in lab and production environment and its works successfully!

    Thanks, 


    Friday, February 22, 2019 7:20 PM
  • Hi,

    has anyone some News with the error, i have the same with two 2019er Cluster.

    The Workaround with the hyper-v modules 1.1 does not work for me.

    Thanks, Heinrich

    Tuesday, March 19, 2019 11:04 AM
  • Same with Win2019:

    > Hyper-V could not find the virtual machine 'TestVMFromRep' on the Hyper-V Replica Broker. (Virtual machine ID 4FEB777E-8D89-4DBC-B58C-ACDB77A5D3EF)

    Wednesday, March 20, 2019 8:53 AM
  • Confirm the same issue with Win2019 Cluster.

    1. Win2019 Cluster CAN'T receive inbound replication, but can send replica from Hyper-V Replica Broker to any standalone Hyper-V server.


    2. Cluster to cluster replication DOESN'T WORK.

    As Keiran Kenny wrote, I get the same error:

    > Hyper-V could not find the virtual machine 'TestVMFromRep' on the Hyper-V Replica Broker. (Virtual machine ID 4FEB777E-8D89-4DBC-B58C-ACDB77A5D3EF)


    Wednesday, March 20, 2019 8:54 AM
  • this workaround doesnt work for me. @Bruno H. Secchi

    i got the same problem just like Aram Dzhulfalakyan & Heinrich Klassen

    Outgoing Replication to a Hyper-V Host does work from the 2019 Cluster.

    Old Cluster Server 2012R2 trying to Replicate to the New Cluster Server 2019.

    The VM is getting created on the Cluster.  The VHD is also created with the standard 4096KB Size..

    VM-ID is correct,  everything seems fine.  but the primary replicationserver tells me:

    ErrorCode: 32370

    The VM was not found on the Hyper-V-Replicabroker.

    Any new Idea?? 





    Wednesday, March 20, 2019 12:53 PM
  • Not a productive answer, but for us we had to buy a 3rd party replication tool to get over this. We were moving from an old cluster to a new cluster - with new hosts/san etc. I think this is 100% crazy that it is 100% reproducible yet MS do nothing to fix this. 

    Phil

    Wednesday, March 20, 2019 1:21 PM
  • replying to this thread as I'm also having this issue on a client site that setup up a new 2019 cluster to replace there aging / failing older server 2012R2 cluster
    • Edited by DubG83 Wednesday, March 20, 2019 9:18 PM
    Wednesday, March 20, 2019 9:17 PM
  • I know it doesn't really help anyone but I have the exact same issue since upgrading from 2012 R2 to a 2019 cluster. Tried everything but always get a 'Hyper V could not find the virtual machine on the replica server' It creates the VM but never starts the initial replication
    Tuesday, March 26, 2019 9:03 AM
  • Right So while we all wait to see if MS comes up with a fix for this issues, maybe their is one and it just hasn't made it to this forum, I did find a work around that seems to be working for me. I have been able to get 2 VM's moved over so far. 

    In my case I'm just trying to move VM's off old servers to new hardware running 2019, So this may not work for everyone. 

    Veeam has a free version of it's Backup and replication suite. It does allow you to do replication and permanent fail over. tho it takes a few steps.  you can find out more / download this version at the link below. 

    https://www.veeam.com/virtual-machine-backup-solution-free.html

    Like I said, this may not work for everyone, It works for me, I was able to move 2 VM's over with out much issue. 


    • Edited by DubG83 Wednesday, March 27, 2019 1:24 PM
    Tuesday, March 26, 2019 10:20 PM
  • With your ticket with Microsoft did they ever get back to you on this. Or give you any ETA on when they would have it fixed?
    Thursday, April 4, 2019 8:12 AM
  • This may sound silly . . .but make sure both the source and destination VMs are in the Root of the CSV or Storage you are using.   We have had this issue after our upgrade and this one change made all of the replication work.  Is this awesome . . not at all . . but working is better than having a structure.

    To expand a little.  Once your initial Replication fails . . move the vm on the destination to a root level folder, create it and move the new copy.  Then restart the replication and it should just go.

    • Edited by Richard L Whitson Friday, April 19, 2019 10:24 PM forgot information.
    • Proposed as answer by Criztof Wednesday, April 24, 2019 2:10 PM
    Friday, April 19, 2019 10:09 PM
  • Wow thanks that does fix it. Definitely not ideal but it is a work around! Thanks
    Wednesday, April 24, 2019 2:10 PM
  • Hi Richard,

    i've the same Problem with the replication between two Hyper-V 2019 Cluster.

    i've try your solution with move the vm ( i use the function in hyper-v move Storage ) to the root of CSV,

    but i've the same issue.

    What do i wrong or could you tell me the steps what i must do.

    Thanks, Heinrich

    Thursday, May 2, 2019 2:27 PM
  • We have a VM scattered across multiple volumes due to other limitations.  Does this solution require the whole VM to be on one volume in the root?  I have tried just moving the machine config and this doesn't work.  Thanks in advance...

    • Edited by Jamie3826 Sunday, June 2, 2019 6:31 PM
    Sunday, June 2, 2019 6:30 PM
  • Related thread https://social.technet.microsoft.com/Forums/en-US/80153dc2-9051-4d0d-adf6-ad11c14887ca/hyperv-2019-vm-cluster-replication?forum=winserverhyperv

    A fix for this issue is planned to release in September 2019 update package for Windows Server 2016.

    Thursday, July 4, 2019 9:40 PM