none
Hyper-V Server 2016 Replication Issues, stuck on Creating a reference point

    Pertanyaan

  • Hi all,

    Config: 2 Node Cluster, with Windows SMB 3 Storage for VM's

    I took the plunge, and migrated from Hyper-V Server 2012 R2 to 2016, and have had some fun and games on the way...  I've now got one major issue, and was hoping someone would shed some light on the issue:

    Enable replication for a VM, whilst it is powered off.  Replication completes successfully.  When I power up the VM, changes are detected and replication kicks off to copy the changes to the replica.  At the same time there is a new "Creating a reference point" part which starts off, and this is where the issue is.... it is getting stuck there at around 50%.  I can't shut the VM down, power off doesn't work, can't kill the process, or shut the Host down, apart from a Physical power off.

    Host Event Log Entry that ties in with the issue is:
    'VMTEST01' timed out while waiting to perform the 'Cleaning up stale reference point(s)' operation. The virtual machine is currently performing the following operation: 'Creating a reference point'

    I have tried a few different VM's, cleanly built, and they have the same issue.

    The exact same setup on 2012 R2 worked perfectly.

    Any ideas?


    01 Nopember 2016 21:16

Jawaban

  • OK...

    After spending over a week, building, re-building, testing a complete new cluster, i've managed to resolve this myself.

    I was going to log a call with Product Support today, but I thought I'd investigate my environment further...  I built a host with the computer account in a Blocked GPO OU, to eliminate GPO's, and funny enough, Replication worked perfectly.  After hours of investigation I narrowed it down to just 1 setting...  The Disable Windows Defender setting.

    This setting works fine for all other Windows 2016 and Windows 10 machines, but with Hyper-V it causes this weird issue.

    10 Nopember 2016 19:25

Semua Balasan

  • Hi Bharat,

    >>'VMTEST01' timed out while waiting to perform the 'Cleaning up stale reference point(s)' operation

    I'm afraid I could not find related information about this issue.

    Is it happening on all VMs?

    Best Regards,

    Leo


    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    02 Nopember 2016 6:32
    Moderator
  • Hi Leo,

    I guess as 2016 is new, this may not have been picked up by anyone.

    This is happening to all VM's, both Windows & Linux VM's.

    I believe the Reference Creation is a new thing with 2016.

    Regards,

    Bharat

    02 Nopember 2016 11:55
  • Update from further testing:

    VM sitting on the Host's Local Disk, replicates fine, and I can power it up, with no issues.  I cannot see the "Creating a reference point" mentioned anywhere in the Event Log.

    When I move the machine storage back to the Windows 2016 SMB Share, I see the issue again.

    When I move the machine storage to another Windows 2016 Server's SMB Share, I still see the issue.

    Possibly an SMB issue?



    02 Nopember 2016 14:41
  • Hi Bharat,

    Probably the feature is not working with VMs located on SMB share.

    I checked the official related documents, but it seems it is not stated as supported or not supported.

    https://technet.microsoft.com/en-us/windows-server-docs/compute/hyper-v/what-s-new-in-hyper-v-on-windows

    https://technet.microsoft.com/en-us/windows-server-docs/compute/hyper-v/manage/set-up-hyper-v-replica

    Best Regards,

    Leo


    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    03 Nopember 2016 7:34
    Moderator
  • OK...

    After spending over a week, building, re-building, testing a complete new cluster, i've managed to resolve this myself.

    I was going to log a call with Product Support today, but I thought I'd investigate my environment further...  I built a host with the computer account in a Blocked GPO OU, to eliminate GPO's, and funny enough, Replication worked perfectly.  After hours of investigation I narrowed it down to just 1 setting...  The Disable Windows Defender setting.

    This setting works fine for all other Windows 2016 and Windows 10 machines, but with Hyper-V it causes this weird issue.

    10 Nopember 2016 19:25
  • Thanks for posting this. We ran into this issue in our environment and installing Windows Defender fixed the replication but not an acceptable solution for us. We use Trend for antivirus so installing it will disable Windows Defender. Could this be a bug with Windows 2016? I did not check local drive vs SMB shares thing but I was able to reproduce this issue consistently. Here's the summary of tests I did.

    1. Windows 2016 Datacenter with latest updates on source and destination with Trend Antivirus installed and NO Windows Defender - Replication breaks after initial replication and causes Hyper-V Management service to go unresponsive.

    2. Windows 2016 Datacenter with latest updates on source and destination with NO Trend and NO Windows Defender - Replication breaks after initial replication and causes Hyper-V Management service to go unresponsive.

    3. Windows 2016 Datacenter with latest updates on source and destination with Widows Defender and NO Trend - Replication works normally.

    4. Windows 2016 Datacenter with latest updates on source and destination with Widows Defender and Trend Installed (Trend Disables Windows Defender) - Replication breaks after initial replication and causes Hyper-V Management service to go unresponsive.

    Anyone else seen this issue and is there a fix other than installing Windows Defender?

    24 April 2017 16:46
  • Hey RC_P,

    Good to know someone else is having a similar issue, obviously not good for you :(

    I believe this is a bug with 2016 and Replication relying on Defender to be present (Microsoft Support would not acknowledge this, even after numerous testing on my side).

    I spent weeks with support, and got very frustrated by them repeating the tests I had performed, and having to explain the whole issue from scratch at each escalation.

    They kept insisting that SOFS is required for SMB Shares to function correctly, and that they behave differently if you are using a Standalone File Server (which is nonsense IMHO).

    I did not mind being their guinea pig, but as it was taking too much of my time, I gave up in the end, and have now moved to using iSCSI with CSV.  Working perfectly with no issues.

    Feel free to log it with them again, and use my name as another customer, so they can look up the case details, and hopefully escalate it further up.

    Good Luck !!!

    24 April 2017 17:51
  • Hey RC_P,

    Good to know someone else is having a similar issue, obviously not good for you :(

    I believe this is a bug with 2016 and Replication relying on Defender to be present (Microsoft Support would not acknowledge this, even after numerous testing on my side).

    I spent weeks with support, and got very frustrated by them repeating the tests I had performed, and having to explain the whole issue from scratch at each escalation.

    They kept insisting that SOFS is required for SMB Shares to function correctly, and that they behave differently if you are using a Standalone File Server (which is nonsense IMHO).

    I did not mind being their guinea pig, but as it was taking too much of my time, I gave up in the end, and have now moved to using iSCSI with CSV.  Working perfectly with no issues.

    Feel free to log it with them again, and use my name as another customer, so they can look up the case details, and hopefully escalate it further up.

    Good Luck !!!

    FYI, I've emailed support, requesting they review this post, maybe they'll look into it, maybe not ?
    24 April 2017 17:58
  • Thanks Bharat! Hope they'll look into this.
    24 April 2017 19:12
  • So do disable it or don't for the HyperV Machines?

    07 Maret 2018 20:05
  • Been a while since I posted it...  Maybe they've fixed it ???

    MS Support case was rasied, but they were useless, kept insisting that I build a Scale Out File Cluster - completely irrelevant.

    I wasted enough time on this, so I ended up moving to CSV Volumes, which don't have this issue.

    To answer your question, Do not disable it (unless if has been fixed).  Leave it enabled, but disable other things like real-time protection, etc.  Or just leave it enabled, and the default rules should exclude the processes.

    07 Maret 2018 20:19
  • After adding a 2016 Hyper-V Host to the 2012 R2 Cluster I got the same problem when the replica destinations are on the new 2016 host.

    When I move them back to the 2012 R2 they apply without issues.

    Storage is CSV on iSCSI.

    VMs are running fine on the 2016 host, backup with DPM works fine too.

    Defender is installed on both Hosts and not disabled or anything.

    Just opened a case with Microsoft but no solution yet.

    When the replicas are stuck at "applying replica changes" or "apply registered delta" I see a higher disk queue on the volume in the monitoring of the storage array without seeing any high I/O rate.

    16 Mei 2018 9:30