none
2012 R2 Hyper-V VM Disk 2 has been surprise removed | An error was detected on device \Device\Harddisk2\DR2 during a paging operation RRS feed

  • Question

  • I have been scouring the internet for over a week now, and I cannot figure this one out. Some background... I am retiring an old physical file share server, and I am creating a new 2012 R2 VM on a 2012R2 DataCenter Core Hyper-V host. Creating the VM went without issue. It is configured as follows:

    8GB RAM

    4 CPU Cores

    72GB system drive (C:\)

    2TB data drive (D:\)

    12GB page file drive (Z:\)

    I have about 1.8TB of data to copy over from a physical box, so I am using RoboCopy. I have gotten about 1.5TB of data over, then the new VM's data drive (D:\) disappears and the copy fails. The disk no longer appears in Windows Explorer or diskmgmt.msc. The even log shows many event IDs 153 (The IO operation at logical block address 0x*** for Disk 2 was retried) and event IDs 51 (An error was detected on device \Device\Harddisk2\DR2 during a paging operation, and the last event when the drive disappears is a single Event ID 157 (Disk 2 has been surprise removed).

    After racking my head over this for several days, I deleted the VHDX for the 2TB data drive, created a new one, and started the data copy all over... same issue. I completely deleted the VM and all files, reinstalled on new VHDX drives... same issue.

    This leads me to think it is an issue with the Hyper-V host, but there are no event IDs pertaining to this, and the other 53VMs are running without issue.

    I would SINCERELY appreciate any help with this.

    Sorry for huge post, but thank you in advance!


    • Edited by Mako-Wish Tuesday, July 22, 2014 3:30 PM
    Monday, July 21, 2014 11:28 PM

All replies

  •   What did you search for? "disk has been surprise removed" gives 114,000 hits on Bing!

      This is as good an explanation of the error message as any I have seen.

    http://ppe.blogs.msdn.com/b/ntdebugging/archive/2013/12/27/event-id-157-quot-disk-has-been-surprise-removed-quot.aspx

      I doubt that it has anything to do with the fact that you are using a virtual machine. It is more likely to be a disk error in the source file. 


    Bill

    Tuesday, July 22, 2014 2:15 AM
  • I go along with Bill.

    I would bet that the issue lies at your storage layer, not the hypervisor.

    Something is causing the disk / LUN to go offline and thus the virtual disk 'disappears' from your VM.


    Brian Ehlert
    http://ITProctology.blogspot.com
    Learn. Apply. Repeat.

    Tuesday, July 22, 2014 2:51 PM
    Moderator
  • Thank you for the replies. There is definitely no lack of search hits for the event ID, but none of them have pointed me in the right direction. 99% of them refer to USB flash or external drives, or USB cameras. The server is an HP G7 using RAID 50 DAS. I have checked the disks for errors (none found), and the strange thing like I was saying, no other VMs are having the issue.

    Thank you for the link, but I have already seen that page, and though informative, it doesn't help me resolve the issue.

    "These errors are most often caused when something disrupts the system's communication with a disk, such as a SAN fabric error or a SCSI bus problem.  The errors can also be caused by a disk that fails, or when a user unplugs a disk while the system is running.  An administrator that sees these errors needs to verify the heath of the disk subsystem."

    What is disrupting the communication? If it is a SCSI issue, why is this the only VM experiencing it? If a disk had failed, the information is redundant and I would just slap a new drive in. And a user is definitely not unplugging any disks.

    Again, thank you both for the replies, but do you have any more ideas on what I should be looking at to figure this out?

    Thank you!

    Tuesday, July 22, 2014 3:30 PM
  • I doubt that it has anything to do with the fact that you are using a virtual machine. It is more likely to be a disk error in the source file.

    That was my first thought, but I deleted that file and created a new one. Ran into the same issue. I then completely deleted the VM and drive files, then started from scratch. Still ran into the same issue.
    Tuesday, July 22, 2014 3:32 PM
  • I am not going to get too excited yet, but I converted the VHDX from thin- to thick-provisioned, and so far so good. This was just going to be a data share, so I wasn't too concerned with a performance hit with thin-provisioning, but I thought perhaps it had something to do with the error. I have copied about 200GB onto it so far this morning. I wanted to post an update, but I am still crossing my fingers at this point....
    • Edited by Mako-Wish Tuesday, July 22, 2014 4:09 PM
    Tuesday, July 22, 2014 4:07 PM
  • There may be some interaction with the dynamic virtual disk block size and the block size / alignment of the array.

    And it is the act of copying the files over that is causing multiple disk expansion actions that are hitting some magical point in which the two cross or are massively out of alignment with each other.

    Not that it is a VM, nor that it is a dynamically expanding disk by themselves, but it is the particular loading of the file copy, the virtual disk expansion, and the default array options that are magically creating the situation.

    Did you ever search through HP's knowledge base or support?


    Brian Ehlert
    http://ITProctology.blogspot.com
    Learn. Apply. Repeat.

    Tuesday, July 22, 2014 4:20 PM
    Moderator
  • I did not search HPs website. The more I think about it, your explanation does make sense. No matter how many times I rebuilt the VM and/or the VHDX, the issue happened right about the same time (about 1.2TB VHDX size). It was like a barrier I could not get past. Now that it is thick-provisioned, I have about 1.5TB on it so far.
    Tuesday, July 22, 2014 4:30 PM
  • Hi MotoCrazy,

    Maybe you can try to use command to create a vhd with larger blocksize then copy file again for test  :

    New-VHD -Path d:\test.vhdx -SizeBytes 2TB -BlockSizeBytes 64MB -Dynamic

    Best Regards

    Elton JI


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time.
    Thanks for helping make community forums a great place.


    • Edited by Elton_JiModerator Thursday, July 24, 2014 3:07 AM
    • Marked as answer by Mako-Wish Wednesday, July 30, 2014 10:17 PM
    • Unmarked as answer by Mako-Wish Wednesday, August 6, 2014 11:47 PM
    Thursday, July 24, 2014 3:07 AM
    Moderator
  • Hi MotoCrazy,

    Maybe you can try to use command to create a vhd with larger blocksize then copy file again for test  :

    New-VHD -Path d:\test.vhdx -SizeBytes 2TB -BlockSizeBytes 64MB -Dynamic


    I just may have to try this. The issue just happened again over the weekend as I was copying the last bit of data over (right about 2.2TB). I wish I knew what the underlying problem was. This makes me leery about trusting it as a data share. Thankfully it is just our IT Software share, but still... What if someone has to copy a large file over? Is the drive going to drop again?
    Monday, July 28, 2014 3:08 PM
  • That seems to have done the trick. I wanted to throw as much as I could at this disk as quickly as possible, so I copied all 2.4TB in one go with RoboCopy. It completed without issue in about 14 hours. I still feel this should never have been an issue to begin with, but it appears the larger block size was the resolution.

    Thank you kindly for the suggestion Elton!

    Wednesday, July 30, 2014 10:19 PM
  • Unfortunately, it just happened again. The larger block size got me further, but the problem is still there. :(
    Wednesday, August 6, 2014 11:48 PM
  • Mako

    I am having the EXACT same issue were you ever able to resolve this or determine the root cause? Any insight would be very welcomed!!

    Tuesday, June 30, 2015 2:28 PM
  • Hi Diggity,

    Honestly, I am still not confident the issue is resolved, but what I did was disable deduplication on the VHDXs for the VM, then just deduped within the VM. Deduplication is huge for this VM as it is our software share for IT and is close to 7TB raw. I would prefer to dedup at the host OS, but at least the VM is running now.

    Thank you,

    Eric

    • Proposed as answer by Diggity747 Monday, July 20, 2015 11:21 PM
    Tuesday, June 30, 2015 2:45 PM
  • Thanks for you help... This did correct the issue I was experiencing.
    Monday, July 20, 2015 11:18 PM
  • I am seeing a similar issue. Did you see the issue recur after you had disabled deduplication.

    Dedup is the only feature I've enabled in the last two weeks and now I've hit this issue. Had about 6 weeks of stability until now!

    Disabling TRIM on my Host connection to the SAN also improved things a little (no SSD in my SAN) and seemed to make things more stable until now.

    Monday, September 26, 2016 1:49 AM
  • Hi Danny,

    The VM was running okay for quite some time without issue. Here is where things start leading me to believe this could be an OS issue:

    It was originally hosted on our development Hyper-V servers with direct attached storage. No backups were occurring for any of the VMs on these hosts. The more data that got saved here, the more reliant were became on the VM. It got to the point where we would be up the creek without a paddle if we lost the VM. This forced us to spin up a new VM on our SAN-attached production VMware cluster. This was not a V2V conversion; rather a brand new VM using a RDM for the data drive.

    Everything copied over successfully, and things ran perfectly fine for a couple months, when POOF! The drive disappeared again with the same error. It has only happened twice since we spun it up on our production hosts, but this is still not 100% resolved.

    Very strange, very frustrating, I have just learned to deal with restarting it from time to time.

    Tuesday, September 27, 2016 3:31 PM