Windows Server 2012 - Can occasionally not access second virtual hard drive inside a VM
-
Saturday, October 20, 2012 8:57 PM
I run Windows Server 2012 RTM Hyper-V and I can occasionally not access the second virtual hard drive (dynamically expanding VHDX) attached to the VM through the virtual SCSI controller. I can however access the first hard drive that is connected with the virtual IDE controller.
I get the following warning in the event log under “Administrative Events” every 30 seconds when this happens:
- Log Name: System
- Source: Storvsc
- ID: Storvsc
- Message: “Reset to device, \Device\RaidPort0, was issued.”
I get this error once or twice a week and it has caused serious problems since one of the virtual servers that have this problem is a fileserver and the second hard drive contains all the data.
The only quick solution to the problem that I have found is to force the virtual machine to stop using the “Turn Off” feature since a normal shut down does not work (stops at shutting down the event log or similar) and then start the virtual machine again.
You can also wait for about 30 minutes or longer until the disk for some reason becomes accessible again by itself.
My research into this problem shows that:
- Only 2 of the 10 VMs running Windows Server 2012 RTM that I have, have this problem.
- Both these VMs have a second virtual hard drive (dynamically expanding VHDX) that cannot be accessed for 30 minutes or longer.
- Check Disk of the virtual hard drive shows no errors.
- The second hard drive is attached to the virtual SCSI controller.
- I can find no problems at all with the physical storage on the (not related) 4 hosts that I have. The problem exists only in the VMs.
I have now attached the second virtual hard drive to the virtual IDE controller to see if this permanently fixes this problem (i.e. does not happen for at least a week).
Is there something wrong with the virtual SCSI controller or the virtual SCSI device driver that comes with Windows Server 2012 RTM? Does anyone else have this problem?
All Replies
-
Monday, October 22, 2012 3:11 AMModerator
Hi,
I found some post which discussed similar issue; all these posts indicate the issue may be caused by driver issue. After update or configure SCSI controller driver, they fix the issue.
Seems we now don’t have new related SCSI driver, change disk controller to IDE should work around this issue. And unlike previous versions of Microsoft virtualization technology, there is no performance difference between using a virtual IDE controller or a virtual SCSI controller when accessing virtual hard disks.
Since you have changed to use IDE controller, monitor its status and give us feedback for further troubleshooting.
For more information please refer to following MS articles:
Windows 7 64 bit RC Random Freezes, especially during Installation of Applications
http://social.technet.microsoft.com/Forums/en/w7itproperf/thread/1b73d120-a341-40ce-bdf5-8a519dd2ba88
Event 129 Source "storahci" in Win8RTM
http://social.technet.microsoft.com/Forums/en-US/w8itprohardware/thread/b7194f3a-3dba-441b-9f2e-bd4a47db3b19
Serious problem with the Standard SATA AHCI controller driver
http://social.technet.microsoft.com/Forums/en-US/W8ITProPreRel/thread/fe1bb186-7bdf-4fb7-9b83-71169cd7e52f
Installing and Configuring a Hyper-V Virtual Machine for use with BizTalk Server
http://technet.microsoft.com/en-US/library/dd722832(v=BTS.10).aspxHope this helps!
If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.
Lawrence
TechNet Community Support
- Edited by Lawrence LvMicrosoft Contingent Staff, Moderator Monday, October 22, 2012 3:11 AM
-
Monday, October 22, 2012 9:47 PM
Hallo!
I have found these and other similar posts and that together with my internal research is why I am now trying the virtual IDE controller instead.We have not had any issues with these two servers, but it is too early to tell if this "fixed" the issue since this occures randomly and it can be days between occurrences. I will report back in a about a week if the error does not come back, or earlier, if the error occurs earlier.
It would, however, be interesting to know if anyone else has had this issue?
-
Tuesday, October 23, 2012 5:41 AMModerator
Hi,
We don’t receive more report for such issue in Windows server 2012, however we will continue perform research and follow up this post.
We will give you feedback if there is any progress.
If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.
Lawrence
TechNet Community Support
-
Tuesday, October 23, 2012 12:55 PMAny chance you have a malware detection application that is scanning .vhdx files? You should most likely exclude those files from the scanning.
tim
-
Sunday, November 11, 2012 3:37 PM
No, we are not running any kind of malware detection application on the Hosts.
-
Sunday, November 11, 2012 4:16 PM
It has now been three weeks since we changed the second harddrive from the virtual SCSI controller to the virtual IDE controller and the error has not appeared again on any of our two affected virtual servers.
The question remains if anyone else has hade this problem?
A reason for no other reports is that this might be an unusual setup since most users probably only use 1 virtual harddisk. Therefore, Microsoft Tech Support should test this in a lab environment by following these easy steps:
- Install Windows Server 2012 RTM on a host and set up a virtual server with two virtual harddisks, one connected to the virtual IDE controller (primary, C:) and one connected to the virtual SCSI controller (secondary, D:). Then create a share on D: and insert some small and large files and folders.
- Set up a number (5-10) of virtual Windows 7 and 8 computers that randomly accesses the share (read and write) to simulate user trafic to the share and record performace and any problems with delayed reading or writing, or storage related errors on both the clients and on the server.
- Do the exact same test setup on another host (with the same hardware), but attach both virtual harddisks to the virtual IDE controller on the server.
- Let both servers run for one or two weaks and compare the results and look for any differences in speed, errors in the event log, etc. Look especially for event id 129 - “Reset to device, \Device\RaidPort0, was issued.” on the virtual servers.
The above steps should take at most 4 hours of dedicated work for a skilled Microsoft technician to setup and evaluate to see if this is indeed a bug with the virtual SCSI controller, or if this is an isolated insident.
Please let me know if anyone else have had this problem, or if you (Microsoft) have or will test this thoroughly.
In the meantime, we will continue to set up all our virtual servers requering a second virtual harddrive on Windows Server 2012 by only using the virtual IDE controller.
-
Friday, January 18, 2013 10:47 PMwe have the same issue under compareable conditions on a w2012 fileserver VM, we are now using the virtual IDE controller since today ....
-
Friday, February 08, 2013 12:47 PM
I was experiencing this same issue; however a bit more frequently. One of my VMs had 3 drives connected via a SCSI controller - perhaps that's why the time between occurrences was less.
I have changed to using IDE controllers instead; however, it should be noted that without the ability to use SCSI drives the VM is limited to using 4 drives total (2 IDE controllers, 2 drives per controller). If SCSI didn't have this issue, you could connect 256 SCSI drives. Point being, it is a pretty big limitation.
-
Wednesday, February 13, 2013 2:38 PMJust experienced same problem. Grateful for your thread here ThomasN which gave us some good clues to the cause.
My thread here:
http://partnersupport.microsoft.com/en-us/mpndataplat/forum/mpncatvirt-mpnmshyperv/eventid-129-reset-to-device-deviceraidport0-was/65654c0f-1acf-4ba1-a131-1bff14e1afe2
Has anyone in Microsoft got a fix for this yet? This is a serious show-stopper and enough to make many think twice about converting their VMWare to Hyper-V. -
Wednesday, February 20, 2013 9:11 PMI can confirm that I am encountering this exact same issue with Hyper-V VM running on Server 2012. The guest VM is also Server 2012 running as a file server. The File Server contains a dynamic virtual disk connected via the virtual SCSI controller. For the past two days at the exact same time, the guest will start throwing 129 errors and the virtual drive will become unresponsive. Only way to resolve is to turn off the VM (shutting down the VM will only cause it to hang) and turn it back on again.
-
Thursday, February 21, 2013 1:54 PM
Today we encountered the same issue:
Hyper-V VM Server 2012,
Fileserver 2012,
2 x IDE VHDX
1 x IDE
3 x SCSI VHDX
Shutdown was not possible; only turning off and rebooting solved the problem.
Maybe some more information - we recently activated DFS-R for the fileservices. Whats about y'all? DFS on or off?
-
Thursday, February 21, 2013 2:33 PMI haven't been using DFS.
-
Wednesday, February 27, 2013 8:21 PM
Is anyone using Windows Server 2012's Deduplication feature? I ask because my timeline was something like this:
- 3 Data drives connected using a virtual SCSI controller
- Deduplication enabled
- Started noticing 'Reset to device' errors
- Found this post
- Changed to virtual IDE controllers
- Stopped having 'Reset to device' errors
- Started having random file share issues (documented here: http://omniacs.wordpress.com/2013/02/27/file-server-issues-caused-by-deduplication/ )
- Disabled deduplication
- No more errors
So - I guess my question is - does the SCSI issue only happen with deduplication?
-
Friday, March 15, 2013 4:28 PM
Hallo!
We did not use the deduplication feature, DFS, or any features (except using regular fileshares in the virtual disk that was connected to the virtual SCSI controller when we got the error.
We have not had this error since we changed to only use virtual IDE controller.
-
Monday, March 18, 2013 6:14 PMNo we're not using deduplication and still encountered the problem.
-
Monday, March 18, 2013 11:15 PM
Same problem here, change the vhdx to ide controller and the problem has not been reproduced for me.
Thx
-
Monday, March 25, 2013 4:09 PM
Since I'm encountering the very same issues I want to know:
What kind of hardware are you guys using?
I have two IBM x3650 M4 machines with a ServeRAID M5110e SAS/SATA controller.
For some more detailed information please check omniacs' blog (he?) mentioned below:
http://omniacs.wordpress.com/2013/02/27/file-server-issues-caused-by-deduplication/
In my experience the machine does not freeze/crash/hang completely but since the storage gets some timeouts it takes a *looong* time to read/write data. Each timeout is 30 seconds (each 30 seconds an 129 event is logged) so this sums up very quickly to 30 minutes and more.
Furthermore the switch to IDE didn't really help... at least one server had problems again but didn't log anything anymore so I switched back to SCSI. That way I at least have event log entries.
-
Monday, March 25, 2013 5:10 PM
'He' is correct :-)
I'm using a Perc H710 contoller. I believe it has the same PowerPC processor as the M5110e.
That is my blog, by the way. I have experienced the same behavior - that switching to IDE from SCSI eliminated the errors being logged, but didn't solve the underlying problem.
I'm on the phone with Dell support (again)... hopefully something will come of this.
-
Wednesday, March 27, 2013 5:39 PM
Hi,
Same problems here.
Deduplication activated -> because it's a fileserver.
I have this problem if there is much traffic/connections -> e.g. in morning if a lot of user do logon, while replication to the second hyper-v-host is activated (disabled at the moment).
Today I had this Problem while lunch-time while the DPM synchronized the VM.
Now I will try with IDE-Mode instead SCSI... Thank you for this Information
Greetings from Switzerland
Dani
Edit: Hyper-V Hosts DL380 G8 P420 Controller / Win2012 (Hosts and Guests)
- Edited by WEDOTRONIC Wednesday, March 27, 2013 5:40 PM
-
Thursday, March 28, 2013 3:06 PMIs everyone here using VHDXs, or has anyone experienced the problem with VHDs? I'm currently converting to VHDs to see if I could replicate the problem.
-
Friday, April 05, 2013 9:13 AM
Is everyone here using VHDXs, or has anyone experienced the problem with VHDs? I'm currently converting to VHDs to see if I could replicate the problem.
I'm using the VHDX format *only* on each VM (currently 5 distributed over 2 Hyper-V hosts). -
Friday, April 05, 2013 9:27 AM
I'm not sure how DPM (or Volume shadow Copy/VSS) is exactly related to this issue but since I had shutdown the DPM server for about 10 days I have not seen these errors anymore.
Two days ago I have switched on the DPM server again (disconnected from the network so no backup will be running) and I deactivated all agents.
Yesterday I reactivated the DPM agent for 2 servers again:
1 DC -> not much is going on here
1 Management Server with SQL Server 2012 SP1, WSUS, System Center 2012 SP1 Configuration Manager, Endpoint Protection etc. -> the machine has work to do
On the DC I cannot see any errors or warnings. About hangs? I'm not sure since I don't watch/monitor the machine all the time.
On the management server I can see 4 disk error events:
Source: disk
Event ID: 153
Event time: 18:31:10 (4 times the same time)
Description (German only, sorry):
Der E/A-Vorgang bei der logischen Blockadresse "40028" für Datenträger "3" wurde wiederholt. It means something like "The I/O operation on logical block address "40028" for disk "3" was repeated."
Interestingly the logical block address (LBA) varies but it's always the same disk.
I'm in contact with some Microsoft guy and he told me to disable a new Windows 2012 feature called ODX: http://technet.microsoft.com/de-de/library/hh831628.aspx
He said to disable it on *all* machines: physical Hosts + VMs -> I didn't had the time so far but perhaps someone can test that? It's a registry key and I doubt that it makes much sense in my environment since I don't have a SAN but only DAS (SATA/NL-SAS and SAS) discs but...
-
Friday, April 05, 2013 12:35 PM
I am guessing that DPM exposed the problem because it generates higher IO, which is probably the same reason I thought deduplication was related.
Using a simple disk IO utility (Parkdale) to write large files to the disks for testing has allowed me to reproduce the problem somewhat more predictably than sitting and waiting. So far, I have been able to reproduce the problem only on dynamic VHDX's. On a fixed VHDX, the problem has not occurred. On a dynamic VHD, the problem has not occurred.
I have been in touch with Dell - who has now opened a case with Microsoft. So, we'll see what happens.
-
Friday, April 05, 2013 4:14 PM
Hi,
Here same problem with VHDX FIXED DISK (2TB) IDE atach!!
This is a very very important problem, this file server have all profile users, bd, etc.. etc...
Microsoft support has investigating the problem.
Please post any upgrade.
Thx
-
Monday, April 08, 2013 3:39 PM
Hi,
Today -> same problem in IDE-Controller-Mode.
I think I will convert the VHDX to VHD - maybe this is the solution until this bug is resolved.
Thank you.
Dani
-
Wednesday, April 10, 2013 7:42 AM
Hi everybody.
We have an open case with Microsoft.
After many tests, they don’t give us any solution yet, today they will change to level 2.
WEDOTRONIC,
Did you convert VHDS to VHD? Do you have any advance after conversion?
We are thinking to change VHDX to VHD this weekend.
We will keep you informed.
Regards,
-
Wednesday, April 10, 2013 10:36 AM
Hi all,
Converting to VHD did NOT solve the problem.
I found a fix for our Hyper-v MS server 2012 guests.
Last night Microsoft released: KB 2822241
Windows Server 2012 cumulative update: April 2013
Note: you need to update HOST and GUESTS.
It did not fix my MS server 2003 guest.
Regards,
Frank
I am not able to post any links (account verification)
- Edited by Frank Hofsteden Wednesday, April 10, 2013 10:36 AM
- Edited by Frank Hofsteden Wednesday, April 10, 2013 10:37 AM
-
Wednesday, April 10, 2013 1:38 PM
http://support.microsoft.com/kb/2822241 does not appear to list the problem being discussed.
The closest thing mentioned is http://support.microsoft.com/kb/2819476 - which is related to Storport.sys logging.
I have applied KB2822241 to a VM and the host - and am still able to reproduce the issue.
-
Wednesday, April 10, 2013 1:58 PMDo you have KB2813630 installed?
-
Wednesday, April 10, 2013 2:09 PMNo - because I am not using a CSV, and the problem is not occurring while backing up a VM. http://support.microsoft.com/kb/2813630
-
Wednesday, April 10, 2013 4:44 PM
http://support.microsoft.com/kb/2822241 does not appear to list the problem being discussed.
The closest thing mentioned is http://support.microsoft.com/kb/2819476 - which is related to Storport.sys logging.
I have applied KB2822241 to a VM and the host - and am still able to reproduce the issue.
Hi, abs Struct,
Did you check if after the KB Installation, you can update the ICS? -
Wednesday, April 10, 2013 5:05 PM
Do you mean integration services? If so, then upon trying to update it, I am told the VM is already running the most up-to-date version.
-
Thursday, April 11, 2013 8:52 PMOwner
Hi All,
Can I request that anyone who is seeing this issue post here with the following details:
- What storage are you using in the host (controller & configuration)
- What guest operating system are you running
- Whether you are using VHD or VHDX
- Whether disks are connected to IDE or SCSI
- Whether you are using spaces (in the host or in the guest)
- Whether you are using CSV
Thanks!
Cheers,
Benjamin Armstrong
============================
Windows Virtualization
Senior Lead Program Manager
This posting is provided AS IS with no warranties, and confers no rights. You assume all risk for your use. -
Thursday, April 11, 2013 8:59 PM
1. Dell Perc H710 - 6 2TB SATA Drives RAID 10
2. Host and guest are running Windows Server 2012
3. VHDX - I have converted to VHD, but was not able to reproduce the problem
4. Tested both. Both exhibit the same behavior - the only difference is that Event ID 129 is logged when connected using SCSI (nothing is logged using IDE)
5. Not using spaces in either
6. Not using CSV
Also - I have only been able to produce the issue on dynamic VHDX's, not fixed. I have not been able to reproduced the issue on either type of VHD.- Edited by abs Struct Thursday, April 11, 2013 9:00 PM
-
Friday, April 12, 2013 7:19 AM
Hi Ben.
Our environment is:
- Servers: Fujitsu Primergy RX300 S7 configured to Boot from SAN from Storage through a Fibre Channel Emulex LPE1250. NPIV Enable.
- Storage EMC VNX 5100.
- Host OS: Windows Server 2012 Datacenter.
- Guest OS: Windows Server 2012 Standard.
- We are Using VHDX.
- System in one CSV.
- Data (2 Tb, fixed disk) in a different CSV.
- Originally it was iSCSI, changed to IDE and just eliminated error ID 129. This disk has the same behaviour.
- Not using spaces.
Many Thanks,
-
Saturday, April 13, 2013 4:32 PM
I have the same Problem.
Hyper-V Server 2012 and a Virtual Server 2012.
Hardware: Dell PowerEdge R710
One Hdd connected to IDE, 3 HDD connected to SCSI.I connected now all four HDD to IDE and hope that next week the Server will work properly again.
-
Monday, April 15, 2013 2:31 PMHaven't you read through this enough to see that changing from virtual SCSI controllers to virtual IDE controllers has only been eliminating the error being logged - but the problem still exists?
-
Monday, April 15, 2013 3:56 PM
I did.
But some entries are saying, that the problem will be solved.
And today I had the first time since 3 months no error and i did not have to restart the Server.
My hope is still alive...
-
Thursday, April 18, 2013 8:35 AM
Hi
Sorry for my late reply.
Same problem after converted to VHD.
I think I will install a Server 2008 R2 because this is a customer-system with 150 Clients and the customer has no more Patience for more tests...
Thank you Microsoft... I'm disappointed.
Dani
-
Thursday, April 18, 2013 8:45 AM
My reply for Ben Armstrong
- What storage are you using in the host (controller & configuration) -> Local Storage at HP DL380G8 P420i Controller (SAS, RAID 10) for VMs and SAS, RAID1 for parent-OS
- What guest operating system are you running -> Server 2012 Std.
- Whether you are using VHD or VHDX -> actually VHD (converted from VHDX) -> same problem.
- Whether disks are connected to IDE or SCSI -> actually IDE -> same problem with SCSI
- Whether you are using spaces (in the host or in the guest) -> No
- Whether you are using CSV -> No, local storage
-> Deduplication activated
-> Backup with DPM 2012 SP1
Thank you.
Best regards,
Dani
Edit: Controller is P420i not P410i
- Edited by WEDOTRONIC Thursday, April 18, 2013 8:56 AM
- Edited by WEDOTRONIC Thursday, April 18, 2013 11:19 AM
-
Thursday, April 18, 2013 1:12 PMI have been able to reproduce the issue with Windows Server 2008 R2 (as the guest).
- Edited by abs Struct Friday, May 10, 2013 12:30 PM
-
Thursday, April 18, 2013 2:59 PM
Hi abs Struct
With Server 2012 as host and Server 2008 R2 as guest?
Because with Server 2008 R2 hosts and guests I didn't have such troubles....
Best regards,
Dani
-
Monday, April 22, 2013 1:13 PM
Hi All,
Can I request that anyone who is seeing this issue post here with the following details:
- What storage are you using in the host (controller & configuration)
- What guest operating system are you running
- Whether you are using VHD or VHDX
- Whether disks are connected to IDE or SCSI
- Whether you are using spaces (in the host or in the guest)
- Whether you are using CSV
Thanks!
Cheers,
Benjamin Armstrong
============================
Windows Virtualization
Senior Lead Program Manager
This posting is provided AS IS with no warranties, and confers no rights. You assume all risk for your use.Hi Ben,
Here are some details about my config:
- What storage are you using in the host (controller & configuration) -> Two IBM x3650 M4 machines with a ServeRAID M5110e SAS/SATA controller. One server is using SAS disks, the other server is using NL-SAS (so an "optimized" SATA). All HDDs are directly attached to the server/controller so there is no SAN in use.
- What guest operating system are you running -> Hosts + all VMs are Windows 2012
- Whether you are using VHD or VHDX -> VHDX only
- Whether disks are connected to IDE or SCSI -> Mixed since switching to IDE only silenced the event logs but the symptoms (hanging VMs) were the same so I decided to at least get a log of the messages for debugging purposes
- Whether you are using spaces (in the host or in the guest) -> No
- Whether you are using CSV -> No
There is nothing fancy here. No Cluster, no Hyper-V replica, no deduplication, no spaces, no SAN, no Fibre Channel, no iSCSI. All Software is Microsoft based. I updated all drivers and firmware to the latest versions from IBM (back to end of March).
One important thing I noticed: since I deactivated all DPM (Data Protection Manager) 4.1.3333.0 the errors are gone but I think the issue is somehow IO related and the backup produces more IO than the normal usage of the systems.
In conclusion I have at least these two event log entries:
Event ID: 129, Source: storvsc, Description: Reset to device \Device\RaidPort0 was issued.
This event is logged multiple times (about 100 (?) times) after exactly 30 seconds (I think 30 seconds is a timeout value).
Event ID: 153, Source: disk, Description: "The IO operation at logical block address "40028" for Disk 1 was retried." The block address varies, of course.
I noticed that at the moment there are only 153 events present. Not sure if that is the case after installing some recent updates since it looks like Microsoft is actively working on that issue and released some hotfixes. For testing-purposes I have re-activated the DPM agent only on 2 virtual servers so far since they aren't that important.
Best regards
Manfred Gloiber
-
Friday, April 26, 2013 11:57 AM
Hello everybody.
Who of you rollback the host from WS2012 to WS2008 R2?
Did the hard disk problems disappear?
We are thinking evict one node from the cluster and install WS2008 R2 with Hyper-V, create a new VM, and transform second disk from VHDX to VHD. Result: Host WS2008 R2 with hyper-v role and a FS with a WS2012.
Best Regards,
- Edited by abiurrunc Friday, April 26, 2013 12:08 PM Change opinion
-
Saturday, April 27, 2013 8:05 PM
Hi All,
Can I request that anyone who is seeing this issue post here with the following details:
- What storage are you using in the host (controller & configuration)
- What guest operating system are you running
- Whether you are using VHD or VHDX
- Whether disks are connected to IDE or SCSI
- Whether you are using spaces (in the host or in the guest)
- Whether you are using CSV
Thanks!
Cheers,
Benjamin Armstrong
============================
Windows Virtualization
Senior Lead Program Manager
This posting is provided AS IS with no warranties, and confers no rights. You assume all risk for your use.Config:
- Windows 2012 Host
- Storage: LSI SAS 9271-8i, RAID 6 (7x SATA 7200) / OCZ Vertex boot drive
- 3 Hyper-V guests: Windows 2008 R2 all using vhdx, all stored on LSI RAID
- 1 guest has a 2nd vhdx attached via virtual SCSI mounted to a folder without a drive letter. This guest is running SQL Server 2012.
- iSCSI NAS mounted on Host over 1GbE
- iSCSI volume was dedicated to Windows Sever Backup. I had to disable the backup schedule after a few days because of the performance and stability problems.
- Backups settings for Hyper-v guests indicated they were using child partition snapshots
- No storage spaces, no CSVs, no deduplication (except for whatever Windows Server Backup was doing)
- Shadow Copies are disabled on all volumes
Symptoms:
- Windows Server Backup brings entire system to a crawl on and off during the 2-4 hours it takes to back up about 550GB.
- A variety of storage and vss related event IDs (25, 129, 130, 137, 153) appear in the logs, usually when backups are active:
- Reboot of host took several hours after I attempted to cancel a backup in progress (this triggered hundreds of 129 events from vhdmp)
- Intermittent performance problems, especially when backups are active
Could the problems posted in this thread actually be VSS related?
I have tested several backup software packages and strategies over the last few months. In my case, I'm investigating if VSS could be misconfigured or if remnants of a previous evaluation could be in conflict.
Conditions when DPM fails to back up Hyper-V virtual machines in an online state
http://technet.microsoft.com/en-us/library/hh757866.aspx#bkmk_unabletoprotectvm"shadow copies can degrade the performance of write operations for the volume they are on (read operations are not affected)"
http://technet.microsoft.com/en-us/library/dd759145.aspxLive Backup of a Hyper-V Guest VM with Hyper-V VSS Writer
http://www.altaro.com/hyper-v/requirements-for-live-backup-of-a-hyper-v-guest-vm-hyper-v-vss-writer/If anyone knows of a good guide to configuring VSS for Hyper-V backups that would be helpful. I will continue to research the VSS theory and post back whatever I find out.
-Jeremy
-
Saturday, May 04, 2013 1:55 PM
I have been pulling my hair about this issue for several weeks! Have a case in with Dell, but nothing back so far.
Controller: PERC H710P Mini, Raid-1 for OS (WS 2012 Datacenter), 2 Raid-5 arrays SAS 6.0Gbps
Guest OS: WS2012 Datacenter
Using VHDX
Connected to SCSI
Not using spaces or CSV.
Using DPM 2012 for backup. Using data deduplication. Was using DFSR but stopped til this is resolved.
Symptoms in VM: users access to drives stops. Unable to "shutdown" the VM, unable to "turn off" the VM. If I wait long enough, 1 or more hours! the VM may begin to respond again.
System event log in VM shows 129 error from source storvsc.
Today I had a new symptom. Early while doing a DPM tape backup of one of the drives in the VM, the errors occured and after some time, the VM went into PAUSE state.
Symptoms in HOST: unable to kill the VM. Other VMs operate normally. UNABLE TO SHUTDOWN the host. Have to force a power off.
Has there been any movement on this over the last week or so?
-
Tuesday, May 07, 2013 7:49 AM
Dear Ben Armstrong,
as you see, we urgently need your help!
Maybe you need further Information? Any logs?
Thank you!
Best regards
Dani
-
Friday, May 10, 2013 9:49 AM
Hi all,
Here we continue with the same problem, microsoft support does not give solution.
our client has lost patience and will migrate the machine to VMware,
finally, someone is store in the file server firefox / thunderbird users files?
Thanks MS
-
Friday, May 10, 2013 10:45 AM
Hi
Yes here, we store the firefox-profiles in the file server.
Why? Do you see a connexion?Best regards,
Dani
-
Friday, May 10, 2013 10:56 AM
ok,
when the disk goes down, there are about 50 users connected with Firefox/thunderbird profile.
Virtually, this is the only traffic/processes trough disk and network....
We are thinking it may be the cause and the customer will move these files to another server or local machines. -
Friday, May 10, 2013 11:10 AM
We too are have lost a number of customers to VMWare because of this bug. We can't even tell them that Microsoft are working on finding a fix because we don't know that they are.
Has this actually been recorded as a bug? Is anyone at MS working on finding the cause of this major show stopper and fixing it? As far as we know at the moment MS have just ignored this in the hope that it might go away by itself!
On a more positive note we have now been running for almost three months without encountering this problem. What we have done to cure it or whether we are just experiencing a temporary respite, I don't really know. The only possible is that we uninstalled the AMD 'Fuel' service.
Our config:
Server 2012, Win 8
multiple dynamic .vhdx on virtual SCSI controller <- this seems to be the problem
no de-dupe or anything else exoticIf if MS are hoping to take a share of the VMWare market and nobody at MS can find the bug in the virtual SCSI controller code then perhaps the whole virtual SCSI controller should be scrapped and re-written!
Rgds,
Nick -
Friday, May 10, 2013 12:17 PM
Some of the symptoms I observed, at first appeared similar to what others have reported, but now I am suspecting a DNS problem. The server is 2012 standard with 2008R2 guests (see my previous post for details). On two different days, one of the guests locked up and the whole server needed to be restarted. The host restart took over half an hour one time and the server was hard-reset the next time. On each occasion there was a DNS conflict with the guest that locked up - ping of the guest was resolving to the wrong IP address (conflicting with a different physical pc). A windows server backup to an iSCSI device was in progress during one incident, but not the other.
-
Friday, May 10, 2013 12:24 PM
I have a case open with Microsoft. I opened a case with Dell in February. In April, Dell passed it on to Microsoft. I have been providing a bunch of xperf / logman captures, but so far the only feedback I have gotten is that they are trying to determine if it is a bug with Windows, or a problem with my environment. I have a lot of opinions about the situation - but I'll keep them to myself.
I doubt the problem is caused by AMD - we are running Intel CPU's - and still have the problem. The level of intermittence is somewhat bizarre. Sometimes the problem will present multiple times a day, other times it can be weeks between occurrences.
The only difference I have noticed between using the virtual SCSI controller and the virtual IDE controller is that the SCSI controller logs the error (Reset to Device) and the VHD controller does not. Other than that, symptoms are the same.
- Edited by abs Struct Friday, May 10, 2013 12:34 PM
-
Friday, May 10, 2013 12:25 PMIn my experience, any load on disk IO can cause the problem. In your situation, Firebox/Thunderbird profiles are the problem - but only in that they require accessing the disk. If you task the server with nothing, then it is likely that the problem will not occur.
-
Friday, May 10, 2013 12:27 PMYour problem may well be a DNS problem; however, this issue has been reproduced with no virtual NIC connected to the VM and disk IO being generated directly in the VM (not over the network).
-
Thursday, May 16, 2013 6:33 PM
I think I'm seeing the same issue
Dell R820 Server. Perc 710P and 810 Adapters. All updates, firmwares up to date etc.
2012 Std Host Server all updates
VM is using IDE controller, the 2nd drive is dynamic and using the new vhdx format. VM is also a 2012 std server, file server containing user profiles/data shares.
We also use DPM 2012 SP1 on another physical machine. Deduplication is only turned on on the DPM, not the Host or VM.
All other VM's on the same host are fine. Some are also using second dynamic disks but not as much activity.Issue: Users randomly lose connection to the VM Data Server, yet it's still pingable, and I can remote into it. OS drive on VM seems fine, I can browse it without issue. I noticed the last time when users lost connection to it, the second I opened file explorer and clicked on the second drive the entire VM locked up. There was also no information showing the size of the drive when looking at the my computer view in file explorer.exe. Also noticed the Host server is a little sluggish at this time but the other VM's are fine. The ONLY solution is to turn the VM off and back on. It will not shut down on its own.
VM has latest integration tools installed.
This troubled server VM was very built very recently. The only things installed on it were Forefront, DPM Agent, and SCCM Agent. We initially thought Forefront was the cause since it seemed to happen right after it was installed. We left it uninstalled for a week, and was fine, then reinstalled it and sure enough it locked again. We currently have it and SCCM agent uninstalled and are waiting to see if it locks up again. Size of the second drive is 2.2TB with the max growth size set to 4Tb.
-
Thursday, May 16, 2013 7:39 PM
Hi newf123
Yes, that's exactly the problem we're talking.
Since two days I did not have the problem more.
I installed the newest updates from May patchday, and I connected the second VHD to the secondary IDE-Controller instead to the primary controller.
However, maybe there was during this days not much activity...
Best regards,
Dani
-
Monday, May 20, 2013 11:05 AM
Hello Everybody.
The last Tuesday (05/20/2013) we make a big change on the VM FS, let me explain:
- We create a new CSV in the storage.
- We create a new Virtual SAN on Hyper-V hosts.
- We attach the storage through Fibre Channel directly to the VM FS.
- We copy all the VHDX content to the new Volume on the VM.
- Finally, we detach the VHDX.
Usually, we reproduce the error on Thursday and Friday around 10 times each day, as you can see these were the critical days.
The last Thursday VM FS had not errors, I was singing a Victory.
The last Friday we reproduce the error again:
- Error 129 0ver the FC.
- This Time I can see the volume through File Explorer working on the VM FS. The volume was not accessible from another server o client machine.
- The Server Service hang, I try to restart, but the system tell me that cannot restart.
- Just power off the VM FS through the Cluster arrange the service.
- After the VM FS power on, I went to Events Viewer to see the 129 error and it surprises me, the error 129 on FC isn’t there, yes, it disappeared.
We are thinking to change the VM FS OS to 2008 R2, Did somebody reproduce the error with a WS 2012 host hyper and a WS 2008 R2 VM?
Best Regards,
Angel Biurrun C.
- Edited by abiurrunc Monday, May 20, 2013 11:07 AM error
-
17 hours 5 minutes agoI have reproduced the error with WS 2012 as the host, and WS 2008 R2 as the VM - but only using local storage (no SAN or CSV).
-
8 hours 6 minutes ago
Many Thanks abs Struc.
We will change one host and the VM to WS2008 R2.
Regards,
Angel Biurrun C.
-
7 hours 53 minutes ago
Hi abiurrunc
Did you already try to connect the second VHD(X) to the secondary IDE-controller?
And did you install the newest Microsoft updates?
Since this changes our server is running... I hope it stays...Best regards,
Dani
-
7 hours 44 minutes ago
Hi Wedotronic.
Yes I did, I connected the VHDX to the secondary IDE-Controller, I installed all the MS Updates and the error reproduced again and again, it's terrible.
Regards,
Angel Biurrun C.
-
7 hours 19 minutes ago
Hi abiurrunc
I think you installed all updates at the HV Host, too?
On secondary IDE-controller I disconnected the DVD/CD-drive. I don't know if this has a connexion....
Best regards,
Dani
-
4 hours 8 minutes ago
Hi Wedotronic
Yes, all servers are updated.
I disconnected the DVD/CD-Drive.
Regards,
.png)
