Windows Server 2012 - Hyper-V - Cluster Sharded Storage - VHDX unexpectedly gets copied to System Volume Information by "System", Virtual Machines stops respondig
-
Monday, February 18, 2013 11:07 AM
We have a problem with one of our deployments of Windows Server 2012 Hyper-V with a 2 node cluster connected to a iSCSI SAN.
Our setup:
Hosts - Both run Windows Server 2012 Standard and are clustered.
- HP ProLiant G7, 24 GB RAM. This is the primary host and normaly all VMs run on this host.
- HP ProLiant G5, 20 GB RAM. This is the secondary host that and is intended to be used in case of failure of the primary host.
- We have no antivirus on the hosts and the scheduled ShadowCopy (previous version of files) is switched off.
iSCSI SAN:
- QNAP NAS TS-869 Pro, 8 INTEL SSDSA2CW160G3 160 GB i a RAID 5 with a Host Spare. 2 Teamed NIC.
Switch:
- DLINK DGS-1210-16 - Both the network cards of the Hosts that are dedicated to the Storage and the Storage itself are connected to the same switch and nothing else is connected to this switch.
Virtual Machines:
- 3 Windows Server 2012 Standard - 1 DC, 1 FileServer, 1 Application Server.
- 1 Windows Server 2008 Standard Exchange Server.
- All VMs are using dynamic disks (as recommended by Microsoft).
Updates
- We have applied the most resent updates to the Hosts, VMs and iSCSI SAN about 3 weeks ago with no change in our problem and we continually update the setup.
Normal operation:
- Normally this setup works just fine and we see no real difference in speed in startup, file copy and processing speed in LoB applications of this setup compared to a single host with two 10000 RPM Disks. Normal network speed is 10-200 Mbit, but occasionally we see speeds up to 400 Mbit/s of combined read/write for instance during file repair.
Our Problem:
- Our problem is that for some reason a random VHDX gets copied to System Volume Information by "System" of the Clusterd Shared Storage (i.e. C:\ClusterStorage\Volume1\System Volume Information).
- All VMs stops responding or responds very slowly during this copy process and you can for instance not send CTRL-ALT-DEL to a VM in the Hyper-V console, or for instance start task manager when already logged in.
- This happens at random and not every day and different VHDX files from different VMs gets copied each time. Some time it happens during daytime wich causes a lot of problems, especially when a 200 GB file gets copied (which take a lot of time).
What it is not:
- We thought that this was connected to the backup, but the backup had finished 3 hours before the last time this happended and the backup never uses any of the files in System Volume Information so it is not the backup.
An observation:
- When this happend today I switched on ShadowCopy (previous files) and set it to only to use 320 MB of storage and then the Copy Process stopped and the virtual Machines started responding again. This could be unrelated since there is no way to see how much of the VHDX that is left to be copied, so it might have been finished at the same time as I enabled ShadowCopy (previos files).
Our question:
- Why is a VHDX copied to System Volume Information when scheduled ShadowCopy (previous version of files) is switched off? As far as I know, nothing should be copied to this folder when this functionis switched off?
List of VSS Writers:
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.Writer name: 'Task Scheduler Writer'
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No errorWriter name: 'VSS Metadata Store Writer'
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No errorWriter name: 'Performance Counters Writer'
Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
State: [1] Stable
Last error: No errorWriter name: 'System Writer'
Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
Writer Instance Id: {7848396d-00b1-47cd-8ba9-769b7ce402d2}
State: [1] Stable
Last error: No errorWriter name: 'Microsoft Hyper-V VSS Writer'
Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
Writer Instance Id: {8b6c534a-18dd-4fff-b14e-1d4aebd1db74}
State: [5] Waiting for completion
Last error: No errorWriter name: 'Cluster Shared Volume VSS Writer'
Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
Writer Instance Id: {d46c6a69-8b4a-4307-afcf-ca3611c7f680}
State: [1] Stable
Last error: No errorWriter name: 'ASR Writer'
Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Instance Id: {fc530484-71db-48c3-af5f-ef398070373e}
State: [1] Stable
Last error: No errorWriter name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {3792e26e-c0d0-4901-b799-2e8d9ffe2085}
State: [1] Stable
Last error: No errorWriter name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {6ea65f92-e3fd-4a23-9e5f-b23de43bc756}
State: [1] Stable
Last error: No errorWriter name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {71dc7876-2089-472c-8fed-4b8862037528}
State: [1] Stable
Last error: No errorWriter name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {cb0c7fd8-1f5c-41bb-b2cc-82fabbdc466e}
State: [1] Stable
Last error: No errorWriter name: 'Cluster Database'
Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
Writer Instance Id: {23320f7e-f165-409d-8456-5d7d8fbaefed}
State: [1] Stable
Last error: No errorWriter name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {f23d0208-e569-48b0-ad30-1addb1a044af}
State: [1] Stable
Last error: No errorPlease note:
- Please only answer our question and do not offer any general optimization tips that do not directly adress the issue! We want the problem to go away, not to finish a bit faster!
All Replies
-
Tuesday, February 19, 2013 6:19 AMModerator
Hi,
System Volume Information folder is root of every drive. The folder contains following information:
- System Restore points. You can disable System Restore from the "System" control panel.
- Distributed Link Tracking Service databases for repairing your shortcuts and linked documents.
- Content Indexing Service databases for fast file searches.
- Information used by the “Volume Snapshot Service” (also known as "Volume Shadow Copy") so you can back up files on a live system.
- Longhorn systems keep WinFS databases here.
So according to “System Volume Information” definition, the operation you mentioned is Volume Shadow Copy.
Check event viewer to find Volume Shadow Copy related event logs and post them.
> All VMs are using dynamic disks (as recommended by Microsoft).
It’s okay if you use dynamic disks in a test environment, but for virtual machine that run server workloads in a production environment, dynamic VHD are not recommended.
Hyper-V: Dynamic virtual hard disks are not recommended for virtual machines that run server workloads in a production environment
http://technet.microsoft.com/en-us/library/ee941151(v=WS.10).aspxFor more information please refer to following MS articles:
What's the deal with the System Volume Information folder
http://blogs.msdn.com/b/oldnewthing/archive/2003/11/20/55764.aspx
Windows Server 2008 R2 Backup problem
http://social.technet.microsoft.com/Forums/en/windowsbackup/thread/0fc53adb-477d-425b-8c99-ad006e132336Hope this helps!
If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.
Lawrence
TechNet Community Support
- Marked As Answer by Lawrence LvMicrosoft Contingent Staff, Moderator Monday, March 11, 2013 9:28 AM
-
Wednesday, February 20, 2013 12:17 AM
Hallo Lawrence!
Thankyou for youre reply, some comments to help you and others who read this thread:- First of all, we use Windows Server 2012 and the VHDX as I wrote in the headline and in the text in my post. We have not had this problem in similar setups with Windows Server 2008 R2, so the problem seem to be introduced in Windows Server 2012.
These posts that you refer to seem to be outdated and/or do not apply to our configuration:
- The post about Dynamic Disks:
http://technet.microsoft.com/en-us/library/ee941151(v=WS.10).aspx is only a recommendation for Windows Server 2008 R2 and the VHD format. Dynamic VHDX is indeed recommended by Microsoft when using Windows Server 2012 (please look in the optimization guide
for Windows Server 2012).
Infact, if we use fixed VHDX then we would have a bigger problem since fixed VHDX are generaly larger then Dynamic Disks, i.e. more data would be copied and that would take longer time = the VMs would be unresponsive for a longer time. - The post "What's the deal with the System Volume Information folder" http://blogs.msdn.com/b/oldnewthing/archive/2003/11/20/55764.aspx is for Windows XP / Windows Server 2003 and some things has changed since then. for instance In Windows Server 2012, Shadow Copies cannot be controlled by going to Control panel -> System. Instead you right-click on a Drive (i.e. a Volume, for instance the C drive/Volume) in Computer and then click "Configure Shadow Copies".
- Windows Server 2008 R2 Backup problem
http://social.technet.microsoft.com/Forums/en/windowsbackup/thread/0fc53adb-477d-425b-8c99-ad006e132336 - This post is about the Antivirus software trying to scan files used during backup that exists in the System Volume Information folder and we do not
have any antivirus software installed on our hosts as I stated in my post.
Comment that might help us:
- So according to “System Volume Information” definition, the operation you mentioned is Volume Shadow Copy. Check event viewer to find Volume Shadow Copy related event logs and post them.
Why?
- Furhter investigation suggests that a volume shadow copy is somehow created even though the Schedule for Shadows Copies is turned off for all drives. This happens at random and we have not found any pattern. Yesterday this operation took almost all available disk space (over 200 GB), but all the disk space was released when I turned on scheduled Shadow Copies for the CSV.
I therefore draw these conclusions:
- The CSV Volume has about 600 GB of disk space and since Volume Shadows Copy used 200 GB, or about 33% of the disk space, and the default limit is 10% then I conclude that for some reason the unscheduled Volume Shadow Copy did not have any limit (or ignored the limit).
- When I turned on the Schedule I also change the limit to the minimum amount which is 320 MB and this is probably what released the disk space. That is, the unscheduled Volume Shadow Copy operation was aborted and it adhered to the limit and deleted the Volume Shadow Copy it had taken.
- I have also set the limit for Volume Shadow Copies for all other volumes to 320 MB by using the "Configure Shadow Copies" Window that you open by right clicking on a drive (volume) in Computer and then selecting "Configure Shadow Copies...".
- It is important to note that setting a limit for Shadow Copy Storage, and disabaling the Schedule are two different things! It is possible to have unlimited storage for Shadow Copies when the Schedule is disabled, however I do not know if this was the case Before I enabled Shadow Copies on the CSV since I did not look for this.
- I now have defined a limit for Shadow Copy Storage to 320 MB on all drives and then no VHDX should be copied to System Volume Information since they are all larger than 320 MB.
Does this sound about right or am I drawing the wrong conclusions?
Limits for Shadow Copies:
Below we list the limits for our two hosts:
"Primary Host":
C:\>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.
Shadow Copy Storage association
For volume: (\\?\Volume{e3ad7feb-178b-11e2-93e8-806e6f6e6963}\)\\?\Volume{e3ad7feb-178b-11e2-93e8-806e6f6e6963}\
Shadow Copy Storage volume: (\\?\Volume{e3ad7feb-178b-11e2-93e8-806e6f6e6963}\)\\?\Volume{e3ad7feb-178b-11e2-93e8-806e6f6e6963}\
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 320 MB (91%)
Shadow Copy Storage association
For volume: (E:)\\?\Volume{dc0a177b-ab03-44c2-8ff6-499b29c3d5cc}\
Shadow Copy Storage volume: (E:)\\?\Volume{dc0a177b-ab03-44c2-8ff6-499b29c3d5cc}\
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 320 MB (0%)
Shadow Copy Storage association
For volume: (G:)\\?\Volume{f58dc334-17be-11e2-93ee-9c8e991b7c20}\
Shadow Copy Storage volume: (G:)\\?\Volume{f58dc334-17be-11e2-93ee-9c8e991b7c20}\
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 320 MB (3%)
Shadow Copy Storage association
For volume: (C:)\\?\Volume{e3ad7fec-178b-11e2-93e8-806e6f6e6963}\
Shadow Copy Storage volume: (C:)\\?\Volume{e3ad7fec-178b-11e2-93e8-806e6f6e6963}\
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 320 MB (0%)C:\>cd \ClusterStorage\Volume1
Secondary host:
C:\>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.
Shadow Copy Storage association
For volume: (\\?\Volume{b2951138-f01e-11e1-93e8-806e6f6e6963}\)\\?\Volume{b2951138-f01e-11e1-93e8-806e6f6e6963}\
Shadow Copy Storage volume: (\\?\Volume{b2951138-f01e-11e1-93e8-806e6f6e6963}\)\\?\Volume{b2951138-f01e-11e1-93e8-806e6f6e6963}\
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 35,0 MB (10%)
Shadow Copy Storage association
For volume: (D:)\\?\Volume{5228437e-9a01-4690-bc40-1df85a0e6736}\
Shadow Copy Storage volume: (D:)\\?\Volume{5228437e-9a01-4690-bc40-1df85a0e6736}\
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 27,3 GB (10%)
Shadow Copy Storage association
For volume: (C:)\\?\Volume{b2951139-f01e-11e1-93e8-806e6f6e6963}\
Shadow Copy Storage volume: (C:)\\?\Volume{b2951139-f01e-11e1-93e8-806e6f6e6963}\
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 6,80 GB (10%)C:\>
There is something strange about the limits on the Secondary host!
- I have not in any way changed the settings on the Secondary host and as you can see, the Secondary host has a maximum limit of only 35 MB storage on the CSV, but it also shows that this is 10% of the Volume. This is clearly not the case since 10% if 600 GB = 60 GB!
- The question is, why does it by default set a too small limit (i.e. < 320 MB) on the CSV and is this the cause of the problem? I.e. is the limit ignored since it is smaller than the smallest amount you can provide using the GUI?
- Is the default 35 MB maximum Shadow Copy limit a bug, or is there any logical reason for setting a limit that according to the GUI is too small?
- First of all, we use Windows Server 2012 and the VHDX as I wrote in the headline and in the text in my post. We have not had this problem in similar setups with Windows Server 2008 R2, so the problem seem to be introduced in Windows Server 2012.
-
Saturday, February 23, 2013 8:02 AM
Hi ,
It is very strange that the CSV has only 35MB reserved. Please first run the following command on two nodes to verify the GUID of the CSV.
mountvol
Then check which guid matches the CSV from the vssadmin list shadowstorage output. Run the following command to resize the diff area for CSV.
vssadmin resize shadowstorage /for=<GUID> /on=<GUID> /maxsize=60GB
If you have scheduled or manual backup of CSV disk, there may be also shadow copies in System volume information, not only scheduled shadow copies.
Please apply the following hotfix to eliminate known issue when backing up a CSV disk in Windows 2012.
http://support.microsoft.com/kb/2813630
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
-
Saturday, February 23, 2013 11:30 AM
Hallo!
Thankyou for your response!
Possible values for VolumeName along with current mount points are:
\\?\Volume{e3ad7feb-178b-11e2-93e8-806e6f6e6963}\
*** NO MOUNT POINTS ***\\?\Volume{dc0a177b-ab03-44c2-8ff6-499b29c3d5cc}\
E:\\\?\Volume{f58dc334-17be-11e2-93ee-9c8e991b7c20}\
G:\\\?\Volume{e3ad7fec-178b-11e2-93e8-806e6f6e6963}\
C:\\\?\Volume{640527ec-b537-4a78-b8c2-90911d5e4927}\
C:\ClusterStorage\Volume1\\\?\Volume{e3ad7ff3-178b-11e2-93e8-806e6f6e6963}\
Z:\I mistook the system volume for the clustered shared storage and it is that volume that has the 35 MB limit. The ShadowCopy Storage for the CSV is not listed. See above output. I have tried to run the vssadmin resize and add shadowstorage commands, but this only generates an error, see below.
C:\>vssadmin resize shadowstorage /on=\\?\Volume{640527ec-b537-4a78-b8c2-90911d5e4927}\ /for=\\?\Volume{640527ec-b537-4a78-b8c2-90911d5e4927}\ /maxsize
e=10GB
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.Error: The shadow copy provider had an error. Please see the system and
application event logs for more information.----- Generates below error in event log -----
Error 2013-02-23 12:11:47 VSS 12289 None "Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{640527ec-b537-4a78-b8c2-90911d5e4927} - 00000000000003F4,0x00530024,0000000000000000,0,0000009B7BBCBD60,4096,[0]). hr = 0x80070001, Incorrect function.
.Operation:
Changing diff-area maximum sizeContext:
Volume Name: \\?\Volume{640527ec-b537-4a78-b8c2-90911d5e4927}\
Diff-area volume: \\?\Volume{640527ec-b537-4a78-b8c2-90911d5e4927}\
Diff-area maximum size: 10737418240"C:\>vssadmin add shadowstorage /on=\\?\Volume{640527ec-b537-4a78-b8c2-90911d5e4927}\ /for=\\?\Volume{640527ec-b537-4a78-b8c2-90911d5e4927}\ /maxsize=1
0GB
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.Error: The shadow copy provider had an error. Please see the system and
application event logs for more information.----- Generates below error in event log -----
Level Date and Time Source Event ID Task Category
Error 2013-02-23 12:12:02 VSS 12289 None "Volume Shadow Copy Service error: Unexpected error DeviceIoControl(\\?\Volume{640527ec-b537-4a78-b8c2-90911d5e4927} - 0000000000000370,0x00530024,0000000000000000,0,0000009B7BBC4CF0,4096,[0]). hr = 0x80070001, Incorrect function.
. "I have not yet run the hotfix since I am not sure that it apply to us. We only run backups on the VMs by using Child Partition Backup, but we do not run a backup of the entire CSV. Does this hotfix apply anyway?
-
Thursday, April 11, 2013 12:17 PM
Hi,
Yes, the hotfix applied to the Windows Server 2012 host. From the vss error it seems that there is a filter driver on the storage system that cause the IO error. Check if there is any 3rd vss provider or disk related app, update them to latest version.
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
-
Friday, May 17, 2013 10:25 AM
At last, i have found somebody with the same issue as me!!!
We have been having random system freezes with our VM's all locking up and losing disk access for 5-10minutes, they then all come back online as if nothing had happend. During the freeze today i was able to login and take a screen shot of Resource Monitor (see below), one of the VM's had massive amounts of reads and system volume information had lots a writes. Shadow Copies are disabled but the System Volume Information folder is 92GB!
How did you fix the issue? Did you apply the hotfix?
-
Friday, May 17, 2013 10:28 AM
Also, i should add some setup deails...
HP ML350p Gen8 (Xeon E5-2620, 64GB ram) running Server 2012/HyperV
RAID 1 SAS (2x146GB) C: Partition
Server 2012 (HOST OS)RAID 10 SAS (6x450GB) E: Partition
SBS 2011 (Guest1 VHDX)
Server 2008R2 (Guest2 VHDX)
Server 2012 (Guest3 VHDX)Here is a better picture.
-
Friday, May 17, 2013 10:39 AMAlso just to add, the AYSERVER3 VHDX is only one of the three that is dynamic, the others are fixed as they host Exchange and SQL.
.png)
