locked
Cluster Shared Volume has entered a paused state because of 'STATUS_USER_SESSION_DELETED(c0000203)'. All I/O will temporarily be queued until a path to the volume is reestablished RRS feed

  • Question

  • I have been getting the above error on my Windows Server 2016 Hyper-V Cluster ever since setting it up. At first it was every 30 min, but now it is every hour. I am not running SUA and can find very little help in were to start to resolve this error.
    • Edited by leemm81 Monday, July 10, 2017 12:33 AM
    Monday, July 10, 2017 12:33 AM

All replies

  • Hello,

    To take a further investigation for this issue, you can find additional information in Event Viewer under  ‘Applications and Services Logs \ Microsoft \ Windows \ FailoverClustering-CsvFs \ Operational’.

    It contains the source of the auto-pause, and in some cases may contain additional parameters helping to further narrow down the scenario. 

    More details about Troubleshooting Cluster Shared Volume Auto-Pauses, you can see the following article.

    https://blogs.msdn.microsoft.com/clustering/2014/12/08/troubleshooting-cluster-shared-volume-auto-pauses-event-5120/

    Best regards,
    Andy Liu

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Monday, July 10, 2017 6:52 AM
  • Lots of people are having the same issue after the rolling upgrade from WS 2012R2 to WS 2016. Is this your case?
    Friday, July 14, 2017 12:14 PM
  • Hello,

    Could you please install the most recent Windows Update KB4025334 (OS Build 14393.1532) for the hosts, and then to see if this issue happen again.

    You can get the update from the following link.

    https://support.microsoft.com/en-za/help/4025334/windows-10-update-kb4025334

    Best regards,

    Andy Liu


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, July 19, 2017 1:29 AM
  • Same here.

    EDIT : Server 2016 Hyper-V Cluster.

    Not really sure if this is related to when we are performing Veeam Backup.

    Any suggestions Microsoft?


    Thursday, November 2, 2017 7:23 PM
  • Lots of people are having the same issue after the rolling upgrade from WS 2012R2 to WS 2016. Is this your case?
    Do you think this is related to the upgrade rather than just because VSS is using the CSV for Checkpoints?
    Friday, November 3, 2017 12:27 PM
  • Hello,

    Could you please install the most recent Windows Update KB4025334 (OS Build 14393.1532) for the hosts, and then to see if this issue happen again.

    You can get the update from the following link.

    https://support.microsoft.com/en-za/help/4025334/windows-10-update-kb4025334

    Best regards,

    Andy Liu


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Hey Andy,

    Has anybody confirmed that KB4025334 solves this?


    Friday, November 3, 2017 12:28 PM
  • We are also seeing this warning, though less frequently. We upgraded our cluster from 2012 R2 -> 2016 and are backing up our VMs with VEEAM.
    Wednesday, November 29, 2017 7:46 PM
  • I had an outage yesterday and believe its related to this.

    My 2016 hosts lost access to the shared storage with the error:

    The IO operation at logical block address 0xdad30b0 for Disk 3 (PDO name: \Device\MPIODisk3) was retried.

    I have also seen these errors in event viewer.

    Cluster Shared Volume 'Volume6' ('Cluster Disk 5') has entered a paused state because of 'STATUS_USER_SESSION_DELETED(c0000203)'. All I/O will temporarily be queued until a path to the volume is re-established.

    I have applied KB4025334.

    I also use Veeam Backup & Recovery 9.5 update 3 to backup the cluster roles.

    Is there an update on this?

    Best regards,

    Ben 

    Tuesday, February 20, 2018 9:44 AM
  • Having the same issues here since upgrading to server 2016.  Have all of the updates current.  I get csv pauses every hour exactly, someimes every 30 minutes.  Tried disabling Veeam backups, still get the errors.  Have also had VM pauses and crashes related to this issue.

    Server 2012r2 worked just fine.  SAS HBAs firmware and drivers updated.  So far nothing has made it any better.

    Thanks!

    Thursday, April 19, 2018 10:16 PM
  • having this issue too. between 30 minutes and 2 hours. OpsMgr reports this warning every time. Using DELL R740 with FC HBA firmware and drivers up to date.

    • Proposed as answer by SebyB79 Saturday, May 5, 2018 5:39 AM
    • Unproposed as answer by SebyB79 Saturday, May 5, 2018 5:39 AM
    Tuesday, April 24, 2018 9:00 PM
  • Same problem here ... dell 730dx, windows server 2016, veeam b&r 9.5-3

    Saturday, May 5, 2018 5:40 AM
  • we've been troubleshooting this for over a year. Updates all installed, latest firmware and bios on all equipment, checked nics and settings for hyper-v. Still get these weekly.

    really I think 2016 is simply broken.  There's rds issues as well since RTM which haven't been addressed.

    basically, we gave up and now waiting for server 2019

    Tuesday, August 7, 2018 6:51 AM
  • Hello!!!

    I'm facing the same problem in ClusterHyperV on 2016, we think backup task from Backup Exec, was stopping the VM than rebooting the VMs, I've tried to split bkp tasks in different times, but have no success,, also I've tried to update nics, recreate teams update windows, update agents from bkp, but I have no success.

    What leave me calm, I'm not alone on this issue.

    If some one solve please help me.


    Monday, August 13, 2018 11:56 PM
  • Hi Glen,
    We did the same, but have no success =(

    Did you tried to reinstall cluster nodes?


    Tuesday, August 14, 2018 12:15 AM
  • Hi Rawlinson,
    That solve your problem?
    Tuesday, August 14, 2018 12:18 AM
  • Hi Mikkel,
    Did you solve with that patch?
    Tuesday, August 14, 2018 12:18 AM
  • I have been getting the above error on my Windows Server 2016 Hyper-V Cluster ever since setting it up. At first it was every 30 min, but now it is every hour. I am not running SUA and can find very little help in were to start to resolve this error.

    Solved my Friend.

    https://support.microsoft.com/en-us/help/4338822/windows-10-update-kb4338822

    Install this KB 4338822 

    Good Luck! 

    Regards.

    Lucas Grzybowski

    Tuesday, August 14, 2018 4:49 PM
  • Lucas, not seeing anything about STATUS_USER_SESSION_DELETED in that KB...
    Saturday, August 25, 2018 1:15 AM
  • How many cluster shared volumes do you have in the cluster, and how many Hyper-V hosts? I've seen this error escalating when the number of CSV are greater than the number of hosts in the cluster.

    It seems that if a Hyper-V server hosts more than one CSV at a time, this error sometimes occur.

    Tuesday, August 28, 2018 7:41 AM
  • KB4338822 doesn't work for me. I confirmed that it is installed on all cluster nodes, but as soon as too many VMs are running on CSVs that are not owned by the cluster node the VM is running on, the error occurs.
    Thursday, September 6, 2018 11:36 AM
  • Hi Carsten,

    Right, please check the versions of Integration Services of VMs they need to be same.

    If no update it :)

    Check too, the configuration of heartbeat, check this one 

    https://blogs.msdn.microsoft.com/clustering/2012/11/21/tuning-failover-cluster-network-thresholds/

    In my cluster I've set as below.

    PS C:\Windows\system32> (get-cluster).SameSubnetDelay = 2000
    PS C:\Windows\system32> (get-cluster).SameSubnetThreshold= 20 
    PS C:\Windows\system32> (get-cluster).CrossSubnetDelay= 4000 
    PS C:\Windows\system32> (get-cluster).CrossSubnetThreshold= 40 

    Try this too.

    Check your Backup System, if are starting bkp task for the Cluster and at the same time executing for VMs inside the cluster, that is one reason to lot your traffic at network adapter, cuz of that we change the heart beat time.


    Good Lucky

    Tuesday, September 11, 2018 6:38 PM
  • Seems like they finally solved it.

    Addresses an issue that occurs when using multiple Windows Server 2016 Hyper-V clusters. The following event appears in the log:

    “Cluster Shared Volume 'CSVName' ('CSVName') has entered a paused state because of 'STATUS_USER_SESSION_DELETED(c0000203)'. All I/O will temporarily be queued until a path to the volume is reestablished.”



    October 18, 2018—KB4462928 (OS Build 14393.2580)

    https://support.microsoft.com/en-us/help/4462928/windows-10-update-kb4462928
    Friday, October 19, 2018 7:31 AM
  • Well, not really.

    I am on current

    February 12, 2019—KB4487026 (OS Build 14393.2791)

    and still get it once a night for a fraction of a second

    Cluster Shared Volume has entered a paused state because of 'STATUS_CONNECTION_DISCONNECTED(c000020c)'. All I/O will temporarily be queued until a path to the volume is reestablished.

    CSV in question is on 8Gb fully redundant FC all SSD storage array SAN (just one volume, second volume is fine)


    • Edited by scerazy Sunday, February 17, 2019 8:47 AM
    Sunday, February 17, 2019 8:47 AM
  • Hi,

    Same here, I have been struggling with this issue for more than half a year. October 2018 CU did not fix it.

    Have tried pretty much everything I know - guess that is not enough.

    Any solutions?

    Sunday, February 24, 2019 8:50 PM
  • I remember that on 2012 R2 it took close to 3 years after release that MS eventually managed to patch everything to the level that I eventually did not get any of these error on (then) iSCSI SAN.

    So one would expect that Server 2016 should eventually be sorted as well (unless thay do not care any more & want to force people to 2019 - which DEFINITELY is not production ready!)

    Saturday, March 2, 2019 5:19 PM
  • Almost daily at random times I get this error

    Witness is lost, cluster crashes, a total nightmare!

    edit

    Above was unrelated to storage (but somehow it manifested itself in that way), it was due to having unsupported network card (which is dedicated, not CNA) in Dell Poweredge


    • Edited by scerazy Saturday, November 16, 2019 5:24 PM
    Thursday, March 28, 2019 9:28 AM
  • Just bumping this thread to say we finally found a fix for these timeout errors.  We evicted our nodes one at a time and rebuilt them all to server 2019.  Errors gone!
    Monday, June 3, 2019 8:17 AM
  • Curious as to what vendor your server hardware is?

    Despite having most of the Server 2019 drivers for some of their server models, if you talk to a Dell rep/tech they will say that Dell doesn't actually support Server 2019 on their servers yet.

    Friday, July 12, 2019 2:16 PM
  • This is still an issue. We just lost 17 VM due to this. - The error started and these VM had a crash and lost all data from time stamp until the server got powered off. (one file server managed to have data in memory or something for an entire day, so we lost production data for a day.....)

    Been trying to find the reason for it and got an idea as some of you mention. BACKUP!!! - We had a restore of an entire VM the time we saw the events. (using Veeam). So our HyperV/ Clusterservice does not at all like the amount of data it is presented during a restore.

    Then I started looking at the network setup on our hosts. (teamed network) and the vSwitch is configured with 5 interfaces using LACP (teaming mode) and Hyper-V port (Load balancing)

    How is your network configured?!? - A colleague mentioned that we should go for Dynamic on Load Balancing. But the setup is configured by Hyper-V specialist consultant and the specified that it should be Hyper-V port....

    Tuesday, November 12, 2019 11:10 AM
  • Load Balancing would apply to networking, this error comes from storage, so unless you use Converged Network Adapters for both data & storage, it makes no sense.

    I use dedicated Qlogic 4x 8Gb Fibre Channel per server & still (much less than previously) get on occasion this stupid error.

    It seems to do nothing apart from eventlog (nothing crashes, corrupts etc)

    Seb

    Saturday, November 16, 2019 4:21 PM