none
Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups

    Question

  • We're running various version of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.    On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).    As a result, we must reboot the Ubuntu guest to get the file system to mount RW again.

    The Error we see is the following:

    Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] 
    Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition
    Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.
    Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82
    Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt ffff88006eb6c700 autosense data valid - len 18
    Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] 
    Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition
    Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.

    This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later.   

    Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have.   We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both.

    As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this.   We can stop the backup process and all is good, but we need nightly backups to image all of our VM's.   All the windows guests have no issues of course.   We also have some CentOS based guests running without issues from what we've seen.

    Any assistance would be great.

    Brian

    Thursday, April 30, 2015 8:56 PM

All replies

  • This failed write is definitely not expected. Can you tell me a little bit more about your environment, we'd like to replicate your conditions locally.

    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Friday, May 01, 2015 4:06 PM
  • We're running as a host, a Supermicro 2U Server with Two E5-2690 Xeon CPU's, 8 cores each.   160GB of RAM and an adaptec 71605 16 port SATA RAID controller, with mostly all Samsung SSD's attached except for 4 WD 750GB Backup Disks.   These particular VM's are running on a block of 4 Raid 10 SSD's.   However, we've seen this issue on other host platforms we have with different configurations, so we've ruled out hardware as an issue.

    Running Server 2012 R2 Datacenter, Version 6.3.9600.17196

    Backup is standard Windows Server Backup to the dedicated Raid 10 Backup Set, nightly at midnight.   The VM guests are sitting on a standard NTFS file system.    The file system will mount ready-only randomly, but we usually see the problem once every 2-3 weeks...so it's not that often.  Regardless, I wouldn't expect this error to begin with.

    Brian

    Friday, May 01, 2015 10:09 PM
  • Hello everybody!
    We have the same problem with the virtual machine Debian running on Windows 2012.
    After a backup virtual machine, the system becomes - Read Only. 
    Yesterday the problem appeared after checkpoint.
    This problem does not appear every time after backup or Checkpoint, i have this problem once a month.

    Log:
    hv_storvsc vmbus_0_1: cmd 0x28 scsi status 0x2 srb status 0x82
    hv_storvsc vmbus_0_1: stor pkt ffff8801514e0d80 autosense data valid - len 18
    storvsc: Sense Key : Unit Attention [current]
    storvsc: Add. Sense: Changed operating definition
    sd 2:0:0:0: [sda] Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.
    hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x82
    hv_storvsc vmbus_0_1: stor pkt ffff88062b2190c0 autosense data valid - len 18
    storvsc: Sense Key : Unit Attention [current]
    storvsc: Add. Sense: Changed operating definition
    Wednesday, May 06, 2015 8:43 AM
  • I have the same problem on my Hyper-V Server 2012R2. Please help to resolve. It is very critical issues.
    Wednesday, May 06, 2015 2:55 PM
  • Any Update on this?   Here is the error message we received this morning from one of our systems:

    The Log Files showed this:

    Brian

    Wednesday, May 13, 2015 4:14 PM
  • We developed an upstream patch to the kernel for this and will be working it into the distributions right away. The public bug for Ubuntu is https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1454758

    I filed the bug today and will be escalating it with Canonical. It is likely the fix will appear in a Stable Release Update (SRU) in a few weeks.

    I am also working similar upstream integration requests into other Linux distributions.


    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Wednesday, May 13, 2015 4:35 PM
  • Thanks!   I look forward to the updated kernel when released and will monitor the bug.   I'll report back any findings when I can test the fix.    I could re-compile and test now, but I'll wait for the upstream release.

    Brian

    Wednesday, May 13, 2015 5:39 PM
  • We started testing this morning. In our case, vss snapshot failed the second time with the build provided by Ubuntu (using 3.19 kernel). Two options: wrong build or the patch is not successful. For more details, see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1445195 .

    • Edited by Genkgo Friday, May 22, 2015 10:13 AM
    Friday, May 22, 2015 10:13 AM
  • Is there any news on this one? We have stopped making automatic backups. This is certainly not a situation we want to be in. We would be willing to provide any assistance required to solve the issue.
    Tuesday, May 26, 2015 12:45 PM
  • Is there any news? We have still problem with backup.After a backup virtual machine, the system becomes - Read Only.
    Tuesday, July 07, 2015 1:56 PM
  • Odd enough, I just had the same issue last night with our only Ubuntu 14.04 LTS guest.

    File system went in read-only and we had to reboot the VM.

    We do have a few other vss problems with our CentOS guests which another thread has been already created

    Wednesday, July 08, 2015 12:43 PM
  • The Ubuntu bug

    https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1454758 is progressing. The patches there will be backported to 14.04 and I will respond here when that kernel is available to everyone, but right now the fix is only in development and testing releases. We are still tracking down additional reported problems, but the patch does greatly reduce the frequency of VSS backups getting errors that make the volume go read-only.


    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Wednesday, July 08, 2015 4:10 PM
  • Is there any more progress on this? Last time there was insisted not to take the patches into production. So we did not. Creating backups with the old kernel is really painful. It is extremely unreliable. Furthermore, the patches are also not a complete fix to the problem. This is causing unnecessary downtime and headaches. I hope this can fixed very soon.
    Monday, July 20, 2015 7:04 PM
  • https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1454758 is fixed in 14.04 at this point, if you do an update and pull it in, you should pick up the patches we submitted to Ubuntu.

     $ sudo apt-get update

     $ sudo apt-get upgrade

    There might be other updates in this list, so be sure to verify you want those packages to be updated before going forward.

     $ sudo apt-get dist-upgrade

    Sometimes dist-upgrades will remove some packages, so be careful with this list of updates as well. However, you should be getting a new release of the kernel and associated tools.

    There are reports of VSS issues even with the fix, but the patch has greatly reduced the number of incidents.


    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Monday, July 27, 2015 6:14 PM
  • Any news? We have still problem with backup.

    Ubuntu 14.04.3 LTS (GNU/Linux 3.19.0-25-generic x86_64)

    Wednesday, August 12, 2015 8:57 AM
  • Also still problems for us. Just had another crash. Any progression on the cause of this issue?
    Wednesday, September 09, 2015 4:35 PM
  • I had similar issues on my Azure instances with Ubuntu 14.04. Trusty. I uninstalled the kernel Linux 3.16.0-48-generic and used Linux 3.19.0-46-generic which runs stable again. 

    As I've seen in the changelog Linux 3.19.0-25 has the same issues like 3.16.0-48. 

    from this https://bugs.launchpad.net/ubuntu/+source/linux-lts-utopic/+bug/1492146 launchpad post it seems that vivid kernel should fix this issue, too. Anybody to confirm this?

    Best Chris

    http://stappz.com

    Thursday, September 10, 2015 8:43 AM
  • The latest 3.19 kernels (installing linux-virtual-lts-vivid will get you those, and they are now the default in the gallery Ubuntu images) have all of the latest fixies, but we still are seeing customers report occasional VSS backup failures.

    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Monday, September 21, 2015 7:00 PM
  • Hi,

    Do you have any updates? When we will have solution?

    Monday, January 25, 2016 3:57 PM
  • We are continuing to look at the problem, it could be that there is another storvsc bug we haven't found. Additional storvsc fixes are in progress for 3.19 and 4.2 kernels, and will be coming out of proposed and into GA this weekend.


    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Tuesday, January 26, 2016 7:58 PM
  • Hi,

    we have exactly the same problem and error messages with a Debian Jessie 3.16.0-4 on a Windows Server 2012R2 Hyper-V. Once or twice a week after the backup (done with Altaro Hyper-V-Backup) the VM doesn't work anymore and we have to repair the file system.

    What can we do here? Any help would be greatly appreciated.

    Dino

    Tuesday, May 24, 2016 9:30 AM
  • We are still tracking down the root cause on Ubuntu in the bug listed above. Since Debian Jessie's kernel shares the same pedigree as Ubuntu's 3.16 kernel, it is likely to have the same root cause. Once we discover it, I will work with Debian's kernel team on fixing the issue.

    Thanks, --jrp


    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Monday, June 06, 2016 4:45 PM
  • I had a similar problem with backups a while ago with one of my ubuntu 14.04.03 machines. This machine was put into production and when it came time to setup VeeamZIP to back it up, it wouldn't work. Everytime I did a backup with VeeamZIP the machine would lock, the backup would fail and the only way to resolve it was to force a reset on the VM.

    I am not sure if the problem in this thread is the same as this one because I fixed it so long ago but what I did was I upgrade to the wily kernel. Upgraded to the wily specific linux-cloud-tools linux-tools and then my problem went away.

    Wednesday, June 08, 2016 2:44 AM
  • It sounds like your issue is the VSS daemon wasn't running. When you installed Linux-tools and Linux-cloud-tools it would have ensured the daemon was present on your next boot.

    Thanks, --jrp


    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Monday, June 27, 2016 4:38 PM