none
Ubuntu guests to read-only filesystem while running backup

    Question

  • Hello,

    Via our hosting party we have 4 VPS machines on a Hyper V platform. Three machines with Ubuntu 14.04, one with CentOS 7. The Ubuntu machines are using the latest stable kernel available (3.13.0-49 or 3.13.0-52).

    This environment has been up since Febuary 1st 2015. During the last 4 weeks we have a lot of problems with the Ubuntu machines. The problems occuring can be described as follows:

    • Server creates backup on the three specific times a day. A Powershell script creates backup to a DFS-share between both servers, but I also believe other backup strategies have been tried before.
    • Ubuntu servers go into - sometimes - a read-only fileystem during these backups. This never happended with multiple servers at the same time, but all three Ubuntu machines have had this problem once or multiple times.

    Have there been significant changes in the last 4 weeks?

    • On all guests, including the CentOS machine, the diskspace has increased due to usage of the machine
    • One of the partititions of he CentOS machine has been expanded using GParted

    Are there any logs available? Yes there are, but during a read-only mode logging stops (of course). So we are not able to see why the filesystem goes into read-only. Nevertheless, please find below two logs that occur usually during backup. The first one is of the CentOS machine. Ubuntu generates the same log most of the time. However, sometimes, it generates a different one. This is the second one.

    May  3 14:00:03 kernel: [1089161.488037] hv_storvsc vmbus_0_1: cmd 0x2a scsi status 0x2 srb status 0x82
    May  3 14:00:04 kernel: [1089161.488042] hv_storvsc vmbus_0_1: stor pkt ffff8800e03eca80 autosense data valid - len 18
    May  3 14:00:04 kernel: [1089161.488044] storvsc: Sense Key : Unit Attention [current]
    May  3 14:00:04 kernel: [1089161.488047] storvsc: Add. Sense: Changed operating definition
    May  3 14:00:04 kernel: [1089161.493155] sd 2:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters.

    And the one that is geneated by Ubuntu (sometimes).

    May  3 06:47:01 kernel: [1063180.054514] hv_storvsc vmbus_0_1: cmd 0x85 scsi status 0x2 srb status 0x86
    May  3 06:47:01 kernel: [1063180.054524] hv_storvsc vmbus_0_1: stor pkt ffff8800f2f5b640 autosense data valid - len 18
    May  3 06:47:01 kernel: [1063180.054526] storvsc: Sense Key : Illegal Request [current]
    May  3 06:47:01 kernel: [1063180.054529] storvsc: Add. Sense: Invalid command operation code
    Questions:
    • Is there any one that has experience with these problems?
    • Why is only Ubuntu crashing and is CentOS doing just fine?
    • And of course; the central question: how should we do to solve this problem?

    Kind regards and thanks for the help,
    Frederik



    • Edited by Genkgo Friday, May 08, 2015 10:15 AM
    • Moved by BrianEhMVP Friday, May 08, 2015 1:47 PM
    Friday, May 08, 2015 10:12 AM

All replies

  • We developed an upstream patch to the kernel for this and will be working it into the distributions right away. The public bug for Ubuntu is https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1454758

    I filed the bug today and will be escalating it with Canonical. It is likely the fix will appear in a Stable Release Update (SRU) in a few weeks.

    I am also working similar upstream integration requests into other Linux distributions.


    Joshua R. Poulson, Program Manager, Microsoft Open Source Technology Center

    Wednesday, May 13, 2015 4:45 PM