none
How to take a S2D server offline for maintenance correctly

    Întrebare

  • Hi Expert, 

    I recently deployed three nodes S2D with 3-way mirrors and all three nodes are fully patched monthly, and the patch level is as of Feb-2018 KB4074590, and I would like to ask you guys how to put the S2D node in maintenance correctly, as mine seems behavior different than the MS docs: https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/maintain-servers 

    Here is what I do: 

    1. Drain the role in Failover Cluster Mgr 

    2. Ensure there is no background storage job and virtual disk is healthy by using get-virtualdisk and get-storagejob command 

    3. Install the patch 

    4. Reboot the server 

    5. Resume the role

    Here is my questions: 

    1. After I drain the role without reboot, what the status of the volume should be? On the MS doc, the status should be In service, but mine is actually said healthy, please see below:

    [ Image 1 ]

    2. As soon as I restart node 1, the storagejob start straight away as below, but shouldn't it be no job until i resume the role:

    [ Image 2 ]

    3. As soon as node 1 come back without resume the role, the storage repair job run automatically, is it normal? Shouldn't it run after I resume the role, please see below?

    [ Image 3 ]

    My assumption is even I drain one of the node, the S2D volume is still online and accepting IO. It feels like I kill the server and the whole volume need to resync. Ideally the resync should be quick as it only do the changes during the downtime, but mine took almost 12 hours to resync for 3 x 8TB volumes. 

    Can you please give me some idea?

    [ It won't let me post images for some reason, they are available here: https://imgur.com/a/FnEl0 ] 

    Thanks

    joi, 22 februarie 2018 05:45

Răspunsuri

  • Hi darth_sun,

    >1. After I drain the role without reboot, what the status of the volume should be? On the MS doc, the status should be In service, but mine is actually said healthy, please see below:

    In my test, after "Pause" the node, when run "Get-VirtualDisk", the status of the disk in "health", and I checked the virtual disk's owner node, it is the left node:

    >2. As soon as I restart node 1, the storagejob start straight away as below, but shouldn't it be no job until i resume the role:

    When node2 is running while not resume, the result of "Get-StorageJob" is as below:

    >3. As soon as node 1 come back without resume the role, the storage repair job run automatically, is it normal? Shouldn't it run after I resume the role, please see below?

    As far as I'm concerned, the repair should start after we resume the node.

    As for the behavior not the same as we concerned, I would recommend to open a case with MS, since we are limited to troubleshoot the root cause of the behavior. Here is the link to open a case with MS:

    https://www.microsoft.com/en-us/worldwide.aspx

    Best Regards,

    Anne


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    vineri, 23 februarie 2018 06:53
    Moderator
  • Hello Everyone,

    What you are seeing is a change made in updates back in May/June.  

    When a node is drained, the resources/workloads are moved from the server but the disks on that server in the S2D pool continue to provide read/writes.  This means that the window in which writes are not done on that nodes disks is reduced to when the node shuts down and when it's back up.  The smaller the window, the less data needs to be repaired.  Some customers put the node in maintenance for hours and sometimes days.  Reducing the repair dataset means smaller windows when your 3way mirror only has 2 copies of its data.

    The other change you are noting is when the repair job starts.  To reduce the time that there is less than full 3 copies of data, as soon as a node goes down the repair job is started.  This allows for the scenario where you have 4 or more nodes and 3-way mirror (or 5 or more nodes with parity or Mirror Accelerated Parity) and a node is shutdown or restarted, the repair job might find space on another node to start capturing the writes and ensuring more resiliency.  For 3 node and 3-way mirror there is no other node to safe a copy since the remaining nodes already have copies.  However, the repair job is started and as soon as there is a place to put the data, it will do so and until then the job is there ready and you just don't see progress.

    What you are seeing is expected.  I'd recommend keeping your systems updated.  We continue to optimize and resolve issues by providing fixes in the monthly updates.

    I'll review the documentation to ensure it's consistent with the new behaviors. 

    I hope this helps,

    Steven Ekren

    Program Manager

    Storage Spaces Direct

    Microsoft


    This posting is provided "AS IS" with no warranties, and confers no rights.

    marți, 15 mai 2018 15:04

Toate mesajele

  • Hi darth_sun,

    >1. After I drain the role without reboot, what the status of the volume should be? On the MS doc, the status should be In service, but mine is actually said healthy, please see below:

    In my test, after "Pause" the node, when run "Get-VirtualDisk", the status of the disk in "health", and I checked the virtual disk's owner node, it is the left node:

    >2. As soon as I restart node 1, the storagejob start straight away as below, but shouldn't it be no job until i resume the role:

    When node2 is running while not resume, the result of "Get-StorageJob" is as below:

    >3. As soon as node 1 come back without resume the role, the storage repair job run automatically, is it normal? Shouldn't it run after I resume the role, please see below?

    As far as I'm concerned, the repair should start after we resume the node.

    As for the behavior not the same as we concerned, I would recommend to open a case with MS, since we are limited to troubleshoot the root cause of the behavior. Here is the link to open a case with MS:

    https://www.microsoft.com/en-us/worldwide.aspx

    Best Regards,

    Anne


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    vineri, 23 februarie 2018 06:53
    Moderator
  • Hi darth_sun,

    Appreciate your feedback.

    Best Regards,

    Anne


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    luni, 26 februarie 2018 02:07
    Moderator
  • Hi Anne,

    Thanks for your help, I have also opened a support ticket with MS, will see how we go.

    Regards,

    Sun

    luni, 26 februarie 2018 02:44
  • I'm having a VERY similar issue. I'm deploying S2D with 6 nodes, each with 12 Seagate EXOS 6TB SAS drives firmware E004, and 2x Intel P4600 4TB drives per node. 

    At current full patch as of April CU, when I suspend-clusternode -drain, all physical disks still show online health, as do all vdisks...even if I stop the cluster service on the drained node the same stays true. However, when shutdown the drained node, the same repair job as above starts, and the physical disks go to lost communications status, and the vdisks exhibit odd behavior, changing from a status of Incomplete;Degraded;Degraded,Incomplete,Inservice;Degreaded,Inservice;Inservice.

    Disk IO at this point on the cluster is reduced significantly, but it continues. When the node comes back, all IO stops, and the vdisk goes in to a paused state due to IO Timeout, and all IO on the vdisk stops. I have been unable to remedy this problem. I have tried with specifying ONLY 3 way mirroring, as well as the standard multi tier configuration FOCM / VMM offers. This is a new deployment that has failed testing. This behavior was not observed in my POC build with test hardware last november, so I suspect an update has tanked this. Is there any information that anyone else has come across on this problem?

    miercuri, 18 aprilie 2018 07:38
  • Hi!

    any updates on this? Did the support ticket with MS help?

    we had vdisks go down as well, as soon as the node came back up after maintenance. This happened multiple times with 2 out of 3 vdisks, but sometimes there is no problem at all. Last month we purchased new disks and created new vdisks, so far so good but i can't say i trust it. Storage jobs are still starting immediately when a drained node goes down, and it takes many hours te complete them, every byte is read and the storage throughput is very high during the repair jobs. This is not normal i hope? During our next maintenance schedule at the end of this month we will run a cluster validation report.

    Best regards,
    Peter

    luni, 14 mai 2018 13:29
  • Hi Peter,

    I have MS ticket 118022317695525 and the technical lead actually said this is normal behavior which I don't believe it is, such as storage job kick in as soon as node is drained and takes many hours to complete the repair job...

    Right now, I still keep on doing what I'm used to do, I'm hoping someone can give us some expert advise for this post.

    marți, 15 mai 2018 00:41
  • Hello Everyone,

    What you are seeing is a change made in updates back in May/June.  

    When a node is drained, the resources/workloads are moved from the server but the disks on that server in the S2D pool continue to provide read/writes.  This means that the window in which writes are not done on that nodes disks is reduced to when the node shuts down and when it's back up.  The smaller the window, the less data needs to be repaired.  Some customers put the node in maintenance for hours and sometimes days.  Reducing the repair dataset means smaller windows when your 3way mirror only has 2 copies of its data.

    The other change you are noting is when the repair job starts.  To reduce the time that there is less than full 3 copies of data, as soon as a node goes down the repair job is started.  This allows for the scenario where you have 4 or more nodes and 3-way mirror (or 5 or more nodes with parity or Mirror Accelerated Parity) and a node is shutdown or restarted, the repair job might find space on another node to start capturing the writes and ensuring more resiliency.  For 3 node and 3-way mirror there is no other node to safe a copy since the remaining nodes already have copies.  However, the repair job is started and as soon as there is a place to put the data, it will do so and until then the job is there ready and you just don't see progress.

    What you are seeing is expected.  I'd recommend keeping your systems updated.  We continue to optimize and resolve issues by providing fixes in the monthly updates.

    I'll review the documentation to ensure it's consistent with the new behaviors. 

    I hope this helps,

    Steven Ekren

    Program Manager

    Storage Spaces Direct

    Microsoft


    This posting is provided "AS IS" with no warranties, and confers no rights.

    marți, 15 mai 2018 15:04