none
Server 2016 Storage: StorDiag EventID 504 Error Causing Extremely Slow Disk Write Speeds RRS feed

  • Question

  • Recently, maybe within the last week, my Server 2016 installation has had extremely slow write speeds. I have two drives in my system. A SSD (C:) and an HDD (D:) and both have been generating Storage errors in the event log. Specifically StorDiag EventID 504, whenever a large files are being written (I've copied an example below). When observing the speeds of a large write, I can see that initially it's fine, but after a few seconds a major slow down takes place, and from then on the speed is very inconsistent. Such a write can generate 30 or more 504 errors. There are also associated errors in the NTFS log, such as Event ID 149 which says that IO failures have taken place (also copied below).

    Because this is suddenly happening to both my drives simultaneously, I doubt its a driver issue or a hardware failure issue. My initial Google searches have only related the 504 errors to Card Readers and other removable media, but not disk drives. The 504 and 149 errors implies that write operations are failing, but I just can't see why. My system is fully upgraded, and the Windows reports that drives have the latest drivers. I'll include my system configuration below.

    If anyone has an idea as to what my problem is or how to troubleshoot, I would really appreciate the help. Thank you.

    --------

    System Configuration:

    OS Name            Microsoft Windows Server 2016 Datacenter
    Version            10.0.14393 Build 14393
    System Type        x64-based PC
    Processor          Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz, 
                       3401 Mhz, 4 Core(s),8 Logical Processor(s)
    BIOS Version/Date  American Megatrends Inc. 0301, 9/16/2011
    SMBIOS Version     2.6
    BIOS Mode          Legacy
    Motherboard        ASUSTeK Computer INC.
    
    Platform Role            Workstation
    Secure Boot State        Unsupported
    Installed Physical       Memory (RAM)   16.0 GB DDR3
    Total Virtual Memory     18.3 GB
    Available Virtual Memory 12.5 GB
    Page File Space          2.38 GB
    
    C Drive:                 Boot Drive
    Model                    INTEL SSDSA2CW120G3 (SATA SSD)
    Bytes/Sector             512
    Size                     111.79 GB (120,031,511,040 bytes)
    Free Space               44.94 GB (48,249,929,728 bytes)
    
    D Drive:
    Model                    ST2000DL003-9VT166 (SATA HDD)
    Bytes/Sector             512
    Size                     1.82 TB (2,000,396,321,280 bytes)
    Free Space               746.33 GB (801,364,180,992 bytes)

    -------------

    StorDiag Event ID 504

    Log Name:      Microsoft-Windows-Storage-ClassPnP/Operational
    Source:        Microsoft-Windows-StorDiag
    Date:          6/12/2019 00:53:30
    Event ID:      504
    Task Category: Class
    Level:         Error
    Keywords:      Device I/O control request
    User:          computer\Administrator
    Computer:      my.computer.net
    Description:
    Completing a failed IOCTL request.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-StorDiag" Guid="{F5D05B38-80A6-4653-825D-C414E4AB3C68}" />
        <EventID>504</EventID>
        <Version>1</Version>
        <Level>2</Level>
        <Task>200</Task>
        <Opcode>101</Opcode>
        <Keywords>0x800000040000000</Keywords>
        <TimeCreated SystemTime="2019-06-12T04:53:30.284022800Z" />
        <EventRecordID>997253</EventRecordID>
        <Correlation />
        <Execution ProcessID="1784" ThreadID="2336" />
        <Channel>Microsoft-Windows-Storage-ClassPnP/Operational</Channel>
        <Computer>my.computer.net</Computer>
        <Security UserID="S-1-5-21-4150441485-2800377868-2508223801-500" />
      </System>
      <EventData>
        <Data Name="DeviceGUID">{D393DD4A-6271-DA94-0AA1-DD1DE3F36251}</Data>
        <Data Name="DeviceNumber">0</Data>
        <Data Name="Vendor">NULL</Data>
        <Data Name="Model">ST2000DL003-9VT166</Data>
        <Data Name="FirmwareVersion">CC32</Data>
        <Data Name="SerialNumber">            6YD0GE9Q</Data>
        <Data Name="IrpStatus">0x80000011</Data>
        <Data Name="IoctlControlCode">0x7c088</Data>
      </EventData>
    </Event> 

    --------------

    NTFS Event ID 149

    In the past 3603 seconds we had IO failures.
    This may indicate a failing disk.
    
               High latency IO count: 7
               Failed writes: 0
               Failed reads: 0
               Bad clusters relocated: 0
    
               Volume Id: {139daef3-0000-0000-0000-100000000000}
               Volume name: C:
               Is boot volume: true
    


    Thursday, June 13, 2019 5:34 PM

All replies

  • Hi,

    Thanks for your question.

    May I ask is this your new installed server 2016?

    Is it a clean bootup system, or any other features or third party programs?

    Any other Extra-plug devices to the server?

    Based on my experience, we can try the following steps to inspect the OS in first troubleshooting.

    1) Set the computer to clean boot state to check for software conflicts. We can refer to this link,

    http://support.microsoft.com/kb/929135

    2)Please try the System File Checker tool to repair missing or corrupted system files to see if it helps. Detailed command, we can run “SFC /Scannow” in the CMD with admin right by referring the following docs,

    https://support.microsoft.com/en-sg/help/929833/use-the-system-file-checker-tool-to-repair-missing-or-corrupted-system

    3)Please check the disks mounted to the system with Cmdlets “CHKDSK”. The following discussed this commands in detail for your reference.

    https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/chkdsk

    https://www.howtogeek.com/howto/windows-vista/guide-to-using-check-disk-in-windows-vista/

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    4) We need to monitor if there's any problem such as high CPU, low RAM on this server to impact I/O communication. We can also use Performance Monitor to analyze which process occupy much I/O flight on the disks. 

    Windows Performance Monitor Overview  

    Hope above information can help you. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com


    Friday, June 14, 2019 6:18 AM
    Moderator
  • Hello Michael,

    Thanks so much for responding to me. I've taken the time to go through your list and follow each of your steps.

    This installation is a clean install that's a few months old. As far as I know, I didn't have this problem a week ago. It's hard to tell since I don't often transfer/copy large files, which is when this problem becomes obvious. I get the 504 StorDiag error about every second as I transfer files.

    I don't have any plug and play devices except for a disabled device used for communicating with a Cisco Switch. 

    The only recent program installed around the time that I noticed this problem was Docker Desktop for Windows. I uninstalled it, but saw no improvement.

    Step 1:

    I performed a clean boot with no improvement.

    Step 2:

    I performed a system file check and it found no errors.

    Step 3:

    I performed Check Disk on both my drives. Errors, where found and fixed, but there was no improvement with transfers. However, I've noticed that I don't get those NTFS 149 errors anymore. This stopped before I ran Check Disk, but I thought I should note it. 

    Step 4:

    I've run a custom made log during two transfers of large files. I'm not sure if you'd like to see the results or how I could relay them to you. Let me know.

    Thanks for your help so far. Any more advice would be greatly appreciated.

    Stan

    Saturday, June 15, 2019 9:44 PM
  • Hi,

    Thanks for your detailed update. 

    You could post the logs to OneDrive. So that we get it for deep troubleshooting. 

    Please also note, OneDrive is public sharing tool. If you upload something that may include your privacy information (like IPs, hostname, or other sensitive information), please cover up it for your security.

    If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael 


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, June 18, 2019 10:00 AM
    Moderator
  • Okay, here is the link the logs I recorded: https://1drv.ms/u/s!AlpS5dfXcetRfFEVJrUKyl2_yu0?e=8ZDetM

    Let me know if you need additional counters or a larger data set.

    Again, thanks so much for your attention on this matter.

    -Stan

    P.S. How do I verify my account? I can't put links into my messages until I'm verified, but it's not clear how to do that. I don't think I've received a verification email.

    Wednesday, June 19, 2019 2:29 AM
  • Hi,

    Thanks for your information.

    We could check the following link to verify your TN account. Later you can upload something like screenshots. Please remember to cover up the information if it is sensitive related to your privacy for your security.

    https://social.msdn.microsoft.com/Forums/en-US/090972cb-b81f-498f-b718-948caca975c4/verify-account-41?forum=reportabug

    Highly appreciate your effort and time. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Wednesday, June 19, 2019 3:00 AM
    Moderator