none
Virtual Exchange 2016 complains about faulty storage hardware

    Question

  • Hi, we have Exchange 2016 running on Windows Server 2012 R2 as Generation 2 VMs on Hyper-V 2012 R2. The Hyper-V cluster is connected to a Dell Compellent SAN over FC. The SAN seems to perform fine (it has SSDs as Tier 1), and other VMs are not experiencing performance issues. However, Exchange keeps logging database issues and failing over databases between nodes:

    ESE (ESE) 509 Information Store ... A request to read from the file "....edb" at offset 91187904512 (0x000000153b390000) for 262144 (0x00040000) bytes succeeded, but took an abnormally long time (49 seconds) to be serviced by the OS. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem. 

    ESE (ESE) 510 Information Store ... A request to write to the file "....edb" at offset 32768 (0x0000000000008000) for 32768 (0x00008000) bytes succeeded, but took an abnormally long time (58 seconds) to be serviced by the OS. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem. 

    ESE (ESE) 532 Information Store ... A request to read from the file "....edb" at offset 0 (0x0000000000000000) for 32768 (0x00008000) bytes has not completed for 28 second(s). This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem. 

    ESE (ESE) 533 Information Store ... A request to write to the file "....edb" at offset 0 (0x0000000000000000) for 32768 (0x00008000) bytes has not completed for 28 second(s). This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem. 

    ExchangeStoreDB 153 At '4/25/2017 9:06:26 AM' the Exchange store database ... copy on this server detected a Hung IO and logged an event. For more details about the failure, consult the Event log on the server for other storage and "ExchangeStoreDb" events.

    The SAN itself usually shows around 10 ms latency for these volumes; the Hyper-V host usually shows around 20 ms latency for the database volumes with few spikes up to 100 ms (and similar for the 'virtual storage device'/ VHDX file). However, in the VMs there are numerous log entries stating that reading or writing small amounts of data took 80 or more seconds. Task Manager in the VMs sometimes shows incredible latencies:

    The virtual Exchange servers have three volumes:

    C: OS (NTFS)
    D: 5 Databases (ReFS)
    E: 5 Databases (ReFS)
    F: Logs (ReFS)

    What could explain the gigantic latency difference between the virtual storage device on the Hyper-V host and the disk within the VM? 


    Wednesday, April 26, 2017 3:33 PM

Answers

  • Seems like it was a hardware issue. 

    https://social.technet.microsoft.com/Forums/en-US/db034a9e-f8eb-4e90-9b3c-851ddaca6a82/poor-disk-performance-ese-errors-on-exchange-2016-on-hyperv?forum=Exch2016GD

    • Marked as answer by svhelden Thursday, October 26, 2017 11:56 AM
    Thursday, October 26, 2017 11:56 AM

All replies

  • For Exchange configuration and performance issues, you will find more experts in an Exchange forum, such as https://social.technet.microsoft.com/Forums/en-US/home?forum=Exch2016GD.  Only trying to help you find the best resources for you.

    tim

    Thursday, April 27, 2017 2:07 PM
  • Yes, thanks, posted there today. 

    Since it seems to affect only Exchange and nothing but Exchange, it may be an Exchange, not a Hyper-V issue. But of course, no one really knows ;) ...

    Thursday, April 27, 2017 2:30 PM
  • Hi Sir,

    Have you disabled "integrity feature" :

    http://www.itprocentral.com/configuring-refs-volumes-in-exchange-server-2016/

    Also ,please install latest updates for that exchange server .

    Best Regards,

    Elton


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Saturday, April 29, 2017 4:55 AM
    Moderator

  • Have you disabled "integrity feature" :


    Ahhhhhhhhhhhhhh .. Probably not. Thanks for that hint!

    Do I see correctly that I can't disable it afterwards? So that I would have to create a new volume and copy everything over ... 

    (And .. how can I actually find out if Integrity Streams were enabled during formatting ...)
    • Edited by svhelden Saturday, April 29, 2017 5:52 AM
    Saturday, April 29, 2017 5:51 AM
  • Seems like it was a hardware issue. 

    https://social.technet.microsoft.com/Forums/en-US/db034a9e-f8eb-4e90-9b3c-851ddaca6a82/poor-disk-performance-ese-errors-on-exchange-2016-on-hyperv?forum=Exch2016GD

    • Marked as answer by svhelden Thursday, October 26, 2017 11:56 AM
    Thursday, October 26, 2017 11:56 AM