none
ISCSI write performance RRS feed

  • Question

  • I have now been using iSCSI for over a year and have had very mixed results. Great read performance but deceptively slow write performance especially for large file copies to iSCSI targets. I say deceptive because, depending on the amount of RAM in the initiating system the first part of copy almost seems to zoom along at wire speed but as total memory in system gets filled the write transfer rate drops. So a file copy of 2GB on a 4GB 2008 server may complete with a transfer rate of about 70MB/s (fair performance) but an 8gb file will start at almost 120MB/s but finish at 4MB/s. Task manager shows quite clearly the relationship between memory and file size. I carried out a similar test on a sstem with 48GB of memory and copied a 30GB file to an ISCSI target. The copy seemed to complete at a reported rate of about 80MB/s but the actual file was not written out from memory for another 10-15 mins. Contrast this with read performance where i can get almost 1Gb/s throughput and never drops below 48MB/s for file copy from iscsi target to local SATA drive. So i am sure the network is working fine and can sustain the throughput when required. So why is write performance so poor? No resource other than memory on the host machine is stressed out and the utilization on SAN remain similarly unstressed though CPU does spike to 100% for about a sec then drops to 20% for a few secs before cycle is repeated till entire file is written.

    Servers are all 2008 r2 and a Netapp SAN that has 20 15K SAS drives though also tested against a freenas build with a 6 disk (SATA) raid 5 solution. All netwrok infrastructure is Gigabit and have tried both Pwoerconnect and Procurve switches I have pretty much tried most iterations of network setings that often get quoted in various forums and whitepapers with no perceived improvement in write performance. (disable autotuning, enable/disable jumbo frames, disable nagle's algorithm, uncheck all unused services on the NIC, dedicated Gigabit switches, multiple connections per session and other such stuff).

    So perhaps I am chasing some mythical performance that is just not possible with my setup or i have missed something. i must add that i had done some previous testing against HP's AIO Storage server and the iscsi write performance against this box was far better than both the NetApp san and freenas box that i am using.

    Reason why i want to try and resolve this is so i can use large iscsi targets as storage pools for disk to disk backups. I already run multiple low stress Hyper-v vms off the SAN.

    Thursday, November 11, 2010 7:07 AM

Answers

  • Sort of. What switches are you using? If they are ProCurve 2810s you may as well get rid of them. In fact if you are using any non iscsi optimized switch you will continue to see all sorts of issues. This was the primary cause of bottlenecks on my setup. I replaced these and it solved a whole host of performance issues that i had been having. Till your question i had not attempted large file copy (had forgotten to test that) and so waited till i had run a test. Anyway results are fair. i do not see the spike in memory utilized as copy progresses. In fact memory flat lined and copy progressed at an average rate of about 35MB/s. So though not quite wire speed at least not deceptive either. I suspect the SAN CPU is the limiting factor since it hovered around 90%. I have abandoned the idea of writing large files to this SAN. From an I/O perspective i have significant improvements where i have even managed 3000+ iops (and i guess this is what these devices are really meant for). Hope that helps.
    • Marked as answer by ns2004 Saturday, June 18, 2011 9:07 PM
    Friday, May 20, 2011 6:45 PM

All replies

  • Hi,

    Here is a similar thread which provided suggestions and articles. You can have a look on it:

    worst performace on iSCSI initiator on Windows 2008

    http://social.technet.microsoft.com/Forums/en/winserverfiles/thread/c98d211a-188e-4f95-aef3-9d251fa08b99


    Shaon Shan| TechNet Subscriber Support in forum| If you have any feedback on our support, please contact tngfb@microsoft.com
    Thursday, November 18, 2010 8:00 AM
    Moderator
  • As per original post I had tried disabling autotuning and it made no difference. Am in conversation with Netapp and will update if we make progress.
    Friday, January 14, 2011 3:51 PM
  • Hello, did you ever find anything out with this?  We're experiencing almost the exact same issues with our NetApp storage and Windows 2008 R2 server.
    Friday, May 20, 2011 3:59 PM
  • Sort of. What switches are you using? If they are ProCurve 2810s you may as well get rid of them. In fact if you are using any non iscsi optimized switch you will continue to see all sorts of issues. This was the primary cause of bottlenecks on my setup. I replaced these and it solved a whole host of performance issues that i had been having. Till your question i had not attempted large file copy (had forgotten to test that) and so waited till i had run a test. Anyway results are fair. i do not see the spike in memory utilized as copy progresses. In fact memory flat lined and copy progressed at an average rate of about 35MB/s. So though not quite wire speed at least not deceptive either. I suspect the SAN CPU is the limiting factor since it hovered around 90%. I have abandoned the idea of writing large files to this SAN. From an I/O perspective i have significant improvements where i have even managed 3000+ iops (and i guess this is what these devices are really meant for). Hope that helps.
    • Marked as answer by ns2004 Saturday, June 18, 2011 9:07 PM
    Friday, May 20, 2011 6:45 PM
  • I have some HP 1810G Procurve switches.  Performance has been sub-par on a new MD3200i.  I'm wondering if this may be an issue.  What switches did you end up purchasing?
    Friday, May 20, 2011 7:13 PM