locked
Windows Clustered Shared Volume IOPS RRS feed

  • Question

  • Quick question. I'm reviewing the IOPS for my CSV, and im confused about the "Cluster CSVFS\Reads/sec" counter. I aligned "Physical Disk\Disk Reads/sec" (for the CSV disk) with "Cluster CSV File System\IO Reads/sec" and "Cluster CSVFS\Reads/sec". The first two align (shows roughly the same value), but "Cluster CSVFS\Reads/sec" is way off the chart. Which is the real number? My assumption is that the "Cluster CSVFS\Reads/sec" value is inaccurate, or does not reflect disk IO (aka IOPS).

    • Edited by JoeOs Tuesday, July 7, 2020 5:09 PM
    Tuesday, July 7, 2020 4:23 PM

All replies

  • Hi JoeOs,

    Below is an article about CSV performance counter, for your reference:

    https://techcommunity.microsoft.com/t5/failover-clustering/cluster-shared-volume-performance-counters/ba-p/371980

    Thanks for your time!

    Best Regards,

    Anne


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, July 8, 2020 7:50 AM
  • Hi Anne

    Thanks for the response. I already reviewed that article prior to posting. There is no mention about the "Cluster CSVFS\Reads/sec" counter

    Wednesday, July 8, 2020 6:18 PM
  • Actually, that value is mentioned several times and is used in several examples.  But I will admit that it is not explained very well.

    Could you provide us with a description of your physical environment and a typical report that shows the values "off the charts"?  That might help us better understand your environment.  Also, a description of the type of workload might be beneficial.


    tim

    Thursday, July 9, 2020 1:34 PM
  • Ah, you're right, it is mentioned. Nonetheless, its still not adding up. By off the chart, i mean like the image below (highlighted yellow). As you can see, the rest align perfectly well, and i have no latency nor using redirect. The 101k below is an occasional spike. The avg over 16 minutes is 12k Vs the rest "Reads/Sec" which are roughly around 60

     
    Thursday, July 9, 2020 7:53 PM
  • What is your physical configuration?  How are they configured to CSVs?

    tim

    Friday, July 10, 2020 1:37 PM
  • Its a two node cluster. Each has two 10Gb NICs. The NICs are configured as a converged network and load balanced as a hyper-v switch (as in no LACP config on the switch). See config below from host2. The storage traffic is converged over NICs PrivateA and PrivateB. I have an MPIO ISCSI connection to the storage 

    C:\>netstat -ano | find "3260"
      TCP    10.190.228.132:49736   192.168.122.73:3260     ESTABLISHED     3468
      TCP    10.190.228.132:49737   192.168.122.71:3260     ESTABLISHED     3468
      TCP    10.190.228.132:49738   192.168.122.71:3260     ESTABLISHED     3468
      TCP    10.190.228.132:49742   192.168.122.20:3260     ESTABLISHED     3468
      TCP    10.190.228.132:49745   192.168.122.20:3260     ESTABLISHED     3468
      TCP    10.190.228.133:49739   192.168.122.73:3260     ESTABLISHED     3468
      TCP    10.190.228.133:49741   192.168.122.71:3260     ESTABLISHED     3468
      TCP    10.190.228.133:49743   192.168.122.23:3260     ESTABLISHED     3468
      TCP    10.190.228.133:49744   192.168.122.20:3260     ESTABLISHED     3468
    

    PS C:\> Get-NetAdapter | Sort-Object Name | select Name,InterfaceDescription,Linkspeed Name InterfaceDescription LinkSpeed ---- -------------------- --------- Private-A Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T 10 Gbps Private-B Intel(R) Ethernet Converged Network Adapter X710-T 10 Gbps Public-A Intel(R) Ethernet Converged Network Adapter X710-T #2 10 Gbps Public-B Intel(R) Ethernet Converged Network Adapter X710-T #3 10 Gbps PublicTeam Microsoft Network Adapter Multiplexor Driver 20 Gbps vEthernet (Heartbeat) Hyper-V Virtual Ethernet Adapter #5 20 Gbps vEthernet (LiveMigrate) Hyper-V Virtual Ethernet Adapter #4 20 Gbps vEthernet (LBvSwitch01) Hyper-V Virtual Ethernet Adapter 20 Gbps vEthernet (Public vSwitch) Hyper-V Virtual Ethernet Adapter #7 10 Gbps vEthernet (Storage01) Hyper-V Virtual Ethernet Adapter #2 20 Gbps vEthernet (Storage02) Hyper-V Virtual Ethernet Adapter #3 20 Gbps vEthernet (VMNetwork) Hyper-V Virtual Ethernet Adapter #6 20 Gbps PS C:\> Get-VMSwitch Name SwitchType NetAdapterInterfaceDescription ---- ---------- ------------------------------ Public vSwitch External Microsoft Network Adapter Multiplexor Driver ApplianceNet Private LBvSwitch01 External Teamed-Interface PS C:\> Get-VMSwitchTeam LBvSwitch01 | fl Name : LBvSwitch01 Id : a4b43ca1-a37c-4174-8210-c330405a2856 NetAdapterInterfaceDescription : {Intel(R) Ethernet Controller X710/X557-AT 10GBASE-T, Intel(R) Ethernet Converged Network Adapter X710-T} TeamingMode : SwitchIndependent LoadBalancingAlgorithm : Dynamic PS C:\> Get-VMNetworkAdapter -ManagementOS | Sort-Object name Name IsManagementOs VMName SwitchName MacAddress Status IPAddresses ---- -------------- ------ ---------- ---------- ------ ----------- Heartbeat True LBvSwitch01 00155DE4060A {Ok} LiveMigrate True LBvSwitch01 00155DE40609 {Ok} LBvSwitch01 True LBvSwitch01 AC1F6B0DC882 {Ok} Public vSwitch True Public vSwitch AC1F6B0DC885 {Ok} Storage01 True LBvSwitch01 00155DE40607 {Ok} Storage02 True LBvSwitch01 00155DE40608 {Ok} VMNetwork True LBvSwitch01 00155DE4061F {Ok} PS C:\> Get-VMNetworkAdapterTeamMapping -ManagementOS | Sort-Object NetAdapterName NetAdapterName : Private-A NetAdapterDeviceId : {B69AB1D8-E1C3-41BA-A81B-65BE63273678} ParentAdapter : VMInternalNetworkAdapter, Name = 'Storage01' IsTemplate : False CimSession : CimSession: . ComputerName : WAHOST2 IsDeleted : False NetAdapterName : Private-B NetAdapterDeviceId : {F93A1019-8900-4FD7-9903-8C6ABF831F85} ParentAdapter : VMInternalNetworkAdapter, Name = 'Storage02' IsTemplate : False CimSession : CimSession: . ComputerName : WAHOST2 IsDeleted : False PS C:\> Get-VMNetworkAdapterVlan -ManagementOS VMName VMNetworkAdapterName Mode VlanList ------ -------------------- ---- -------- LiveMigrate Access 2616 Public vSwitch Untagged Heartbeat Access 2618 VMNetwork Access 2615 Storage01 Access 2617 LBvSwitch01 Untagged Storage02 Access 2617

    PS C:\> Get-MPIOAvailableHW

    VendorId ProductId        IsMultipathed   IsSPC3Supported BusType
    -------- ---------        -------------   --------------- -------
    NETAPP   LUN C-Mode       True            True            iSCSI



    • Edited by JoeOs Friday, July 10, 2020 8:55 PM
    Friday, July 10, 2020 8:50 PM
  • I still don't see your CSV configuration.  You did an excellent job of defining your network configuration, but I don't see how you have configured the CSVs.  It looks like you have MPIO configured to a NetApp appliance, but I don't see how many CSVs you have defined on the NetApp.  It also looks like you have defined network access for the VMs to access ISCSI storage (which I would assume would be to the same NetApp appliance).  If you are accessing the iSCSI storage directly from the VMs, that would imply the VM LUNs are not defined as CSVs, and you have a different set of LUNs specific to the VMs.

    Based on those assumptions, I would agree that the CSV IOs look out of whack.  You might have to open a support case to work through the details of why your particular configuration is generating those sorts of numbers.


    tim

    Saturday, July 11, 2020 11:00 PM
  • Hi Tim,

    I have a single CSV Lun on which i configured a single CSV. I then have other (non CSV) clustered luns attached to the VM (from the node perspective) as a clustered resource. The VMs do not have direct access to the ISCSI network, nor are they directly accessing the ISCSI storage. PhysicalDisk 3 (showing in my previous post) is the only CSV disk

    PS C:\> Get-ClusterSharedVolume | fl *
    
    
    Id               : 490212ca-138b-4cb0-adc8-a94fbaf4a8a8
    Name             : Cluster Disk 2
    OwnerNode        : WAHost1
    SharedVolumeInfo : {C:\ClusterStorage\EduranceStorage}
    State            : Online

    Monday, July 13, 2020 4:04 PM
  • Let me see if I understand your configuration.  You have a single CSV defined in your cluster yet you also have some other clustered disks that are defined simply as cluster resources, but were not converted to CSV.  I will assume you have the VM system disks on the CSV.

    This is the part that is confusing.  "I then have other (non CSV) clustered luns attached to the VM (from the node perspective) as a clustered resource. The VMs do not have direct access to the ISCSI network, nor are they directly accessing the ISCSI storage."  It sounds like you have attached one or more clustered disks to different VMs (obviously two different VMs cannot attach to the same clustered disk).  So you are using the iSCSI connection at the host level to provide unshared access to the LUN by the VM, instead of using a direct iSCSI connection from the VM to the LUN.

    I've never encountered a cluster configured like this.  Due to the limitations of configuring a host LUN to a VM, I have always seen a VM have its own iSCSI connection to the LUN.  And even that is rare because it has its own limitations.  It is so much cleaner and easier to manage to have all LUNs on the host as CSVs and then VMs access VHDs.  CSV was designed specifically for use by VMs in a cluster.  I'm not saying the configuration is not valid, it is just that I have never seen one like this and cannot see any benefit to such a configuration.

    I don't know if the strange configuration is causing the anomaly in the numbers.  Since I have never seen a configuration like this before, I would suggest you open a support call with Microsoft to see if they can work through your configuration.


    tim

    Tuesday, July 14, 2020 1:18 PM
  • Hi Tim,

    I would think its a perfectly normal config. Its actually just one such VM (a SQLServer) with its own dedicated LUN. The rest of the VMs (VHDs) storage are all on the CSV. The reason we put the SQL on its own dedicated LUN was to prevent it from suffering any potential CSV performance issues. The reason we connected it from the host is because a)the LUN is hosting the VHDs themselves, including the OS drive b)its the host that has a proper MOIO to the storage. See config details below. I can easily move the storage back to the CSV (Move-VMStorage), but isn't a dedicated LUN giving the SQL server greater performance?

    Any why would any of this have anything to do with the CSVs high IOPS?

    About calling Microsoft's PSS... where shall we begin? Suffice to say that i came to the community as option B. PSS is useless (mildly said). 

    PS C:\> Get-ClusterResource | Where {$_.ResourceType.Name -eq "Physical Disk"}
    
    Name           State  OwnerGroup    ResourceType
    ----           -----  ----------    ------------
    Cluster Disk 1 Online SQLSrvr   Physical Disk
    Cluster Disk 3 Online Cluster Group Physical Disk
    PS C:\> Get-Volume -FileSystemLabel ClustLun1 | fl *
    
    
    OperationalStatus     : OK
    HealthStatus          : Healthy
    DriveType             : Fixed
    FileSystemType        : NTFS
    DedupMode             : NotAvailable
    ObjectId              : {1}\\DataCntr\root/Microsoft/Windows/Storage/Providers_v2\WSP_Volume.ObjectId="{4eb1f4e9-06c8-44a7-aa1e-ab9143a04cd3}:VO:\\?\Volume{0a65bfff-f2ab-46bd-928f-4f8196c2d033}\"
    PassThroughClass      :
    PassThroughIds        :
    PassThroughNamespace  :
    PassThroughServer     :
    UniqueId              : \\?\Volume{0a65bfff-f2ab-46bd-928f-4f8196c2d033}\
    AllocationUnitSize    : 4096
    DriveLetter           : S
    FileSystem            : NTFS
    FileSystemLabel       : ClustLun1
    Path                  : \\?\Volume{0a65bfff-f2ab-46bd-928f-4f8196c2d033}\
    Size                  : 1099374260224
    SizeRemaining         : 270188666880
    PSComputerName        :
    CimClass              : ROOT/Microsoft/Windows/Storage:MSFT_Volume
    CimInstanceProperties : {ObjectId, PassThroughClass, PassThroughIds, PassThroughNamespace...}
    CimSystemProperties   : Microsoft.Management.Infrastructure.CimSystemProperties

    PS C:\> Get-PhysicalDisk
    
    FriendlyName      SerialNumber                     CanPool OperationalStatus HealthStatus Usage            Size
    ------------      ------------                     ------- ----------------- ------------ -----            ----
    NETAPP LUN C-Mode 80GoL]NDQwXP                     False   OK                Healthy      Auto-Select      1 TB
    AVAGO MR9361-8i   000dd7d26bdf460023b0dc870db00506 False   OK                Healthy      Auto-Select  14.55 TB
    Msft Virtual Disk                                  False   OK                Healthy      Auto-Select    500 MB
    NETAPP LUN C-Mode 80911?JmSUjM                     False   OK                Healthy      Auto-Select  20.22 GB
    NETAPP LUN C-Mode 80GoL]NDQuBT                     False   OK                Healthy      Auto-Select    5.5 TB
    AVAGO MR9361-8i   00895c3f2f78b3e022b0dc870db00506 False   OK                Healthy      Auto-Select 893.75 GB
    PS C:\> Get-Volume
    
    DriveLetter FileSystemLabel FileSystem DriveType HealthStatus OperationalStatus SizeRemaining       Size
    ----------- --------------- ---------- --------- ------------ ----------------- -------------       ----
    F           New Volume      NTFS       Fixed     Healthy      OK                     449.5 MB  465.94 MB
                System Reserved NTFS       Fixed     Healthy      OK                    152.79 MB     500 MB
    C                           NTFS       Fixed     Healthy      OK                    783.47 GB  893.26 GB
    E           DFS             NTFS       Fixed     Healthy      OK                      2.99 TB       3 TB
    D                           NTFS       Fixed     Healthy      OK                      3.34 TB   11.55 TB
    S           ClustLun1       NTFS       Fixed     Healthy      OK                    251.63 GB 1023.87 GB
                CSV1            CSVFS      Fixed     Healthy      OK                      1.54 TB     5.5 TB
                Quorum          NTFS       Fixed     Healthy      OK                     20.01 GB    20.1 GB

    PS C:\> Get-VMHardDiskDrive sqlsrvr
    
    VMName      ControllerType ControllerNumber ControllerLocation DiskNumber Path
    ------      -------------- ---------------- ------------------ ---------- ----
    SQLSrvr SCSI           0                0                             S:\SQLSrvr\Virtual Hard Disks\SQLSrvrBoot.vhdx
    SQLSrvr SCSI           0                2                             S:\SQLSrvr\Virtual Hard Disks\SQLData.vhdx
    SQLSrvr SCSI           0                4                             S:\SQLSrvr\Virtual Hard Disks\DynamicsUATDB.vhdx




    • Edited by JoeOs Tuesday, July 14, 2020 5:58 PM
    Tuesday, July 14, 2020 5:35 PM
  • " but isn't a dedicated LUN giving the SQL server greater performance?"

    Not likely.  According to all the testing I have seen, there have been very few instances in which a dedicated LUN has provided better performance than a VHD.  And if there was a performance benefit, it came about in very select environments where one was saturating disk performance, which most likely could have been mitigated by simply moving the VHD to another disk. The engineers worked very diligently to ensure that the CSV would provide all the benefits of virtualization without loss of performance.  Using a dedicated LUN to a VM negates some of the benefits of virtualization.  So by using a dedicated LUN you lose some benefits of virtualization, increase the management requirements of the VM, and receive no or very slight performance benefits.

    "Any why would any of this have anything to do with the CSVs high IOPS?"

    I don't know.  That is why I suggest you get more experts involved who can dig into this in a manner that is not possible through a forum.

    "PSS is useless (mildly said)"

    My experience with PSS has been completely different.  I have been involved in countless situations where their support and expertise were critical to getting situations resolved.  I will grant that it is sometimes frustrating to get through the first level of support.


    tim

    Wednesday, July 15, 2020 1:41 PM
  • Hi JoeOs,

    >About calling Microsoft's PSS... where shall we begin? 

    If you want to open a case with MS for better support, you may refer to the following link:

    https://support.microsoft.com/en-us/help/4051701/global-customer-service-phone-numbers

    Thanks for your time!

    Best Regards,

    Anne


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Tuesday, July 21, 2020 2:20 AM