none
High RPC Average Latency on Exchange Mailbox server

    Question

  • I've run Perfmon on my Exchange 2007 Mailbox server for 24 hours and the results show that the RPC Average Latency is about 80ms on average. I understand this should be 50ms at most.

    How can I find out what's causing this?
    Friday, January 21, 2011 8:56 PM

Answers

  • Chuck_P2101:I've run Perfmon on my Exchange 2007 Mailbox server for 24 hours and the results show that the RPC Average Latency is about 80ms on average. I understand this should be 50ms at most. How can I find out what's causing this?


    MSExchangeIS\RPC Averaged Latency should not be higher than 25 ms on average.
     
    Monitoring Without System Center Operations Manager
    http://technet.microsoft.com/en-us/exchange/bb201720(EXCHG.80).aspx

    This document details the most important counters to monitor and their threshold values. I've used it quite a lot in one environment where we don't have SCOM 2007 and monitoring of Exchange with HP OpenView isn't properly implemented yet. It should guide you to track down other potential issues and the root cause.

    Personally I prefer using SCOM 2007, which I've been so lucky to implement at a customer and has developed to become one of my late obsessions. Btw, should you happen to use BlackBerry devices?


    MCTS: Messaging | MCSE: S+M
    • Marked as answer by Chuck_P2101 Tuesday, January 25, 2011 8:05 PM
    Saturday, January 22, 2011 10:53 PM
  • On Sun, 23 Jan 2011 20:34:27 +0000, Chuck_P2101 wrote:
     
    >Going forward, when using Perfmon to analyse Exch performance, is that a valid counter to use, or should we concentrate on the "MSExchangeIS Client (*)\RPC Average Latency' one?
     
    "MSExchangeIS\RPC Averaged Latency" if you're interested in the
    overall health of the server. But looking at both isn't a bad idea.
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    • Marked as answer by Chuck_P2101 Tuesday, January 25, 2011 8:05 PM
    Sunday, January 23, 2011 8:42 PM

All replies

  • On Fri, 21 Jan 2011 20:56:08 +0000, Chuck_P2101 wrote:
     
    >I've run Perfmon on my Exchange 2007 Mailbox server for 24 hours and the results show that the RPC Average Latency is about 80ms on average. I understand this should be 50ms at most. How can I find out what's causing this?
     
    What other counters hae you collected?
     
    A guess would be your storage system.
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    Saturday, January 22, 2011 2:35 AM
  • Greetings Rich,

    I haven't collected any other counters yet, what sort of counters should I be looking at in your opinion, especially to find out whether it's storage related?

    Saturday, January 22, 2011 12:16 PM
  • On Sat, 22 Jan 2011 12:16:12 +0000, Chuck_P2101 wrote:
     
    >
    >
    >Greetings Rich,
    >
    >I haven't collected any other counters yet, what sort of counters should I be looking at in your opinion, especially to find out whether it's storage related?
     
    Use this to analyze the perfmon log files (you'll need a couple of
    prerequisites, too):
    http://pal.codeplex.com/
     
    Use this to set up the counters and gather the information:
    http://blogs.technet.com/b/mikelag/archive/2010/07/09/exchange-2007-2010-performance-data-collection-script.aspx
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    Saturday, January 22, 2011 4:33 PM
  • Chuck_P2101:I've run Perfmon on my Exchange 2007 Mailbox server for 24 hours and the results show that the RPC Average Latency is about 80ms on average. I understand this should be 50ms at most. How can I find out what's causing this?


    MSExchangeIS\RPC Averaged Latency should not be higher than 25 ms on average.
     
    Monitoring Without System Center Operations Manager
    http://technet.microsoft.com/en-us/exchange/bb201720(EXCHG.80).aspx

    This document details the most important counters to monitor and their threshold values. I've used it quite a lot in one environment where we don't have SCOM 2007 and monitoring of Exchange with HP OpenView isn't properly implemented yet. It should guide you to track down other potential issues and the root cause.

    Personally I prefer using SCOM 2007, which I've been so lucky to implement at a customer and has developed to become one of my late obsessions. Btw, should you happen to use BlackBerry devices?


    MCTS: Messaging | MCSE: S+M
    • Marked as answer by Chuck_P2101 Tuesday, January 25, 2011 8:05 PM
    Saturday, January 22, 2011 10:53 PM
  • > MSExchangeIS\RPC Averaged Latency should not be higher than 25 ms on average.

    Thanks Jon-Alfred. I was reading from http://technet.microsoft.com/en-us/library/aa998266(EXCHG.80).aspx where it stated that 50ms was the general number. The counter I was using was "MSExchangeIS\RPC Averaged Latency". Yet in the link you provided, it does indeed say that the counter should be 25ms or less, and it's the "MSExchangeIS Client (*)\RPC Average Latency" (which shows the breakdown by client type) that can be 50ms. Which begs the questions 

    - Which counter should I be using?

    - Surely the two counters mean the same thing, but the latter one shows by client? I don't understand how the two values can be different if that *is* the case?

    Many thanks, 

    Sunday, January 23, 2011 10:46 AM
  • On Sun, 23 Jan 2011 10:46:22 +0000, Chuck_P2101 wrote:
     
    >
    >
    >> MSExchangeIS\RPC Averaged Latency should not be higher than 25 ms on average.
    >
    >Thanks Jon-Alfred. I was reading from http://technet.microsoft.com/en-us/library/aa998266(EXCHG.80).aspx where it stated that 50ms was the general number. The counter I was using was "MSExchangeIS\RPC Averaged Latency". Yet in the link you provided, it does indeed say that the counter should be 25ms or less, and it's the "MSExchangeIS Client (*)\RPC Average Latency" (which shows the breakdown by client type) that can be 50ms. Which begs the questions
    >
    >- Which counter should I be using?
     
    That depends on what you're measuring. If "MSExchangeIS\RPC Averaged
    Latency" is under 25ms then the server's doing okay -- but you're
    measuring ALL RPC operations and reporting the *average*. If there a
    gazillions of operations that run on the server that aren't associated
    with *users* they may complete very quickly and the average value will
    be low. That doesn't mean the individual protocols aren't a problem,
    or that there's not a problem with really large mailboxes and users
    working in non-cached mode casuing problems.
     
    >- Surely the two counters mean the same thing, but the latter one shows by client?
     
    By client *protocol*, not be individual user.
     
    >I don't understand how the two values can be different if that *is* the case?
     
    Does this help?
    http://technet.microsoft.com/en-us/library/bb201689(EXCHG.80).aspx
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    Sunday, January 23, 2011 5:07 PM
  • So the initial Perfmon I ran was using the "MSExchangeIS\RPC Averaged Latency" counter and this showed an average value of 80ms.

    Bearing in mind your comment that this is only an average for ALL RPC operations, would you actually recommend we use this counter at all? Or should we be using the ISClient one instead? We don't run ActiveSync, IMAP, POP etc, only OWA and MAPI. Should I be running MSExchangeIS Client(*)\RPC Operations/sec as well for example to show which client is hitting the mailbox server the hardest?

    Basically, I'm not sure if I am using the correct Perfmon counter and if I actually have a problem or not!

     

    Sunday, January 23, 2011 5:51 PM
  • On Sun, 23 Jan 2011 17:51:50 +0000, Chuck_P2101 wrote:
     
    >So the initial Perfmon I ran was using the "MSExchangeIS\RPC Averaged Latency" counter and this showed an average value of 80ms.
     
    If it's that high you have a problem. You just don't know where it is.
    That's why I gave you references to PAL and PerfWiz. You can't
    troubleshoot performance problems in a vacuum.
     
    >Bearing in mind your comment that this is only an average for ALL RPC operations, would you actually recommend we use this counter at all? Or should we be using the ISClient one instead? We don't run ActiveSync, IMAP, POP etc, only OWA and MAPI. Should I be running MSExchangeIS Client(*)\RPC Operations/sec as well for example to show which client is hitting the mailbox server the hardest?
    >
    >Basically, I'm not sure if I am using the correct Perfmon counter and if I actually have a problem or not!
     
    You have a problem. Use PerfWiz to set up the necessary trace logging
    and PAL to generate the reports. Fix the stuff that PAL flags.
     
    FYI, using ExPBA (it's in the EMC "Toolbox") is something you should
    do, too. Fix the problems it points out.
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    Sunday, January 23, 2011 6:45 PM
  • Will do.

    Going forward, when using Perfmon to analyse Exch performance, is that a valid counter to use, or should we concentrate on the "MSExchangeIS Client (*)\RPC Average Latency' one?

    Sunday, January 23, 2011 8:34 PM
  • On Sun, 23 Jan 2011 20:34:27 +0000, Chuck_P2101 wrote:
     
    >Going forward, when using Perfmon to analyse Exch performance, is that a valid counter to use, or should we concentrate on the "MSExchangeIS Client (*)\RPC Average Latency' one?
     
    "MSExchangeIS\RPC Averaged Latency" if you're interested in the
    overall health of the server. But looking at both isn't a bad idea.
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    • Marked as answer by Chuck_P2101 Tuesday, January 25, 2011 8:05 PM
    Sunday, January 23, 2011 8:42 PM