locked
VSTEST 2008 with load agents not picking up perfcounter correctly until perfmon somehow gets it going RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote

    In our application we have a problem with too high contention rate shown in the .NET CLR LocksAndThreads \ Contention Rate. This issue (about 1400 contentions / sec) was reported fine by my load test using 2 load agents.

     

    Suddenly the contention rate disappeared and showed 0 consistently throughout several load tests. I thought 'great' the developers solved it. However, I found out that VS reports 0 until I point 'perfmon' to the perfcounter on the server.

     

    Then suddenly VS kicks in and shows 850 mill (!!) contentions / sec and then drops to about 1400 / sec which is expected behavior.

     

    Somehow it seems as if VS all of the sudden doesn't initialize this perfcounter correctly but perfmon does.

     

    Any ideas on how this can happen? This sort of questions my belief in all the other perfcounters and I'm not really sure what to believe?

     

    Thanks,

     

    Jacob

    Wednesday, March 5, 2008 3:15 PM

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote

    Perviously my load test started by stating 'Initializing performance counters...' which doesn't appear anymore.

     

    Could this be the issue and why would it all of the sudden not initialize the perfcounters?

     

    Thanks,

     

    Jacob

    Thursday, March 6, 2008 9:01 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Even though it seems as if I'm talking to myself I'll add some more info ;-)

     

    '% Time in Jit' is 87% and contention rate is 0 until I start perfmon. Then '% Time in Jit' drops to 0 instantly and contention rate increases and matches the user load. Strange...

    Friday, March 7, 2008 8:36 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Jacob, sorry for the delay in responding.   The original problem appears to be some sort of bug in either the .NET classes that the VSTS load test uses to read performance counters, or some problem specific to the ".NET CLR ..." performance counter categories.    Are you only seeing this unusual behavior with the ".NET CLR ..." performance counter categories?

     

    I am in contact with members of the CLR team regarding this issue, and will update you when I find out something.

     

    Are you running your load test using a test rig (controller & agents), or just locally in the VSTestHost process?

     

    We've had one other report of this problem from an internal Microsoft user, but then the problem disappeared and he has not been able to reproduce it yet.  

     

    In the Load Test, which Counter Set is mapped to the computer that is having the problem?   Have you made any changes to the counter set, or are you just using the default definition of that counter set?   (I'm just trying to narrow this down to see if maybe I can repro it.)

     

    It might help to get the controller log file (or VSTestHost log file if running locally):

     

    Enabling VSTS Load Test Controller logging


    If running locally, enable logging for the VSTestHost process:

      * Edit <Program Files>\Microsoft Visual Studio 9\Common7\IDE\VSTestHost.exe.config

      * Add the following system.diagnostics section (change the path to VSTestHost.log if you want):
     
      <system.diagnostics>
        <trace autoflush="true" indentsize="4">
            <listeners>
               <add name="myListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="c:\VSTestHost.log" />
            </listeners>
        </trace>
        <switches>
            <!-- You must use integral values for "value".  Use 0 for off, 1 for
                 error, 2 for warn, 3 for info, and 4 for verbose. -->
            <add name="EqtTraceLevel" value="3" />
        </switches>
      </system.diagnostics>


    If running a test on a rig, enable logging for the controller process:

      * Edit <Program Files>\Microsoft Visual Studio 9.0 Team Test Load Agent\LoadTest\QTController.exe.config and change the value in this line from “no” to “yes”:
     
         <add key="CreateTraceListener" value="yes"/>

      * In the same file, also set the value for "EqtTraceLevel" to 4 for verbose logging:

         <add name="EqtTraceLevel" value="4" />

      * Restart the controller service

      * The file "vsttcontroller.log" will be created in the same directory as QTController.exe.config.

    Monday, March 10, 2008 8:23 PM
    Moderator
  • Question
    Sign in to vote
    0
    Sign in to vote

    Jacob, I heard back from the CLR team that there was a bug with similar symptoms in the past that should have been fixed in VS 2008, and the problem was related to permissions.   Can you provide more info on your test rig configuration (I now notice from the thread title that you are using a rig with agents)?     Specifically:

     

    What are the OS and service pack levels and processor types for both the controller and the machine from which you are trying to collect the .NET CLR perf counters?   

     

    What version of the .NET framework is running on the machine where the .NET CLR perf counters are being reported?

     

    Is the user running the controller services an admin on the system under test, or just a member of the Performance Monitor Users group?

     

     

     

    Tuesday, March 11, 2008 4:49 PM
    Moderator
  • Question
    Sign in to vote
    0
    Sign in to vote

    Jacob, here's some more specific info: if the system under test machine that is reporting the CLR perf counters is running .NET Framework 2.0 without SP1, that could explain this problem as you would be running into a known bug that is fixed in .NET Framework 2.0 SP1, in which case you should install .NET Framework 2.0 SP1 on that machine.

     

    Bill

     

     

     

    Tuesday, March 11, 2008 6:08 PM
    Moderator
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi,

     

    I am experiancing issues with many % Time in Jit and Contention/sec threshold warnings when using the Microsoft CRM 4 Performance Toolkit with VS 2005.  I am running a load test, the issue I am experiancing is having an effect on performance and have performed all the recommended optimisations for CRM v4.  I thought it worth posting as this issue does appear similar to the one reported here.

     

    We have acutally upgraded our CPU's to cope with more load however now the bottleneck is elsewhere as the CPU's are avergaing at most 40% - and these counter threshold are being breached - so looks likley this is related to the problem.

     

    Importantly I do have .NET Framework v2.0 SP1 allready installed on test client and application server.

     

    I am planning to log this with MS so will post back if I get anywhere with them - if anyone has any comment or had simliar issues please post :-)

     

    Andy

     

    Wednesday, March 12, 2008 9:51 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

     

    Hi,

     

    This may also be relvant to the issue raised here - I have had problems when starting a load test with performance counters initilising, more often or not it will fail on a counter set - although the actual counter set it fails on varies - only fix i've found so far is to reboot test client and backend servers it is connecting too.  Interestly ran a test last night and made a change to the application server - and counters mentioned in previous post were not reported as breaching thesholds - although a repeat test after reboots - they were logged - alot of them.  So something doesn't seam quite right and at the moment I have doubts over the performance counter collection/reporting as well as the performance of the SUT application...

     

    Andy

    Wednesday, March 12, 2008 10:01 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Andy, it seems like you have one or two different issues than the original post, so could you please re-post these questions on a new thread so that we can track them separately, and not get confused when replies to the different issues are intermixed.   You can certainly use a link to refer back to this thread where you think there are similarities.

     

     

    Wednesday, March 12, 2008 1:02 PM
    Moderator
  • Question
    Sign in to vote
    0
    Sign in to vote

     

    Thats fair enough - will do :-) 

     

    Am still working on this issue now and there are many varibales contributing to the issues i'm havig; I'm going to wait until i've proven more and narowed down somewhat - and will post to the relevant group / new thread if required

     

    Andy

    Wednesday, March 12, 2008 2:33 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

     

    Hi Bill,

     

    Thanks for getting back to me. I just wanted to let you know that I'm out of office and have limited access to the test environment but I'll try to gather the info you requested and enable the verbose logging as soon as I can.

     

    For now I can tell you that we're using .NET 2.0 with SP1. The collection of the perfcounters seemed to work before so I suspect that the issue is due to one of the patches from February...

     

    Everything (1 web server, 2 agents and 1 controller) is running Windows Server 2003 with quad core processors, 4GB ram and they should be up-to-date with the latest patches (unfortunately I can't verify that but will do as soon as I can).

     

    I did modify the default perfcounter mappings but I haven't touched it lately - i.e. I haven't touched it since it was working.

     

    Thanks,

     

    Jacob

    Wednesday, March 12, 2008 4:01 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Jacob, any update on this?   Have you had a chance to verify that the systems have the latest patches?   Are you still seeing the problem?

     

    Bill

     

    Thursday, April 3, 2008 12:58 PM
    Moderator
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi Bill,

     

    Thanks for getting back on this. Actually I started working on this issue yesterday and I have problems reproducing it. I saw it today and restarted the Rig in order to clear the logs. And after that I haven't been able to reproduce it.

     

    The Web Server was patched 1. March with .NET 2.0 SP1 (and all other available security patches) - and I am sure I didn't see this problem beginning of February but I can't tell if this made the difference.

     

    The controller and the agents have had .NET SP1 all along.

     

    I'll keep trying to reproduce the problem and send the log once it happens. For now a workaround seems to be to restart the Rig before the load test.

     

    Thanks,

     

    Jacob

     

    Thursday, April 3, 2008 2:02 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi Bill,

     

    OK - I've had some time to reproduce it now. It seems consistent that I don't have the problem if I restart the entire Rig before each load test.

     

    I now have verbose logs from the controller and the 2 agents where it doesn't pick up the perf counters until I start perfmon. VS kicks in reading the perf counter 'Contention Rate' when I select it in the 'Add counters' dialog of Perfmon.

     

    What should I do with the logs? They are quite big.

     

    Thanks,

     

    Jacob

    Thursday, April 10, 2008 12:28 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    The controller log is probably the most interesting as the controller collects these perf counters and there should be errors in the log.    Perhaps you can find and copy out the part of the controller log around the time that the load test is starting making sure to include anything that looks related to performance counters.   You can email it to me (alias: billbar) (domain: microsoft.com).

     

    Bill

     

    Thursday, April 10, 2008 1:40 PM
    Moderator
  • Question
    Sign in to vote
    0
    Sign in to vote

    Based on the info that you emailed me, which I forwarded to some CLR developers, one of them has a theory as to why this is occurring.   He is going to try to reproduce this later this week, and I'll update this post when I have some more info.

     

    Bill

     

     

    • Unmarked as answer by Jacob T Tuesday, July 13, 2010 1:32 PM
    Tuesday, April 15, 2008 4:01 PM
    Moderator
  • Question
    Sign in to vote
    0
    Sign in to vote
    After having several problems with getting performance counters from remote servers and finding bits and peices of ways to resolve the errors online.  I put together a blog post of all of the troubleshooting techniques for these problems into a single blog at: http://blogs.catapultsystems.com/tlingenfelder/archive/2009/06/18/performance-counters-timeouts-and-load-testing-with-visual-studio-2008.aspx
    Thursday, June 18, 2009 3:25 PM