locked
APM - Check Client-Side Monitoring Compability Fails (no task output) RRS feed

  • Question

  • Trying to do some client-side monitoring on a .NET app in a POC and running into problem. Need some assistance with next steps to troubleshoot.

    OpsMgr 2012 installed and updated with CU2.

    Server-Side APM is working and generating exceptions and performance alerts in the OpsMgr Console and AppDiagnostics. However, i'm not seeing any client-side alerts (and I've put the Client-Side thresholds very low for the POC). I did restart iis before trying to use client-side monitoring. In addition, when I try to run the 'Check Client-Side Monitoring Compability' task on one of the webservers, it fails with 'No output available' after several minutes.

    I've checked the event logs and don't see any relevant events during the time of the failure.

    I've checked the IP Restrictions for CSM and they are all good. I'd really like to troubleshoot and get the client-side monitoring check to work as a starting point.

    Any help? Thanks!


    gaurhoth

    Tuesday, August 21, 2012 12:50 PM

Answers

  • Hi Gauroth,

    let's separate the issues:

    1) Compatibility task not running

    I would most likely look on the agent's Operations Manger and Application event logs first - the check runs a task on the agent... it might take quite some time, but maybe it fails somewhere (permissions??). Anyway, if you go on the agent, you can launch the utility manually and see what it says? It' located under

    c:\Program Files\System Center Operations Manager\Agent\APMDOTNETCompatibilityCheck

     

    2) Client-Side events not showing. First of all, verify that configuration has been actually *applied* before you do IISRESET. Look at this thread http://social.technet.microsoft.com/Forums/en-US/scomapm/thread/65766a54-53e8-4c2d-95c6-cbb6f5407bd9 - there are separate events written for server AND for client-side monitoring configuration infrastructure... you should be able to follow the configuration flow from the log on the agent.

    Second, verify that /CSMCollector web application has been created in your website, alongside with the monitored app.

    Third, look at the pages of the site (browse to them and look at the source HTML) - you should see the client-side scripts being injected:

    CSM Scripts

    If they are NOT injected, then you most likely have a problem with the filter.

    If they ARE injected, they might be having problems posting back to CSMCollector. I typically use IE's developer tools (the ones you get by hitting F12) to capture network traffic. if it is posting correctly, you should see this at the end:

    CSM Collector Post

    If the scripts were injected but are not posting... probably the threshold isn't surpassed.

    If it is trying to post, but it is not getting a 200 (but maybe it is getting a 40x or 50x error)... then there is something wrong with the CSMCollector's configuration: .Net framework version, authentication, permission, etc...

    Let me know what you find from the steps above.

    HTH,

    Dan

     

     

     

     

     

    Tuesday, August 21, 2012 3:45 PM

All replies

  • Hi Gauroth,

    let's separate the issues:

    1) Compatibility task not running

    I would most likely look on the agent's Operations Manger and Application event logs first - the check runs a task on the agent... it might take quite some time, but maybe it fails somewhere (permissions??). Anyway, if you go on the agent, you can launch the utility manually and see what it says? It' located under

    c:\Program Files\System Center Operations Manager\Agent\APMDOTNETCompatibilityCheck

     

    2) Client-Side events not showing. First of all, verify that configuration has been actually *applied* before you do IISRESET. Look at this thread http://social.technet.microsoft.com/Forums/en-US/scomapm/thread/65766a54-53e8-4c2d-95c6-cbb6f5407bd9 - there are separate events written for server AND for client-side monitoring configuration infrastructure... you should be able to follow the configuration flow from the log on the agent.

    Second, verify that /CSMCollector web application has been created in your website, alongside with the monitored app.

    Third, look at the pages of the site (browse to them and look at the source HTML) - you should see the client-side scripts being injected:

    CSM Scripts

    If they are NOT injected, then you most likely have a problem with the filter.

    If they ARE injected, they might be having problems posting back to CSMCollector. I typically use IE's developer tools (the ones you get by hitting F12) to capture network traffic. if it is posting correctly, you should see this at the end:

    CSM Collector Post

    If the scripts were injected but are not posting... probably the threshold isn't surpassed.

    If it is trying to post, but it is not getting a 200 (but maybe it is getting a 40x or 50x error)... then there is something wrong with the CSMCollector's configuration: .Net framework version, authentication, permission, etc...

    Let me know what you find from the steps above.

    HTH,

    Dan

     

     

     

     

     

    Tuesday, August 21, 2012 3:45 PM
  • 1) Compatibility task not running

    I was able to run the AppCompatibilityCheckUtility locally on the box and it completed. It took over 20 minutes. So I am assuming the task is timing out when run from the OpsMgr console.. I did have numerous warnings centered around end() and flush(), but my understanding of the tests indicates that injection should still work (even if the end result is the app fails to perform as expected). Next step, is to get the injection working and see if the app code is really going to cause a problem.

    2) Client-Side events not showing. 

    I'm not seeing the injected javascript. I verified Event Numbers (including 34217, 34218, 34243, 34242, 34214, 34208, and 34004) in the APM Agent Events. The Events seem to indicate everything went successfully with configuring CSM. Looking in IIS on the web server, I see the CSMCollector app.

    I waited until the 'IIS Restart is required' alert flipped in the opsmgr console and then restarted IIS. 

    The IP Filter is empty for CSM for this initial test/poc. I also left all target groups empty in the initial wizard. 

    Hitting the site and inspecting the page source is void of injected javascript that you described. Any other ideas to troubleshoot the injection?

    Thanks for the help!


    gaurhoth

    Tuesday, August 21, 2012 6:33 PM
  • Are the pages you are hitting *.aspx pages?

    Are you using Internet Explorer?

    Tuesday, August 21, 2012 7:46 PM
  • Yes. The pages are .aspx pages. But when I look in the folder structure, I notice the .aspx pages have the following text in them "This is a marker file generated by the precompilation tool, and should not be deleted!". Would that make any difference?

    I am using IE.


    gaurhoth

    Tuesday, August 21, 2012 7:57 PM
  • I am verifying about the precompiled aspx pages... theoretically we should work in that case, but I am not sure if this is a thoroughly tested scenario. It would be great if you could try if a NON-precompiled app makes a difference?

    Just to make sure something else is not at play here - what version of Internet Explorer?

    Tuesday, August 21, 2012 8:30 PM
  • Tried IE8 and IE9 on a couple different computers (with no AV, Proxies, or anything else that would strip javascript in between).

    I'm not sure we run the app that we want to monitor anywhere non-precompiled, but I'll see what I can round up.



    gaurhoth

    Tuesday, August 21, 2012 8:44 PM
  • Just few notes / comments on Client-side monitoring -

    1) Currently only applications with "Full" trust level can be monitored by CSM. Apps with "High", "Medium" or "Low" trust levels will not be monitored (in that case, Injector doesn't work). To verify trust level - just run IIS Manager, select your app and navigate to ".NET Trust Levels" view.

    2) To troubleshoot this issue further, you might want to turn on tracing for Agent.
    Just click "StartTracing.cmd" (from C:\Program Files\System Center Operations Manager\Agent\Tools folder), then do IIS Restart, and open some pages from the monitored app. After that click "StopTracing.cmd" and "FormatTracing.cmd" to get plain text "TracingGuidsAPM.log" file. It should show you what is going on and where the problem is.


    Best regards, Mike

    Tuesday, August 21, 2012 9:50 PM
  • Getting a little closer thanks to all of the help. 

    The app in question is using FULL Trust and I have actually found that CSM javascript is injected to a couple pages. But the vast majority of pages don't have the javascript. One thing I noticed is that in the pages where CSM Javascript was found did not have any other javascript in the <head> tag area. But most of our apps pages have our own javascript in the <head> area. I'm not a developer, so I don't know what mechanism the engineering team is using... is there a common scenario where the CSM Javascript could be getting replaced? 

    I did manage to get a trace. I've posted a few of the lines below (slightly sanitized). Most of them were [error]s about a checked security token issued for page 'http://...', but received from page 'https://...'. I'm assuming this is because we use a load balancer and the SSL encryption is handled at the load balancer. There were also a couple about CreateNewRedirectInfo. But I don't understand the meaning. 

    I'm starting to get a bad feeling that this is going to require more .net development knowledge than I may have to get working. In any event, here's the snippet of logs:

    [0]1724.2276::08/21/2012-22:41:57.467 [Microsoft.EnterpriseManagement.OperationsManager.Apm.Injector] [] [Warning] :CookiesWorkerBase.CreateNewRedirectInfo{cookiesworkerbase_cs140}The processed request's entrypoint guid has not been initialized before creating new Redirect Info. Most common reason - we have not hit any of the ProcessRequest custom actions. Request information: GET http://fqdn/appname/default.aspx. Enable agentDiagnostings feature in csm.action.config to get more trace information about all requests.
    [0]1724.3648::08/21/2012-22:42:12.623 [Microsoft.EnterpriseManagement.OperationsManager.Apm.Injector] [] [Warning] :CookiesWorkerBase.CreateNewRedirectInfo{cookiesworkerbase_cs140}The processed request's entrypoint guid has not been initialized before creating new Redirect Info. Most common reason - we have not hit any of the ProcessRequest custom actions. Request information: POST http://fqdn/appname/app/login.aspx. Enable agentDiagnostings feature in csm.action.config to get more trace information about all requests.
    [0]1724.2660::08/21/2012-22:42:15.044 [Microsoft.EnterpriseManagement.OperationsManager.Apm.Injector] [] [Warning] :CookiesWorkerBase.CreateNewRedirectInfo{cookiesworkerbase_cs140}The processed request's entrypoint guid has not been initialized before creating new Redirect Info. Most common reason - we have not hit any of the ProcessRequest custom actions. Request information: GET http://fqdn/appname/app/management/page1.aspx?FromLogin=1. Enable agentDiagnostings feature in csm.action.config to get more trace information about all requests.
    [1]4888.4392::08/21/2012-22:42:39.233 [Microsoft.EnterpriseManagement.OperationsManager.Apm.Common] [] [Error] :SecurityTokenHelper.CheckToken{securitytokenhelper_cs188}SecurityToken '[stripped]' is issued for page 'http://fqdn/appname/app/management/ExceptionSummary.aspx?UserMode=1', but received from page 'https://fqdn/appname/app/management/ExceptionSummary.aspx?UserMode=1'
    

    gaurhoth

    Wednesday, August 22, 2012 1:32 PM
  • Thanks for the log. I had our developers look at it. They are seeing TWO different issue in the log:

    1. A security token gets generated for the http page, but proxy/load balancer changes protocol to https and invalidates the token. This is something we didn't test with, but we have now filed it in our system as a bug. We don't yet know how long it would take to fix it (i.e. Update Rollup vs. Service Pack) but we are tracking that. So, I suspect nothing you can do for this at this point (other than changing the load balancer configuration) until we fix it.
    2. The second issue is less clear - whenever an .ASPX page is called, IIS routes the request to ASP.NET engine. Whenever we see the "ProcessRequest" method being called (which is typically where the .NEt framework/ASP.NEt starts executing the page's code) - that's where we hook into and inject extra "stuff" in the page's response. For some reason (it might have to do with the way the pages are built - maybe your run of the compatibility checker gives you a clue here) we are not seeing "ProcessRequest" being called, hence we don't inject. I am not sure if it has to do with your page's code, IIS configuration, or our code... here it would be helpful to look at WHICH pages do get the javascript being added and which pages DON'T... a bit of trial and error, for sure, but trying to find some pattern of when they are instrumented and when not...
    Wednesday, August 22, 2012 9:20 PM
  • Thanks for the feedback.

    Question about #2. Would that also cause a problem with APM Server Side Monitoring? From what I can tell, I'm only getting Server Performance Exceptions for pages that I can see javascript injection working on.



    gaurhoth

    Thursday, August 23, 2012 2:57 PM
  • The symptoms would not normally be related... but in this particular case they might be: ProcessRequest is also what we consider "entry point" for .ASPX pages and if we don't "see" it happen from one part of the code, maybe we don't see it anywhere - or maybe for some reason it doesn't happen...

    It would be interesting to investigate the actual pages and what do they do. All those that work - are they all in the same site? Or do you have a distributions of pages that work or don't work even WITHIN the same site? Any difference in configuration or the application pool (classic/integrated pipeline for example) ? Is there any difference in compilation (aspx pages precompiled or not), etc?

    Ultimately, if none of the above tells us any pattern, we should look at the code of the pages that work and don't work and/or set up a local repro and debug it.

    If that's too sensitive info to post on a forum, you might want to reach out our technical support and reference this thread so that they have a head start and can escalate internally (within Microsoft) to us (the product team) if necessary...

    Thursday, August 23, 2012 3:17 PM
  • All of the pages (both working and non-working) are in the same web app.

    As for the compatability checker, it reports 870 instances of the RuleID 6 Warning "Web page has call(s) to HttpWebResponse . Flush()" and 60 instances of ruleID 7 Warning "Web page has calls(s) to HttpWebResponse . End()".

    There does seem to be a correlation between the pages where javascript injection is not-working with the above compatability list. In addition, I am unable to get Server Performance Exceptions from the pages in that list. That indicates that whatever is causing CSM to fail, is also causing Server Side Monitoring to fail (which is disappointing, I could have lived without CSM... but SSM is rather critical to how we wanted to use SCOM).

    As we are still in a POC/Eval period with SCOM, I'm not sure what my options are for support... but I'll try to look into that.

    We've probably taken this thread as far as we can, so one last time, I appreciate the help.


    gaurhoth

    Thursday, August 23, 2012 7:27 PM
  • The warnings about HttpWebResponse.End() and HttpWebResponse.Flush() are there because, if a page's code manipulates the RESPONSE object and closes it before we are able to insert the script at the end, we would not have code posting back to CSMCollector. So in that case you would not have client-side events (but it really depends... at what point those methods are called).

    In this case there is something else at play, I think.

    I/we don't mind helping on the forum - and you helped us find a bug, we appreciate that - but at this point we'd really need to understand something more about the application itself and even debug what's going on there... I am really not sure what is going on in this second case without seeing the app and debug it, I am afraid...

    Friday, August 24, 2012 2:49 AM
  • Hi, As this thread has been quiet for a while, we assume that the issue has been resolved. At this time, we will mark it as "Answered". Either the previous steps should be helpful for many similar scenarios and will be marked as answer, or this post will be marked as answer in order to close the thread. Feel free to re-open the thread if you have additional information about this specific case or to open a new thread for a new case. In addition, we’d love to hear your feedback about the solution. By sharing your experience you can help other community members facing similar problems. Thanks,


    Bob Cornelissen - BICTT (My Blog about SCOM) - MVP 2012 + 2013 and Microsoft Community Contributor 2011 + 2012 Recipient

    Tuesday, March 19, 2013 3:51 PM