CPU spike being caused by splunkd service RRS feed

  • Question

  • Hello,

    I am seeing a CPU spike caused the the splunkd service on Windows 2008 and Windows 7 machines shortly after it is installed.  This problem does not affect 2003 machines.  We roll out two verions of the package, one for 32bit and one for 64 bit, however this issue appears to be OS specific because I have tried installing on a 32-bit Windows7 machine and it had the same CPU spiking issue.

    This problem is happening on both virtual and physical servers.
    This problem does not happen in my test lab, I am running the exact same version of Splunk.

    I suspect that Splunk is conflicting with another application we run in our production environment.  I understand that Splunk is not a Microsoft product, but I am looking for troubleshooting tools that I can use to help determine what splunk is conflicting with.  I've started with a program called dependancy walker which allows me to see what dll's splunkd.exe is using. 

    Can anyone make some suggestions as to how I can further isolate this issue and what tools are available?



    Thursday, July 21, 2011 6:48 PM

All replies

  • You could start by looking at Process Explorer and Process Monitor. In Process Explorer, go to File, and Show Details for All Processes, then go to Options, Configure Symbols and specify a symbol path of srv*c:\symbols* Then view the properties of the problem process and look at the Threads tab to see the functions and modules where it is spending most of its time. This will be limited however because it is a third-party process so the Microsoft symbols server will not have symbols for it. Also, when you have that process selected, if you go to View, Show Lower Pane, you can then toggle between showing the Dlls for that process or the handles it has open.

    Process Monitor will show you file, registry, and basic networking activity. One approach would be to run Process Monitor while that process is spiking the CPU. Go to File, Capture Events to stop monitoring (but retain what has been monitored so far). You can do CTRL+F to search for the process name in the output. When you see it, right-click directly on the process name and select "Include <process.exe>" to filter only the activity of that process. Then on the Tools menu you can select File Summary (and make sure to maximize that File Summary window so you see the Path column showing the specific file names/locations). Click the top of the Path column to sort it, and that will give you a nice overview of all the file reads/writes it is doing. Then you can do the same type of thing with Tools, Registry Summary.

    The goal being to reveal the functions and modules it is calling, as well as the disk and registry I/O it is doing, to understand why it spikes on one OS version but not another.


    Friday, July 22, 2011 5:04 AM