locked
Parsing IIS Logs for average Time-Taken RRS feed

  • Question

  • Afternoon,

    I am trying to parse IIS logs on individual servers and display the numerical output of 'time-taken' in green if less than 1 second or red if above 1 second. I have managed to get the Time-Taken from an IIS log file on my local machine using the info covered here but this is just for an individual server.

    So... any help with a PS Script that will get the location of the most current IIS log for a list of given servers, parse the file to find the average time-taken and display an output in green (<1 sec) or red (>1 sec)

    I can imagine it will be quite a few lines but any help is much appreciated....


    • Edited by JimAll Thursday, November 6, 2014 3:41 PM error
    • Changed type JimAll Tuesday, November 11, 2014 2:43 PM Its a question
    Thursday, November 6, 2014 3:41 PM

Answers

  • You logs are not in correct format.  Is the IIS 5/6?

    You should set the logs for correct output format and columns.

    You can clean the header.

    $csv=import-csv <file> -delim ' ' -header date,time,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,cs-username,c-ip,cs(User-Agent),sc-status,sc-substatus,sc-win32-status
    $csv[3..$csv.Count]| ....
    Of course you will have to get all of the headers from the file as the line was truncated. Fro the future set the log to be data compatible and in W3C format.


    ¯\_(ツ)_/¯


    • Edited by jrv Tuesday, November 11, 2014 12:16 PM
    • Marked as answer by JimAll Tuesday, November 11, 2014 2:43 PM
    Tuesday, November 11, 2014 12:15 PM

All replies

  • You're free to search the script repository to see if anyone has written a script that does something like what you want. You can also request a script at the request page. However, this forum is really for scripting questions, not script requests.


    -- Bill Stewart [Bill_Stewart]

    Thursday, November 6, 2014 3:47 PM
  • Use baretail.  It will colorize your logs as needed:

    https://www.baremetalsoft.com/baretail/


    ¯\_(ツ)_/¯

    Thursday, November 6, 2014 4:06 PM
  • Keep in mind that those requests are viewed by volunteers and as such you should not have unrealistic expectations that someone is going to take the time to do this for you.

    -- Bill Stewart [Bill_Stewart]

    Friday, November 7, 2014 3:34 PM
  • Getting there, only thing I cant do at the moment is get a average. I have tried Measure-Object -Average but it returns an error. Any ideas?

    $Servers="TCPP2B1W01","TCPP2B1W02"
    Foreach ($Server in $Servers){
    $Logs=(Get-ChildItem "\\$server\L$\Logs\IIS\") | Where {$_.LastWriteTime.Date -eq (Get-Date).Date}
    foreach ($Log in $logs){
    $LogFiles=(Get-ChildItem $log.FullName | Where {$_.LastWriteTime.Date -eq (Get-Date).Date})
    foreach ($LogFile in $LogFiles){
    $Header=((Select-String -Path $LogFile.FullName -Pattern "#fields: " |  Select-Object -First 1).line.Substring("#Fields: ".Length) -split ' ')
    $Csv=Import-Csv $LogFile.FullName -Header $Header -Delimiter ' ' | Where-Object { -not($_.Date.StartsWith('#'))}
    $Csv | Select-Object @{Label="time-taken";Expression={$_.'time-taken'}} -Unique
    }
    }
    }

    Error I recieve:

    Measure-Object : Input object "" is not numeric.
    At D:\Powershell\Dev\IISTimeTaken.ps1:22 char:8
    + $CSV | Measure-Object -Average
    +        ~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidType: (:PSCustomObject) [Measure-Object], PSInvalidOperationException
        + FullyQualifiedErrorId : NonNumericInputObject,Microsoft.PowerShell.Commands.MeasureObjectCommand
     
    Count    : 1
    Average  : 
    Sum      : 
    Maximum  : 
    Minimum  : 
    Property : 

    Friday, November 7, 2014 3:49 PM
  • IIS log files are CSV and can be loaded with Import-csv.  Measure-Object can average or calculate and field:

    $csv=Import-Csv <logfile>
    $csv|Measure-Object time-taken

    You can also use LogParser 2.0 which can easily do complex SQL queries of IIS logs and has examples on how to do what you are trying to do.


    ¯\_(ツ)_/¯

    Friday, November 7, 2014 7:13 PM
  • Thanks jrv, that worked momentarily. I have amended my script...

    $Servers="TCPP2B1W01","TCPP2B1W02"
    Foreach ($Server in $Servers){
    $Logs=(Get-ChildItem "\\$server\L$\Logs\IIS\") | Where {$_.LastWriteTime.Date -eq (Get-Date).Date}
    foreach ($Log in $logs){
    $LogFiles=(Get-ChildItem $log.FullName | Where {$_.LastWriteTime.Date -eq (Get-Date).Date})
    foreach ($LogFile in $LogFiles){
    $csv=import-csv $LogFile.FullName
    $CSV | Measure-Object time-taken -Average
    }
    }
    }

    But I get the following error...

    Measure-Object : Property "time-taken" cannot be found in any object(s) input.


    At D:\Powershell\Dev\IISTimeTaken.ps1:20 char:8

    + $CSV | Measure-Object time-taken -Average

    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo : InvalidArgument: (:) [Measure-Object], PSArgumentException

    + FullyQualifiedErrorId : GenericMeasurePropertyNotFound,Microsoft.PowerShell.Commands.MeasureObjectCommand

    • Edited by JimAll Tuesday, November 11, 2014 9:24 AM
    Tuesday, November 11, 2014 9:14 AM
  • Test with one csv first and resolve issues.

    $csv=import-csv $LogFile.FullName
    $csv | select time-taken


    ¯\_(ツ)_/¯

    Tuesday, November 11, 2014 9:59 AM
  • I dont think it is importing the file correctly.

    $csv=import-csv L:\Logs\IIS\W3SVC1\u_ex141108.log
    $CSV | Select time-taken

    This returns nothing, even though the IIS log has an entry....

    $csv=import-csv L:\Logs\IIS\W3SVC1\u_ex141108.log
    $CSV

    Imports the following...

    #Version: 1.0                                                                                                                                  
    -------------                                                                                                                                  
    #Date: 2014-11-08 02:40:28                                                                                                                     
    #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status tim...
    2014-11-08 02:40:28 172.30.196.93 OPTIONS * - 80 - 10.226.132.92 - 200 0 0 1279                                                                

    1279 being the time-taken. It does display it across single lines...

    Even $csv | Select Date doesn't return anything...

    Thanks for your help.

    Tuesday, November 11, 2014 10:43 AM
  • You logs are not in correct format.  Is the IIS 5/6?

    You should set the logs for correct output format and columns.

    You can clean the header.

    $csv=import-csv <file> -delim ' ' -header date,time,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,cs-username,c-ip,cs(User-Agent),sc-status,sc-substatus,sc-win32-status
    $csv[3..$csv.Count]| ....
    Of course you will have to get all of the headers from the file as the line was truncated. Fro the future set the log to be data compatible and in W3C format.


    ¯\_(ツ)_/¯


    • Edited by jrv Tuesday, November 11, 2014 12:16 PM
    • Marked as answer by JimAll Tuesday, November 11, 2014 2:43 PM
    Tuesday, November 11, 2014 12:15 PM
  • I am running IIS 7 and the logs are in W3C format with the default UTF-8 encoding. Time-Taken is also a selected field so it 'should' work.

    Tuesday, November 11, 2014 12:37 PM
  • They are not in data format.  They cannot be loaded as data without cleaning them. There is a setting that set the format.


    ¯\_(ツ)_/¯

    Tuesday, November 11, 2014 12:58 PM
  • I have run my suggestion and it converts well:

    PS C:\scripts> (Import-Csv iistemp.csv|measure time-taken -Average).Average
    87400.1070707071
    PS C:\scripts>
    You have to replace all of the spaces with commas and remove the first thre lines.  I also replaces the "#FIELDS: " with nothing.  After that Import works


    ¯\_(ツ)_/¯

    Tuesday, November 11, 2014 1:20 PM
  • Got it..

    $csv = import-csv  L:\Logs\IIS\W3SVC1\u_ex141029.log -delimiter ' ' -header date,time,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,cs-username,c-ip,'cs(user-Agent)',sc-satus,sc-substatus,sc-win32-status,time-taken
    $csv=$csv[3..$csv.Count] | select time-taken | where{$_.'time-taken' -ne $Null -and $_.'time-taken' -notlike "s*"} | measure time-taken -Average
    $csv.Average

    The above will display the average time-taken. I had to add the 'where($_.'time-taken'....... as if an IISReset is carried out it will created a new line of headers in the log file which are picked up read as time-taken values, which returns text in the time-taken column, which messes with the .Average

    Anyway, this is working, not quite as neat as yours but thanks for the help...

    Tuesday, November 11, 2014 2:17 PM