locked
Cleaning up Powershell Script - IIS Log Parsing & Reporting RRS feed

Answers

  • well, since I guess im just flat out not going to get powershell assistance in the official microsoft powershell forum, Ill just have to try other forums to get assistance.

    Thanks for answering all my powershell specific questions.

    Monday, April 2, 2018 10:17 PM

All replies

  • Sorry but this forum is not for re-writing scripts you have found on the Internet.

    Please carefully review the following links to set your expectation for posting in  technical forums.

    This Forum is for Scripting Question Rather than script requests

    Script Gallery.

    Script Center

    Learn PowerShell  

    Script requests

    From a Bill Stewart summary of useful forum links:


    \_(ツ)_/

    Monday, April 2, 2018 5:12 PM
  • Thanks JRV - you have helped me in the past - 

    I am not asking to rewrite scripts I found on the internet - I have written 95% of this myself, I am asking for guidance on how to perform the tasks I am doing better.  When I attempt to add {$_ -replace '"', ""}  piped into any other commands, my output gets messed up.

    Should I have phrased it differently, such as "how do I combine these 3 lines into 1" ?

    I am simply attempting to performing a number of tasks on import, and I know there has to be a better way to import > perform actions > export, rather than import > action > export > re-import > action > re-export.

    Monday, April 2, 2018 5:25 PM
  • To parse IIS logs correctly use LogParser2.  It can extract all data into a CSV or a database in one line of code.

    Your question about "{$_ -replace '"', ""} " does not make much sense.  Why are you trying to remove quotes from a CSV file?   This will destroy the CSV file. Quotes are required in a CSV file when commas are present in the fields.

    What is "Parse-IISLOG"?  It is not a PowerShell CmdLet.


    \_(ツ)_/

    Monday, April 2, 2018 5:31 PM
  • Parse IIS-Log is a powershell script that I referenced from an online resource.  After researching the best way to do this, I wrote a similar version.  When I came upon that URL with the following script, I realized that this version of the script worked better with my workflow and handled the header values a little better than the one I wrote (trying to learn and become more efficient)

    The script is as follows:

    param (
    	[Parameter(ValueFromPipeline=$true)]
    	[IO.FileInfo]$logfile
    	
    )
    
    if (!(Test-Path $logfile.fullname)) { 
    	Write-Warning "$($logfile.fullname) does not exist" 
    	Exit
    }
    
    #$DebugPreference = "Continue"
    $header = ""
    $servername = ""
    
    get-content	$logfile | foreach { 
    	Write-Debug "parsing $_"
    	switch -wildcard ($_) {
    		"#Fields:*" {
    			Write-Debug "found $header"
    			$header = $_.substring($_.indexof(" ")+1).trim().split(" ")
    		}
    		"#Server Name:*" {
    			$servername = $_.split(":")[1].trim()
    		}
    		default { 
    			if (!($_.startswith("#")) -and $_ -ne "") {
    				Write-Debug "converting string"
    				Write-Debug "$([string]::Join(" ",$header))"
    				$_ | ConvertFrom-Csv -Delimiter " " -Header $header | Add-Member -MemberType NoteProperty -Name ServerName -Value $servername -PassThru
    			}
    		}
    	}
    }

    The reason I went away from using LogParser2 is because I am trying to use native powershell functionality to reduce any dependencies.  Plus, I am always trying to learn how to do things myself with powershell without having to use pre-written software.

    Regarding removing the quotation marks, it doesn't destroy the CSV, as CSV is just that, COMMA SEPARATED VALUES.  When we import the version of the output file without removing the quotation marks, our reporting tool has some difficulties and makes the reporting less easy to read.  Once we remove the quotation marks, we are left with a clean CSV file that imports into our reporting tool easily.

    My final output looks like this without the quotes:

    Date,Time,ServerIP,sslProtocol,sslCipher,sourceIP
    2018-04-01,00:41:00,10.13.8.125,TLSv1.2,DHE-RSA-AES256-GCM-SHA384,5.189.169.91
    2018-04-01,23:05:52,10.13.8.125,TLSv1.2,AES256-SHA,71.6.202.204
    2018-04-01,18:23:45,10.13.8.126,TLSv1.2,DHE-RSA-AES256-GCM-SHA384,46.161.55.108
    2018-04-01,12:46:34,10.13.8.127,TLSv1.2,AES256-SHA,74.82.47.2
    2018-04-01,06:45:45,10.13.8.134,TLSv1.2,DHE-RSA-AES256-GCM-SHA384,46.161.55.106
    2018-04-01,15:31:01,10.13.8.135,TLSv1.2,AES256-GCM-SHA384,139.162.78.135

    Monday, April 2, 2018 6:02 PM
  • Any one field with a comma or single quote will break the CSV from that point on.

    I cannot strongly enough recommend suing LogParser as it can change the headers and produce a consistent CSV file.  After you have the CSV you can remove the quotes but it may never again load as a CSV.


    \_(ツ)_/

    Monday, April 2, 2018 6:06 PM
  • Thats completely understood - the data (IIS Logs) dont contain quotes (single or double) - and I am 100% certain that goes for the columns I am importing.

    With that being said, the data being what it is, am I stuck opening up the CSV file a 2nd time just to remove the quotation marks?

    Monday, April 2, 2018 9:35 PM
  • With LogParser you can choose a delimited format that does not include quotes.

    You can also just strip out all of the quotes but the file can break if it is not correctly consistent.


    \_(ツ)_/

    Monday, April 2, 2018 9:37 PM
  • well, since I guess im just flat out not going to get powershell assistance in the official microsoft powershell forum, Ill just have to try other forums to get assistance.

    Thanks for answering all my powershell specific questions.

    Monday, April 2, 2018 10:17 PM
  • What is it that we can do?  If the file is breaking then something is amiss with your CSV file.  You need to debug the issue.  Without the file there is really nothing we can do to help you.  I see no way to guess at the cause other than the file has inconsistent contents.

    Believe me.  I have been building an manipulation CSV files for more than 30 years.  If the file is good and you strip the quotes and it is bad there is something wrong with the file.


    \_(ツ)_/

    Monday, April 2, 2018 10:26 PM
  • We cannot delete the thread and your deleting pieces is not helpful to others who may find this useful.


    \_(ツ)_/

    Monday, April 2, 2018 10:27 PM