Sign in
 
HomeLibraryLearnDownloadsRepositoryCommunityForumsBlog
Script Center >
> 
> 
Parse a log file

Answered Parse a log file

  • Saturday, July 07, 2012 12:57 PM
     
      Has Code
    Sign In to Vote
    0

    Hello,

    I have a log file with over a 100 non delivery reports which I would like to parse... there is no fixed delimiter between each non delivery report.

    Each non delivery report does have common fields like IntendedRecipient:, FailureReason:, SMTPErrorCode:,... always in the same order but not always on the next line.

    What I would like to do is iterate through the collection and group the non delivery reports in hash tables with all the subjects as key and the description as value as long as (and including) the $blockdelimiter condition is met and then start over again untill the end of the array.

    The log looks like something like this:

    rubish
    rubish
    intendedrecipient: my.email@technet.com
    FailureRaison: blablabla 12:00:00
    space
    SMTP Errorcode: 550 5.7.1
    rubish
    space
    rubish
    space
    intendedrecipient: your.email@technet.com
    space
    FailureRaison: blablabla 12:00:00
    rubish
    SMTP Errorcode: 422
    space
    rubish
    rubish

    What would be the best way to achieve this and keep the script reusable?

    What I've come up with so far is this

    $file = Get-Content C:\test\NDR.txt

$delimiter = ":"
$blockdelimiter = "SMTP Errorcode"

$report = @()
$block = @()

foreach ($line in $file)
    {
        if($line.contains($delimiter) -and $line -ne "")
            {
            $hash = @{ "title" = $line.substring(0,$line.indexof(":"));"Description" = $line.substring($line.indexof(":")+1) }
            $report += $hash
            }
    }

foreach ($line in $report)
    {
        if (-not $line["title"].contains("$blockdelimiter"))
        {
        $block += $line
        }
    }

    This script is all but finished but I'm stuck on thinking which way to go.

    I need a way to iterate through the collection and write an object to another array or something so I have an object per non delivery report. The ultimate goal is a csv report with per rejected emailaddress, the FailureReason etc... in fact all the info in the NDR for that rejected address. 

    Can someone advise on how to proceed...

    Grts.



     

All Replies

  • Saturday, July 07, 2012 2:00 PM
     
     
    Sign In to Vote
    0
    Is there any specific pattern for each log entry. Can you just post a single non-delivery report within the log file?
     
  • Saturday, July 07, 2012 2:09 PM
     
     
    Sign In to Vote
    0

    See: ConvertFrom-StringData

    HELP ConvertFrom-StringData -full

    Tis can create hashes directly from string data.  The left part is th e name and the right is the value.  Use ':' for the delimiter as = is the default.

    ¯\_(ツ)_/¯

     
  • Saturday, July 07, 2012 2:56 PM
     
      Has Code
    Sign In to Vote
    0

    This seems to do the trick with the supplied sample data:

    $log = Get-Content smtp.txt
$key, $value = $null
$results = @()
foreach ($line in $log) {
    $data = @{}
    if ($line -match '^intendedrecipient:(.+)$') { $key = ($matches[1]).trim() }
    elseif ($line -match '^FailureRaison:(.+)$') { $value = ($matches[1]).trim() }
    if ($key -and $value) { $data.Add($key,$value); $results += $data ; $key, $value = $null }
    }
$results

    Grant Ward, a.k.a. Bigteddy

     
  • Saturday, July 07, 2012 3:28 PM
     
      Has Code
    Sign In to Vote
    0

    For future reference.

    Get-Content has an argument that allows it to send n-lines to the pipeline.  This was included for reading in string data like this from a file.  One approach that I have used is to read the n-lines of each logical record and processes them in batches in the second stage of the pipeline.

    ConvertFrom-StringData  can use this grouping directly to create a single record hash which we can use to generate a new PsObject. It is much more PowerShell-like to do it this way as long as the text is full name:data pairs as stated above. ANy other separated pair will also work as long as teh separtaor is consistent.  Spaces and tabs are frequestly used this way.

    This will dynamically convert a log file in the stated format.

    Get-Content -Path smtp.log -ReadCount 10 | 
     %{$_.Replace(':','=')} |
     ConvertFrom-StringData|
     %{New-Object PSObject -Property $_}

    Tis is roughly what Grant is doing in his example.  This is just an approach that feeds the criteria to PowerShell and lets PowerShell decide how to do all of those things.

    Either will work but this technique is very useful to know about in many other similar situations.

    ¯\_(ツ)_/¯


     
  • Saturday, July 07, 2012 5:25 PM
     
      Has Code
    Sign In to Vote
    0

    Hi Bigteddy,

    Thanks for the reply. 

    Wel, I had something similar worked out that does exactly what I want but our senior system engineer wants me to do it in another way because the script was to specific for this log. Unfortunatelly he is very vage in his hints and tips... he told me to parse the lines with the substring method and to work with hashtables.

    In the example log there are only 3 subjects per NDR (intendedrecipient, FailureReason (typo in the log) and SMTP Errorcode)... there are a lot more subjects in the actual log... I would like to include them all.

    I was thinking, the hashtabels key and value can be set by a variable... is there a way to cycle through an array, create the keys and values that come along in another hashtable until the key reaches a certain value and then write the whole new hashtable array to another array... and then continue where it has left of. I would then have an object per NDR... no? I could than reuse the script for every log file that has blocks of information in it... I would only have to specify the linedelimiter, blockbegin and blockend variables.

    Maybe I am thinking too far... our senior system engineer drives me nuts :-)

    PS. This is what I had worked out before

    If "IntendedRecipient" is found in the foreach loop, it creates a new object and writes it to the array $ndrreport in the script scope. The next line where "FailureReason" is found gets written to the value of the reason key in the last object of $ndrreport.

    $file = Get-Content C:\Test\NDR.txt
    $ndrreport = @()

foreach ( $line in $file )
    {
    $recipient = $line | Select-String "IntendedRecipient"
    $reason = $line | Select-String "FailureReason"
    
    if ($recipient) 
            {
            $MyPSObject = New-Object psobject -Property @{
                Recipient = $recipient
                Reason = "" }
            $ndrreport += $MyPSObject
            }
    if ($reason)
            {
            $ndrreport[-1].reason += $reason        
            }
    }

$ndrreport

    Result:

    Recipient                                                                                   Reason                                                                                    
    ---------                                                                                   ------                                                                                    
    intendedrecipient: my.email@technet.com                                                     FailureReason: blablabla 12:00:00                                                         
    intendedrecipient: your.email@technet.com                                                   FailureReason: blablabla 12:00:00                                                         

    I could then extend it to include each subject in the log but, as my senior said, it would be specific for this log file.

    Grts.

     


     
  • Saturday, July 07, 2012 5:50 PM
     
     
    Sign In to Vote
    0

    We still need you to post some examples of your NDR files.  In your poists you have changed the rules or al least stated them differently.  SHow a full copy or two of the NDR reports.

    The code posted an extract all items but the NDR on different systems can be very different.  On Exchange NDR reports are <messagesid>.ndr and are one per NDR sent.  We can aslo pull NDRs out of teh database.

    ¯\_(ツ)_/¯

     
  • Saturday, July 07, 2012 5:58 PM
     
     
    Sign In to Vote
    0

    Hi jrv,

    Nice! Unfortunatelly, the information blocks in the NDR's report do not have a fixed length. Otherwise it would be exactly what I needed.

    I will play around with the ConvertFrom-StringData cmdlet... on monday :-)

    If you could take a look at my reply to Bigteddy... don't know if it is clear what my meaning is.

    Grts.

     
  • Saturday, July 07, 2012 6:56 PM
     
     
    Sign In to Vote
    0

    Hi jrv,

    Nice! Unfortunatelly, the information blocks in the NDR's report do not have a fixed length. Otherwise it would be exactly what I needed.

    I will play around with the ConvertFrom-StringData cmdlet... on monday :-)

    If you could take a look at my reply to Bigteddy... don't know if it is clear what my meaning is.

    Grts.

    Can youy please post a sample of your NDR.  The format is very prefictable and the techniaue I showed you works.  It doe not require fixed length records.

    I have been processing high speed data loads into data warehouses for years.  This method works.  It is a fully generalized method.

    ¯\_(ツ)_/¯

     
  • Saturday, July 07, 2012 7:45 PM
     
     
    Sign In to Vote
    0

    Hi jrv,

    I don't have the report here... will post it on monday.

    Grts.

     
  • Monday, July 09, 2012 9:27 AM
     
     
    Sign In to Vote
    0

    Hi,

    Below 3 Ndr's...

    Received:  from mail-relay2.mydomain.be([0.0.19.5])          by servd100.mydomain.net(Lotus Domino Release 8.5.3)          with ESMTPid 2012070511400350-43341 ;          Thu, 5 Jul2012 11:40:03 +0200
    Received:  from localhost by mail-relay2.mydomain.be; 05 Jul 2012 11:40:00 +0200
    SMTPRcvd:  from unknown (HELO servd007.mydomain.net)([0.0.114.151])  by mail-relay2.mydomain.be with ESMTP; 02 Jul 201210:40:39 +0200
    SMTPRcvd:  from servic008 ([0.0.114.158])         by servd007.mydomain.net (Lotus DominoRelease 6.5.3)          with ESMTP id 2012070210421545-839890;          Mon, 2 Jul 2012 10:42:15 +0200
    MIME_Version:  1.0
    From:  "company Securities"<equity.research@mydomain.be>
    PostedDate:  02/07/2012 10:42:15
    Subject:  Beurs Bij 't Ontbijt02/07/2012
    $MIMETrack:  Itemize by SMTP Serveron LNTopcall/mydomain(Release 6.5.3|September 14, 2004) at 02/07/2012 10:42:19,Serializeby Router on LNTopcall/mydomain(Release 6.5.3|September 14, 2004) at 02/07/201210:42:19,Itemize by SMTP Server on s868301/mydomain(Release 8.5.3|September15, 2011) at 05/07/2012 11:40:03
    $MessageID:  <OF14E3D502.FAC3AEA9-ONC1257A2F.002FD1CF@mydomain.be>
    $NoteHasNativeMIME:  1
    SMTPDSNFrom:  "Mail DeliverySystem" <administrator@mydomain.be>
    SMTPDSNDate:  05/07/2012 11:40:00
    IntendedRecipient:  nuri.iri@companylease.be
    SMTPDSNType:  0
    FailureReason:  5.4.7 - Deliveryexpired (message too old) 'timeout' (delivery attempts: 0)
    SMTPDSNDeliveryReason:  The followingmessage to <nuri.iri@companylease.be> was undeliverable.
    The reason for the problem:
    5.4.7 - Delivery expired (message tooold) 'timeout'

    SMTPDSNDeliveryStatus:  Reporting-MTA:dns; mail-relay2.mydomain.be

    Final-Recipient: rfc822;nuri.iri@companylease.be
    Action: failed
    Status: 5.0.0 (permanent failure)
    Diagnostic-Code: smtp; 5.4.7 - Deliveryexpired (message too old) 'timeout' (delivery attempts: 0)

    SMTPOriginator: 
    $UpdatedBy:  CN=s868301/O=mydomain
    $Orig:  443BD7C85A476F93C1257A3200351B1E
    Categories: 
    $Revisions: 
    RouteServers:  CN=s868301/O=mydomain
    RouteTimes:  05/07/2012 11:40:03-05/07/201211:40:03
    DeliveredDate:  05/07/2012 11:40:03

    Received:  from mail-relay2.mydomain.be([0.0.19.5])          by servd100.mydomain.net(Lotus Domino Release 8.5.3)          with ESMTPid 2012070511400353-43342 ;          Thu, 5 Jul2012 11:40:03 +0200
    Received:  from localhost by mail-relay2.mydomain.be; 05 Jul 2012 11:40:03 +0200
    SMTPRcvd:  from unknown (HELO servd007.mydomain.net)([0.0.114.151])  by mail-relay2.mydomain.be with ESMTP; 02 Jul 201210:46:21 +0200
    SMTPRcvd:  from servic008 ([0.0.114.158])         by servd007.mydomain.net (Lotus DominoRelease 6.5.3)          with ESMTP id 2012070210475659-840008;          Mon, 2 Jul 2012 10:47:56 +0200
    MIME_Version:  1.0
    From:  "company Securities"<equity.research@mydomain.be>
    PostedDate:  02/07/2012 10:47:56
    Subject:  Beurs Bij 't Ontbijt02/07/2012
    $MIMETrack:  Itemize by SMTP Serveron LNTopcall/mydomain(Release 6.5.3|September 14, 2004) at 02/07/2012 10:48:00,Serializeby Router on LNTopcall/mydomain(Release 6.5.3|September 14, 2004) at 02/07/201210:48:01,Itemize by SMTP Server on s868301/mydomain(Release 8.5.3|September15, 2011) at 05/07/2012 11:40:03
    $MessageID:  <OF16C8A188.315ED0CB-ONC1257A2F.0030570B@mydomain.be>
    $NoteHasNativeMIME:  1
    SMTPDSNFrom:  "Mail DeliverySystem" <administrator@mydomain.be>
    SMTPDSNDate:  05/07/2012 11:40:03
    IntendedRecipient:  cedric.soons@companyfm.com
    SMTPDSNType:  0
    FailureReason:  5.4.7 - Deliveryexpired (message too old) 'timeout' (delivery attempts: 0)
    SMTPDSNDeliveryReason:  The followingmessage to <cedric.soons@companyfm.com> was undeliverable.
    The reason for the problem:
    5.4.7 - Delivery expired (message tooold) 'timeout'

    SMTPDSNDeliveryStatus:  Reporting-MTA:dns; mail-relay2.mydomain.be

    Final-Recipient: rfc822;cedric.soons@companyfm.com
    Action: failed
    Status: 5.0.0 (permanent failure)
    Diagnostic-Code: smtp; 5.4.7 - Deliveryexpired (message too old) 'timeout' (delivery attempts: 0)

    SMTPOriginator: 
    $UpdatedBy:  CN=s868301/O=mydomain
    $Orig:  0AEDFD63DDCFD7EFC1257A3200351B21
    Categories: 
    $Revisions: 
    RouteServers:  CN=s868301/O=mydomain
    RouteTimes:  05/07/2012 11:40:03-05/07/201211:40:03
    DeliveredDate:  05/07/2012 11:40:03

    Received:  from mail-relay2.mydomain.be([0.0.19.5])          by servd100.mydomain.net(Lotus Domino Release 8.5.3)          with ESMTPid 2012070511084425-42494 ;          Thu, 5 Jul2012 11:08:44 +0200
    Received:  from localhost by mail-relay2.mydomain.be; 05 Jul 2012 11:08:40 +0200
    SMTPRcvd:  from unknown (HELO servd007.mydomain.net)([0.0.114.151])  by mail-relay2.mydomain.be with ESMTP; 02 Jul 201210:43:15 +0200
    SMTPRcvd:  from servic008 ([0.0.114.158])         by servd007.mydomain.net (Lotus DominoRelease 6.5.3)          with ESMTP id 2012070210445133-839941;          Mon, 2 Jul 2012 10:44:51 +0200
    MIME_Version:  1.0
    From:  "company Securities"<equity.research@mydomain.be>
    PostedDate:  02/07/2012 10:44:51
    Subject:  Beurs Bij 't Ontbijt02/07/2012
    $MIMETrack:  Itemize by SMTP Serveron LNTopcall/mydomain(Release 6.5.3|September 14, 2004) at 02/07/2012 10:44:54,Serializeby Router on LNTopcall/mydomain(Release 6.5.3|September 14, 2004) at 02/07/201210:44:55,Itemize by SMTP Server on s868301/mydomain(Release 8.5.3|September15, 2011) at 05/07/2012 11:08:44
    $MessageID:  <OF864E0AB4.89D9E042-ONC1257A2F.00300EA9@mydomain.be>
    $NoteHasNativeMIME:  1
    SMTPDSNFrom:  "Mail DeliverySystem" <administrator@mydomain.be>
    SMTPDSNDate:  05/07/2012 11:08:40
    IntendedRecipient:  avbrun@tele2allin.be
    SMTPDSNType:  0
    FailureReason:  5.4.7 - Deliveryexpired (message too old) 'DNS Soft Error looking up tele2allin.be (MX)while asking recursive_nameserver1.parent. Error was: unable to reach nameserveron any valid IP' (delivery attempts: 0)
    SMTPDSNDeliveryReason:  The followingmessage to <avbrun@tele2allin.be> was undeliverable.
    The reason for the problem:
    5.4.7 - Delivery expired (message tooold) 'DNS Soft Error looking up tele2allin.be (MX) while asking recursive_nameserver1.parent.Error was: unable to reach nameserver on any valid IP'

    SMTPDSNDeliveryStatus:  Reporting-MTA:dns; mail-relay2.mydomain.be

    Final-Recipient: rfc822;avbrun@tele2allin.be
    Action: failed
    Status: 5.0.0 (permanent failure)
    Diagnostic-Code: smtp; 5.4.7 - Deliveryexpired (message too old) 'DNS Soft Error looking up tele2allin.be (MX)while asking recursive_nameserver1.parent. Error was: unable to reach nameserveron any valid IP' (delivery attempts: 0)

    SMTPOriginator: 
    $UpdatedBy:  CN=s868301/O=mydomain
    $Orig:  C8AC9DFFDEA25ADFC1257A3200323D0A
    Categories: 
    $Revisions: 
    RouteServers:  CN=s868301/O=mydomain
    RouteTimes:  05/07/2012 11:08:44-05/07/201211:08:44
    DeliveredDate:  05/07/2012 11:08:44

    Thanks for taking a look at it.

    Grts.

     
  • Monday, July 09, 2012 10:56 AM
     
      Has Code
    Sign In to Vote
    0

    Meanwhile I have worked this out which gives all subjects and description per block... still curious on how to do it in other ways though!

    $loglocation = "C:\test\NDR.txt"
$reportlocation = "C:\test\NDR.csv"
$linedelimiter = ":"
$blockdelimiter = "DeliveredDate"

    $file = Get-Content $loglocation
    
$report = @()
$block = @{}

    foreach ($line in $file)
    { 
        if($line.contains($linedelimiter))
            {
            $key = $line.substring(0,$line.indexof($linedelimiter))
            $value = $line.substring($line.indexof($linedelimiter)+1)
            
            if ($block.keys -contains $key)
            {
            $key = $key + "1"
            }
            else
            {
            $block.Add($key,$value)
            }
                        
            if ($block.keys -contains "$blockdelimiter")
                {
                $obj = new-object psobject -property $block
                $report += $obj
                $block = @{}
                }
            }
    }
$report | Select intendedrecipient,failurereason,SMTPDSNDeliveryReason,Subject
$report | Export-Csv $reportlocation -NoTypeInformation
    Grts.

     
  • Monday, July 09, 2012 1:21 PM
     
     
    Sign In to Vote
    0

    You did a bad job of pasting.  Are these all in one  file or in separate files like with Exchange?  You seen to have lost all of the line breaks too.

    As I suspected these are fixed access reports.  Try the followingg on your file:

    09:13 PS>get-content e:\test2\test.ndr -ReadCount 36|%{$_[1]}
    Received:  from localhost by mail-relay2.mydomain.be; 05 Jul 2012 11:40:00 +0200
    Received:  from localhost by mail-relay2.mydomain.be; 05 Jul 2012 11:40:03 +0200
    Received:  from localhost by mail-relay2.mydomain.be; 05 Jul 2012 11:08:40 +0200

    Note that we return th efirst line of each 36 line set and it is always the 'Reveived' line so now we can just ump each vertical record as an object.  It appears sthat somethng has edited this file forem the original.  I suggest going back and getting an unmodified file.  It contains numerous extrea line breaask.  If this is, in fact, part of the raw file then we just need to add a step to remove the line breaks.

    The file as it stands may not be parsable except with some very tricky RegEx.  This is why I believe it has been altered from its original form.  It is not generated by a program in this form or their is a problem with teh system or disk.  I may also be that you have somehow gotten junk incorporated into what you have pasted.

    ¯\_(ツ)_/¯


     
  • Monday, July 09, 2012 2:33 PM
     
      Has Code
    Sign In to Vote
    0

    I see the issue .  The file has multiline lements with etrh txt decode of teh message code.

    This is a starter.  It will get you 99% of what you are looking for.  Y can alos increse teh selectivit of the capture and alter this to use a multiline regex which can then grab all of the lines in a message. 

    get-content e:\test2\test.ndr -ReadCount 36 |
    %{
    $props=@{}
    $_ | %{ 
           if($_ -match 'IntendedRecipient:(?<recip>.*$)'){
              $props.intendedrecipient=$matches.recip.Trim()
           }elseif($_ -match 'FailureReason:(?<failure>.*$)'){
              $props.failureReason=$matches.failure.Trim()
           }elseif($_ -match 'Diagnostic-Code:(?<smtp>.*$)'){
              $props.SMTPErrorCode=$matches.smtp.Trim()
          }
         }
     New-Object PsObject -Property $props
     }

    Try this.  There is no SMTPErrorCode in the file.  I used Diagnostic-COde.

    ¯\_(ツ)_/¯

     
  • Monday, July 09, 2012 2:41 PM
     
     
    Sign In to Vote
    0

    The log file is an export of mails from within Lotus Notes. So the user searched for NDR's in his mailbox, selected them all and exported them to one .txt file...

    If I run the command on the complete log (total of 145 ndr's) it goes well for the first 18, then it goes wrong... probably extra line breaks or something.

    Thanks to the support here I have what I need for this log with the script in my previous post.

    Will definitely try your method with Get-Content -Readcount and ConvertFrom-StringData on more consistent server log's.

    Thanks for the support!

    Grts.

     
  • Monday, July 09, 2012 2:51 PM
    Moderator
     
     
    Sign In to Vote
    0

    Hi,

    PowerShell is good, but there is also the free Log Parser tool which is designed to (you guessed it) parse logs. It provides a SQL-like syntax for parsing log files and may be suitable for your scenario. The best place to ask questions about it is in the Log Parser forums.

    Bill

     
  • Monday, July 09, 2012 3:22 PM
     
     Answered Has Code
    Sign In to Vote
    0

    Here is a cleaned up version - it is easier to understand I think.

    Function Convert-NDRLogFile{
    [CmdLetBinding()]
    Param(
        $logfile='e:\test2\test.ndr'
    )
    Get-Content $logfile -ReadCount 36 |
        ForEach-Object{
            $props=@{
               IntendedRecipient=''
               FailureReason=''
               SMTPErrorCode=''
            }
            $_|
                ForEach-Object{ 
                    if($_ -match 'IntendedRecipient:\s+(?<recip>.*$)'){
                      $props.IntendedRecipient=$matches.recip
                      Write-Verbose $props.IntendedRecipient
                    }elseif($_ -match 'FailureReason:\s+(?<failure>.*$)'){
                      $props.failureReason=$matches.failure
                      Write-Verbose $props.failureReason
                    }elseif($_ -match 'Diagnostic-Code:\s+(?<smtp>.*$)'){
                      $props.SMTPErrorCode=$matches.smtp
                      Write-Verbose $props.SMTPErrorCode
                    }else{
                        Write-Verbose "SKIPPING:" 
                    }
                }
             New-Object PsObject -Property $props
        }
 }
 
 Convert-NDRLogFile -logfile e:\test2\test.ndr -Verbose

    ¯\_(ツ)_/¯

     
  • Monday, July 09, 2012 3:29 PM
     
     
    Sign In to Vote
    0

    Yep, that returns all 145 ndr's.

    Regex is still some unexplored territory... still so much to learn :-).

    Thanks.

     
  • Monday, July 09, 2012 7:46 PM
     
     Answered Has Code
    Sign In to Vote
    0

    Yep, that returns all 145 ndr's.

    Regex is still some unexplored territory... still so much to learn :-).

    Thanks.

    That should get all of them assuming they actually all maintain 36 line.  If not we have to use Regex to separte the records by doing a multiline capture.

    Items within a record have multiple lines.  These multiline sets are terminated by a double newline pair.  It is possible for  multiline set to have a variable number of lines so the 36 lines per record would fail.  The best way to test this is to read a number of ogs and print oky the first line.  These will line up nicely so you can see if this breaks down.

    To add extra items to teh outpu just copy the pattern in the order the items appear in the file.  The items only grab the first line of the field.

    To add subject we would just do this:

    #requires -version 2.0
Function Convert-NDRLogFile{
<#
     .SYNOPSIS
     Extract fields from NDR report
#>
    [CmdLetBinding()]
    Param(
        $logfile='e:\test2\test.ndr'
    )
    Get-Content $logfile -ReadCount 36 |
        ForEach-Object{
            $props=@{
               Subject=''
               IntendedRecipient=''
               FailureReason=''
               SMTPErrorCode=''
            }
            $_|
                ForEach-Object{ 
                    if($_ -match 'Subject:\s+(?<subject>.*$)'){
                      $props.Subject=$matches.subject
                      Write-Verbose $props.IntendedRecipient
                    }elseif($_ -match 'IntendedRecipient:\s+(?<recip>.*$)'){
                      $props.IntendedRecipient=$matches.recip
                      Write-Verbose $props.IntendedRecipient
                    }elseif($_ -match 'FailureReason:\s+(?<failure>.*$)'){
                      $props.failureReason=$matches.failure
                      Write-Verbose $props.failureReason
                    }elseif($_ -match 'Diagnostic-Code:\s+(?<smtp>.*$)'){
                      $props.SMTPErrorCode=$matches.smtp
                      Write-Verbose $props.SMTPErrorCode
                    }else{
                        Write-Verbose "SKIPPING:" 
                    }
                }
             New-Object PsObject -Property $props
        }
 }
 
 Convert-NDRLogFile -logfile e:\test2\test.ndr |fl

    Since 'Subject:' comes first I just inserted that at the top of the filter (IF/ELSEIF).  I add sunject to the hash in the same order (this is protection against missing fields) and gave the capture group a different name for convenience.

    If is is necesssary to yank the record as text and do a multiline capture that this filter can be converted easily into a pure Regex multi group capture statement.

    The 'plus' with starting this way is that we can taylor each capture to 'consume' as much to the text as we need and it has no dependencies on the exisence of earlier elements.  This is nearly impossible to do with a ragged record and purely linear techniques unless the rules are absolutely known.

    ¯\_(ツ)_/¯

     
  • Monday, July 09, 2012 7:49 PM
     
     
    Sign In to Vote
    0

    Hi,

    PowerShell is good, but there is also the free Log Parser tool which is designed to (you guessed it) parse logs. It provides a SQL-like syntax for parsing log files and may be suitable for your scenario. The best place to ask questions about it is in the Log Parser forums.

    Bill

    Bill - I though of that earlier but LogParser is not really good at doing ragged records wih newline field separators.  Tere are a couple of great tools for this that can turn almost any kind of text file into data.  LogParser comes close for most known logfile formats.  I don't thing this one will work however it may be worth a try.

    ¯\_(ツ)_/¯

     
  • Monday, July 09, 2012 8:08 PM
     
     
    Sign In to Vote
    0

    In exchange we can grab NDRs from teh tracking log.  It provides the information in a tab delimited form loadable as data.

    Example:
    2012-7-7 18:10:58 GMT - - - ALPHA - someuser@somewhere.com.com 1033 Za8UXPFs100000006@simplecompany.com 0 0 5460 1 - 0 Version: 6.0.3790.4675 - Delivery Status Notification (Failure) postmaster@simplecompanycorp.com -

    These fields are all tbe delimited.  The message ID is in teh field and it is claerly makred as a delinervy failure.  We can get the rest from the message in the database use the message id.  This is how the Message Tracking tool does this.  All NDR fiedls are avainalbe as dta this way along with other items.  The NDR reports are just codes expanded into text for diagnostic purposes.

    LogPaser can easily read this log.

    ¯\_(ツ)_/¯

     
  • Friday, July 20, 2012 3:02 AM
    Moderator
     
     
    Sign In to Vote
    0

    As there has been no activity in this thread for a few days, we assume the issue is resolved. We will mark it as "answered" to assist others in similar situations. If you disagree, please reply with further information. You can unmark the answer if you wish. If a reply helped answer your question, please mark it as the answer.

    Richard Mueller - MVP Directory Services