locked
Exchange message tracking log archiving RRS feed

  • Question

  • As we are into bancking business history Exchange message tracking logs are very important for auditing.
    But due to space constraint we cant put unlimited retention on message tracking logs.

    Is it possible to archive message tracking logs(after 3 months) at external storage & use for auditing. Any built in tool?
    Pls also advice any third party utility/software (cheap solution) available to achieve it.

    We are Exchange 2010 Sp2 Ru4.

    Friday, November 2, 2012 4:34 AM

Answers

  • I'd think it would work as long as they are put back on the same server they came from.  I can't say whether moving them to another server in a test environment will work or not, but I suspect you'd have better luck if you stop the transport service on that server, and delete all the existing logs and indexes before you move the archived logs and indexes to it for searching. 


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    • Proposed as answer by wendy_liu Tuesday, November 13, 2012 1:19 AM
    • Marked as answer by wendy_liu Tuesday, November 13, 2012 1:20 AM
    Monday, November 5, 2012 2:15 PM
  • I was mucking around with this earlier, and finally got back to getting the bug cornered and stomped on.  It uses the DotNetZip library from CodePlex. 

    http://dotnetzip.codeplex.com/

    $RetentionDays = 55
    $ArchiveDir = <archive dir>
    Add-Type -Path 'C:\Program Files (x86)\Dino Chiesa\DotNetZip Runtime 1.9\Ionic.Zip.dll'
    $TargetDate = ((get-date).AddDays(-$RetentionDays)).ToString('yyyyMMdd')
    $TargetLog = 'MSGTRK{0}*' -f $TargetDate
    $hts = get-exchangeserver |
      where {$_.serverrole -match “hubtransport”} 
      
    foreach ($ht_server in $hts){
       $ht = get-transportserver $ht_server.name 
       $log_unc = “\\$($ht.name)\$($ht.messagetrackinglogpath -replace “:”,”$”)”
       $log_ht = @{}
       $logs = gci $log_unc\*.log |
         where {$_.name -lt $TargetLog} |
          foreach {
                    $log_ht[$_.name.Substring(0,14)] += @($_.name)
                  }
            
       foreach ($day in $log_ht.keys){
        $TargetZip = "$ArchiveDir\$($ht.name)\$day.zip"
        $zipfile =  new-object Ionic.Zip.ZipFile($TargetZip)
          $zipfile.addentry("_Zipfile_Readme.txt","Zip of logs from $log_unc for $($day.substring(6,8))") 
          foreach ($Logfile in $log_ht[$day]) {
            $zipfile.AddFile("$log_unc\$Logfile","\")
            $zipfile.AddFile("$log_unc\Index\$Logfile.message-id.idx","\Index")
            $zipfile.AddFile("$log_unc\Index\$Logfile.sender-address.idx","\Index")
          }
        $zipfile.Save()
       }
    }       

    It will find all the log and index files older than the specificed retention days, and package up each day into a separate zip file, with all the logs for that day in the root of the zip, and all the index files for that day in an /Index subdirectory of the zip. 

    It stores them in the $ArchiveDir path, in a subdirectory per hub server (directory name is the server name).  There's no error checking or automatic directory creation, so you'd need to add that. Limited testing shows the index files compress at about 35%, and the log files compress at about 85%.

    I chose the DotNetZip library because the methods seem to closely match the zip file support that's being included in dotnet 4.5, so conversion to native dotnet methods later should be fairly trivial if you want to eliminate the dependency on the third-party dll.


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    • Proposed as answer by wendy_liu Monday, November 12, 2012 2:26 AM
    • Marked as answer by wendy_liu Tuesday, November 13, 2012 1:20 AM
    Saturday, November 10, 2012 12:27 AM

All replies

  • Friday, November 2, 2012 5:42 AM
  • Hi DareDevil57,

    Yeah i am aware about mailbox audit logging, journalling. Currently we do not have enough budget for storage required for journaling.
    What we are looking for is solution to move/transfer/archive Exchange message tracking logs to alternate storage so that we can implement 3 months retention of logs on Exchange server & older logs will retention at External storage.

    Also we can offload message tracking processing from exchange server to external.

    reply for any doubt. 

    Friday, November 2, 2012 5:55 AM
  • Friday, November 2, 2012 5:58 AM
  • DareDevil57,

    Yes its very useful to export logs to Excel & do message tracking. My question is how we gonna retain the logs.I mean here i want to have policy wherein Message tracking logs from Exchange server directory will get automatically moved to different drive location(file server) & delete from exchange server to avoid drive space crunch on server.

    Friday, November 2, 2012 6:09 AM
  • e.g. there are products like syslog, kiwilog which will move Windows system event logs system,application to Syslog server & reeposite there.

    Similarly i am looking for option to move message tracking log files to remote localtion.
    Friday, November 2, 2012 6:16 AM
  • Hi

    You can write a powershell script to move logs older than 60 days to different store. 

    We are using such script to keep free space on ours Exchange Servers, and to save all logs for future auditing. 


    Remigiusz
    ExchangeBlog

    Friday, November 2, 2012 8:57 AM
  • yes exactly.
    can you pls share the same or give idea.
    Saturday, November 3, 2012 1:47 AM
  • Some idea:

    $RetentionDays = 55
    $TargetDate = ((get-date).AddDays(-$RetentionDays)).ToString('yyyyMMdd')
    $TargetLog = 'MSGTRK{0}-1.LOG' -f $TargetDate
    $hts = get-exchangeserver |
      where {$_.serverrole -match “hubtransport”} 
    foreach ($ht_server in $hts){
       $ht = get-transportserver $ht_server.name 
       if ($ht.messagetrackinglogenabled){
               $log_unc = “\\$($ht.name)\$($ht.messagetrackinglogpath -replace “:”,”$”)”
               $logs = gci $log_unc\*.log |
                where {$_.name -le $TargetLog}
            }
        $logs
        }


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    • Proposed as answer by wendy_liu Monday, November 5, 2012 2:11 AM
    Saturday, November 3, 2012 2:47 AM
  • Hi mjolinor,

    with reference to your reply on similar question another post, I have one management server where only Exchange management tools are installed.so if i copy both index files & message tracking log files from Exchange HUB server Log folder to Management server. Can i run message tracking commands to trace messages?
    If it is possible it wud be useful for us as it wud offload work from prodution HUB server to management server for running routine message traces & also i can retain message tracking logs for unlimited duration thereby.

    Sunday, November 4, 2012 6:42 AM
  • I tried to explain in the other post that you need that log search service running on the server where the logs are in order to use the Exchange tools to search the logs.  Your management server is not going to have that, so the answer is 'No'.


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    Sunday, November 4, 2012 12:59 PM
  • OK. So if i just installed HUB role on management server & have Message tracking logs & index files then i can message trace. Right?

    only reason doing this is we need to keep tracking logs for unlimited retention due to business requirement. & due to space crunch we can keep till only three months of log on production server so we shall need to shift old logs to different server & run message trace from there.
    Sunday, November 4, 2012 1:10 PM
  • Yes, you can install the Hub Transport role on the management server.  If you're going to do that, I would un-install it from the Exchange server, so you don't have to worry about migrating log files.

    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    Sunday, November 4, 2012 1:28 PM
  • yes i have asked it because i just want to check the possibility to check any dependency...say one day due to space issue we have moved 20 days message tracking log files to another location then we created enough free space & moved back all 20 log files back into folder. Do we able to run message trace from those log files I mean any reason why it will not do message trace.

    One query can we move message tracking log files & index files to Exchange test environment which is different from Production exchange organization & run message trace there? Just checking the possibility?  
    Monday, November 5, 2012 4:10 AM
  • I'd think it would work as long as they are put back on the same server they came from.  I can't say whether moving them to another server in a test environment will work or not, but I suspect you'd have better luck if you stop the transport service on that server, and delete all the existing logs and indexes before you move the archived logs and indexes to it for searching. 


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    • Proposed as answer by wendy_liu Tuesday, November 13, 2012 1:19 AM
    • Marked as answer by wendy_liu Tuesday, November 13, 2012 1:20 AM
    Monday, November 5, 2012 2:15 PM
  • thanks
    it seems to be working.. i have moved message tracking logs off from production server to Test exchange HUB server with altogether different exchange organization, forest. Deleted existing logs files & index files. I fired Get-messagetrackinglog command & guess what it is showing message traces of production emails.

    it will be really helpful. Do you find any challenge we could face in running some custom or selective query in message tracking command with output,timestamp,accuracy.

    Ok so for now can we use Robocopy to move message tracking logs from production server to External storage & keep only 3 months message trace files in production server. Any script you wud advice.

    Friday, November 9, 2012 9:21 AM
  • I was mucking around with this earlier, and finally got back to getting the bug cornered and stomped on.  It uses the DotNetZip library from CodePlex. 

    http://dotnetzip.codeplex.com/

    $RetentionDays = 55
    $ArchiveDir = <archive dir>
    Add-Type -Path 'C:\Program Files (x86)\Dino Chiesa\DotNetZip Runtime 1.9\Ionic.Zip.dll'
    $TargetDate = ((get-date).AddDays(-$RetentionDays)).ToString('yyyyMMdd')
    $TargetLog = 'MSGTRK{0}*' -f $TargetDate
    $hts = get-exchangeserver |
      where {$_.serverrole -match “hubtransport”} 
      
    foreach ($ht_server in $hts){
       $ht = get-transportserver $ht_server.name 
       $log_unc = “\\$($ht.name)\$($ht.messagetrackinglogpath -replace “:”,”$”)”
       $log_ht = @{}
       $logs = gci $log_unc\*.log |
         where {$_.name -lt $TargetLog} |
          foreach {
                    $log_ht[$_.name.Substring(0,14)] += @($_.name)
                  }
            
       foreach ($day in $log_ht.keys){
        $TargetZip = "$ArchiveDir\$($ht.name)\$day.zip"
        $zipfile =  new-object Ionic.Zip.ZipFile($TargetZip)
          $zipfile.addentry("_Zipfile_Readme.txt","Zip of logs from $log_unc for $($day.substring(6,8))") 
          foreach ($Logfile in $log_ht[$day]) {
            $zipfile.AddFile("$log_unc\$Logfile","\")
            $zipfile.AddFile("$log_unc\Index\$Logfile.message-id.idx","\Index")
            $zipfile.AddFile("$log_unc\Index\$Logfile.sender-address.idx","\Index")
          }
        $zipfile.Save()
       }
    }       

    It will find all the log and index files older than the specificed retention days, and package up each day into a separate zip file, with all the logs for that day in the root of the zip, and all the index files for that day in an /Index subdirectory of the zip. 

    It stores them in the $ArchiveDir path, in a subdirectory per hub server (directory name is the server name).  There's no error checking or automatic directory creation, so you'd need to add that. Limited testing shows the index files compress at about 35%, and the log files compress at about 85%.

    I chose the DotNetZip library because the methods seem to closely match the zip file support that's being included in dotnet 4.5, so conversion to native dotnet methods later should be fairly trivial if you want to eliminate the dependency on the third-party dll.


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    • Proposed as answer by wendy_liu Monday, November 12, 2012 2:26 AM
    • Marked as answer by wendy_liu Tuesday, November 13, 2012 1:20 AM
    Saturday, November 10, 2012 12:27 AM
  • Thanks a lot. appreciate ur efforts. I will apply above script & let u know how it goes.
    Saturday, November 10, 2012 10:27 AM