none
Frequency of occurrence calculations - DateTime Stamp records - frequency

    Question

  • I am looking to iteratively go through event log records which have as
    one of the fields Date-Time Stamp. I needed to do various frequency counts of
    occurrence of Date-Time field. How many events of a certain type occurred per hour, how many per day,
    how many per week, and how many per-month, and this has to be dynamic, with no static entries in the script.

    For instance, I am able to do something as follows:

    $startDate = get-date "2/10/2012 8:00 AM"
    $endDate = get-date "2/10/2012 10:00 PM"

    $EventOverATimeBand=get-eventlog -logname application -entrytype Information -Source "<Source>" -after $startDate -before $endDate$EventOverATimeBand | group-object -property source -noelement | sort-object -property count –descending

    My issue is in the first two lines, I have to do something
    dynamically, such as (a) from the current point in time to one hour back, (b)same for each day, (c) each week (d) a month. Note that “Source” could beanything.

    I have appended a sample of the events. In it you can see
    events with various time stamps. As you can see I need the following.

    1. Count of All events with string “Starting Core Dump” for a monthly period.
    2. Count of same same as above for past 4 weeks, on a weekly basis.
    3. Count of same if they had occurred say about more than twice in an hour.

    I am appending a sample EventLog below.

    Thanks,
    Girish.

    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 2:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 3:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 2:01:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 2:25:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 2:55:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 3:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 4:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 4:30:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 6:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    22, 2012 8:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    23, 2012 2:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    24, 2012 2:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    25, 2012 2:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 2:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 3:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 2:01:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 2:25:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 2:55:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 3:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 4:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 4:30:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 6:00:00 AM" />
    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:   

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 8:00:00 AM" />


    gpillai

    Wednesday, February 29, 2012 4:25 PM

Answers

All replies

  • Here are some hints:

    1. Use Get-Winevent if you want to access the XML data.

    2. Use FilterXML or FilterXPath to select between dates.

    3. Use the .toXML() method of the event itself to access the event data.


    Grant Ward, a.k.a. Bigteddy

    What's new in Powershell 3.0 (Technet Wiki)

    Wednesday, February 29, 2012 4:35 PM
  • Hi Grant,

    Thanks for the response.

    1. I am not interested in accessing the data per se. I have already collected the data. My need is to calculate the  "frequency of occurrence" of the event.

    2. Again, the objective is not to select event between dates, but to count the occurrence. I dont want to hard code any date times. IF you see the example I started off, it slready uses dates.

    Like I said, say I have a 6 month (or years) dump of the event log, which could be about a Gig worth of data or more. This has been already been collected and given to me. Now I need to do analysis, in which I specifically look for the occurrence of certain types of events, and then count the frequency of occurrence.

    (a) If the event occurred more then twice in an hour, I need to flag it.
    (b) If the event occurred more then say 3 times in a day I need to flag it.
    (c) If the event ocurred more than 5 times a week then flag it.
    (d) If the event occurred more than 10 times in a month flag it.

    And I need to do this without hardcoding any dates, but programmatically. 

    Thanks,
    Girish


    GP

    Wednesday, February 29, 2012 5:27 PM
  • What format are your saved event log files in?  evtx / csv ?

    Grant Ward, a.k.a. Bigteddy

    What's new in Powershell 3.0 (Technet Wiki)

    Wednesday, February 29, 2012 5:29 PM
  • It resembles evtx, though it is not exactly evtx. I use another tool to extract the event logs data.

    For purposes of our discussion, the colletected data is record based as in the very first mail I sent at the beginning of the thread. You can assume the record to contain such similar entries within each of them. The essential thing is that each record will have a Date-Time stamp and one another field which is a Description Text/Message that is in English. I look for particular message strings to exist on that text record. So for example in the sample I provided, the fields to key off will be: Message and DataGenerated.

    Example:

    <EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

    CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

    1416787558.&#xD;&#xA;&#xD;&#xA;Application Domain: TfsJobAgent.exe&#xD;&#xA;Service Host:  

       8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

    26, 2012 2:00:00 AM" />

    Thanks,
    Girish


    GP

    Wednesday, February 29, 2012 5:43 PM
  • Does this help?

      http://mjolinor.wordpress.com/2012/01/22/counting-and-grouping-log-entries-by-arbitrary-time-spans-with-powershell/


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    Wednesday, February 29, 2012 5:46 PM
    Moderator
  • Hello Mjolinor,

     Let me check this one. Seems promising. The collection part is already done, so I might have to spilt the logic, as the events are collected by another tool.

    Thanks,
    Girish.


    GP

    Wednesday, February 29, 2012 6:13 PM
  • Thanks this was very useful. Appreciate it.

    -Girish.


    GP

    Thursday, March 01, 2012 2:59 PM
  • You're  welcome!

     

    I knew somebody would want to do that  some day.


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    Thursday, March 01, 2012 3:06 PM
    Moderator