Frequency of occurrence calculations - DateTime Stamp records - frequency

Ask a question

Wednesday, February 29, 2012 4:25 PM

0
I am looking to iteratively go through event log records which have as
one of the fields Date-Time Stamp. I needed to do various frequency counts of
occurrence of Date-Time field. How many events of a certain type occurred per hour, how many per day,
how many per week, and how many per-month, and this has to be dynamic, with no static entries in the script.

For instance, I am able to do something as follows:

$startDate = get-date "2/10/2012 8:00 AM"
$endDate = get-date "2/10/2012 10:00 PM"

$EventOverATimeBand=get-eventlog -logname application -entrytype Information -Source "<Source>" -after $startDate -before $endDate$EventOverATimeBand | group-object -property source -noelement | sort-object -property count –descending

My issue is in the first two lines, I have to do something
dynamically, such as (a) from the current point in time to one hour back, (b)same for each day, (c) each week (d) a month. Note that “Source” could beanything.

I have appended a sample of the events. In it you can see
events with various time stamps. As you can see I need the following.
1. Count of All events with string “Starting Core Dump” for a monthly period.
2. Count of same same as above for past 4 weeks, on a weekly basis.
3. Count of same if they had occurred say about more than twice in an hour.
I am appending a sample EventLog below.

Thanks,
Girish.

<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 2:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 3:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 2:01:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 2:25:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 2:55:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 3:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 4:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 4:30:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 6:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

22, 2012 8:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

23, 2012 2:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

24, 2012 2:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

25, 2012 2:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 2:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 3:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 2:01:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 2:25:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 2:55:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 3:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 4:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 4:30:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 6:00:00 AM" />
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.

CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

   8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 8:00:00 AM" />
gpillai
- Reply
- Quote

All Replies

Wednesday, February 29, 2012 4:35 PM

0

Here are some hints:

1. Use Get-Winevent if you want to access the XML data.

2. Use FilterXML or FilterXPath to select between dates.

3. Use the .toXML() method of the event itself to access the event data.
Grant Ward, a.k.a. Bigteddy
What's new in Powershell 3.0 (Technet Wiki)
- Reply
- Quote
Wednesday, February 29, 2012 5:27 PM

0

Hi Grant,

Thanks for the response.

1. I am not interested in accessing the data per se. I have already collected the data. My need is to calculate the "frequency of occurrence" of the event.

2. Again, the objective is not to select event between dates, but to count the occurrence. I dont want to hard code any date times. IF you see the example I started off, it slready uses dates.

Like I said, say I have a 6 month (or years) dump of the event log, which could be about a Gig worth of data or more. This has been already been collected and given to me. Now I need to do analysis, in which I specifically look for the occurrence of certain types of events, and then count the frequency of occurrence.

(a) If the event occurred more then twice in an hour, I need to flag it.
(b) If the event occurred more then say 3 times in a day I need to flag it.
(c) If the event ocurred more than 5 times a week then flag it.
(d) If the event occurred more than 10 times in a month flag it.

And I need to do this without hardcoding any dates, but programmatically.

Thanks,
Girish
GP
- Reply
- Quote
Wednesday, February 29, 2012 5:29 PM

0

What format are your saved event log files in? evtx / csv ?
Grant Ward, a.k.a. Bigteddy
What's new in Powershell 3.0 (Technet Wiki)
- Reply
- Quote
Wednesday, February 29, 2012 5:43 PM

0

It resembles evtx, though it is not exactly evtx. I use another tool to extract the event logs data.

For purposes of our discussion, the colletected data is record based as in the very first mail I sent at the beginning of the thread. You can assume the record to contain such similar entries within each of them. The essential thing is that each record will have a Date-Time stamp and one another field which is a Description Text/Message that is in English. I look for particular message strings to exist on that text record. So for example in the sample I provided, the fields to key off will be: Message and DataGenerated.

Example:
<EventDetail UserID="MattG" ReplacementStrings="" Message="Starting Core Dump.
CurrentCacheSize : 7083937792, CacheLimit : 7023187968, Space To Free up :

1416787558.

Application Domain: TfsJobAgent.exe
Service Host:

8ae08be4-ed33-48dc-b1b5-5c68e115217c (Server CORE)" DateGenerated="Wednesday, February

26, 2012 2:00:00 AM" />

Thanks,
Girish
GP
- Reply
- Quote
Wednesday, February 29, 2012 5:46 PM

Moderator

1
Does this help?

http://mjolinor.wordpress.com/2012/01/22/counting-and-grouping-log-entries-by-arbitrary-time-spans-with-powershell/

[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
- Proposed As Answer by jrvMicrosoft Community Contributor Wednesday, February 29, 2012 6:19 PM
- Marked As Answer by IamMredMicrosoft Employee, Owner Monday, March 05, 2012 3:01 AM
- Reply
- Quote
Wednesday, February 29, 2012 6:13 PM

0

Hello Mjolinor,

Let me check this one. Seems promising. The collection part is already done, so I might have to spilt the logic, as the events are collected by another tool.

Thanks,
Girish.
GP
- Reply
- Quote
Thursday, March 01, 2012 2:59 PM

0

Thanks this was very useful. Appreciate it.

-Girish.
GP
- Reply
- Quote
Thursday, March 01, 2012 3:06 PM

Moderator

0

You're welcome!

I knew somebody would want to do that some day.
[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
- Reply
- Quote

Frequency of occurrence calculations - DateTime Stamp records - frequency

All Replies

Products

Resources

Solutions

Updates

Trials

Related Sites

Training

Certifications

Other resources

Support by product

Other support links

Not an IT pro?