none
Decode Key Value Pairs RRS feed

  • Question

  • Hi,

    Can Message Nalyzer decode logfile that are in the form of <DateTime> <Key>=<Value> <Key2>=<Value2>?

    For example

    08.09.2016 19:55:58 MyField1="test" MyField2="dummy"
    08.09.2016 19:56:58 MyField1="test" MyField2="dummy"
    08.09.2016 19:59:58 MyField1="test" MyField2="dummy"

    If not, could I add a custom parser for that?

    Thanks for your help.

    Meinrad

    Monday, September 12, 2016 10:23 AM

All replies

  • You can write a custom parser, but the field names will have to be collected as a map, so that the expression you write in filters look like your_module.fields["test"]="dummy" instead of your_module.test="dummy".  The HTTP parser does the same thing when it parses fields in headers, since they are also described as value pairs.

    You can see examples of log parsers in the ones we ship with the product and this might help you understand how to get started.  They are located in %LOCALAPPDATA%\Microsoft\MessageAnalyzer\OPNAndConfiguration\TextLogConfiguration.  If you need more help, let us know.

    Paul

    • Proposed as answer by Paul E Long Monday, September 12, 2016 8:02 PM
    Monday, September 12, 2016 8:02 PM
  • Thanks for your help Paul.

    I just saw, that the Json DataType has the functionality that I am looking for (dynamic?). When I put a Json object in a log file I can open it and filter like this (JSON.test == "value"). Could I leverage that somehow?

    Tuesday, September 13, 2016 6:01 AM
  • We have special code to open Json data.  There's an API we plug into, and then expose that to the Protocol Object Model (POM).  There's not an easy way to plug-in, though it is possible.  However, I think writing a config file might be easier.

    I started one for you, though it doesn't work yet.  There's still some issues we'll need to figure out, but I'm including it below so you can perhaps play around with it some as well.  When I get some more time, I'll move it forward as well.

    //
    // Property to return date type given a message of type DateTime.  Aspects for IsTimestamp allow you to replace the timestamp value.
    //
    DateTime get LogDate(this MsgDateTime msg) with EntryFieldInfo {IsTimestamp = true, IsLocalTime = true }
    {
        optional DateTime theDateTime = ToDateTime(msg.ToString());
        if(theDateTime == nothing)
        {
            return ToDateTime("1970-01-01T00:00:00.0000000") as DateTime;
        }
        else 
        {
            return theDateTime as DateTime;
        }
    }
    
    message MsgDateTime : LogEntry
    {
        string Month;
    	string Day;
    	string Year;
        string Time;
        
        override string ToString()
        {
            return Year + "-" + Month + "-" + Day + "T" + Time;
        }
    }
    
    message DynamicLogEntry
              with EntryInfo { Regex = @"(?<Month>\d\d)\.(?<Day>\d\d)\.(?<Year>\d\d\d\d)\s(?<Time>[^\s]*)(\s)(?<Params>.*)" }
              : MsgDateTime
    {
    	string Params;
    	
    }
    
    message DynamicFields
    {
    	map<string,string> fields;
    }
    
    //Parsed messages arrive here
    endpoint DynamicMsgEndpoint accepts any message;
    /*
    {
        array<string> FieldsPresent = [];
    }
    */
    
    autostart actor SinkToDynamicLogs(LogSink s)
    {
        //Parses the fields line
        process s accepts m:LogEntry where m is entry:DynamicLogEntry{}
        {
            var e = endpoint DynamicMsgEndpoint;
    		
    		DynamicFields l = ConstructDynamicMsg(entry.Params);
            dispatch e accepts l;
    /*
            //Parses the params field
            var res = IISFields(entry.Params);
            if (res != nothing)
            {
                array<string> fields = res as array<string>;
                if (e.FieldsPresent == [])
                {
                    release entry;
                }
                else
                {
                    IISLine_W3C l = ConstructLine_W3C(e.FieldsPresent, fields);
                    dispatch e accepts l;
                }
            }
            else
            {
                ValidationCheck(false, entry, "Unexpected format in IIS log line.");
                release entry;
            }
        */
    	}	
    }
    
    DynamicFields ConstructDynamicMsg(string params)
    {
    
        DynamicFields m = new DynamicFields();
    	m.fields["hi"] = "blah";
    	
    	return m;
    }
    

    Wednesday, September 21, 2016 12:51 PM
  • Thanks, that was exactly what I needed to get started. I played around with it and came up with this so far:

    //
    // Property to return date type given a message of type DateTime.  Aspects for IsTimestamp allow you to replace the timestamp value.
    //
    DateTime get LogDate(this DynamicLogEntry msg) with EntryFieldInfo {IsTimestamp = true, IsLocalTime = false }
    {
        optional DateTime theDateTime = ToDateTime(msg.Timestamp);
        if(theDateTime == nothing)
        {
            return ToDateTime("1970-01-01T00:00:00.0000000") as DateTime;
        }
        else 
        {
            return theDateTime as DateTime;
        }
    }
    
    message DynamicLogEntry
              with EntryInfo { Regex = @"(?<Timestamp>\d\d\d\d-\d\d-\d\dT[^\s]+)(\s)(?<Params>.*)" }
              : LogEntry
    {
        string Timestamp;
    	string Params;
    }
    
    message DynamicFields
    {
    	optional map<string,string> fields;
    }
    
    //Parsed messages arrive here
    endpoint DynamicMsgEndpoint accepts any message;
    
    pattern Quote = regex{"};
    pattern StringValue = regex{[^"]+};
    pattern NummericValue = regex{[0-9\.]+};
    
    syntax HeaderField = k:(regex{[^=]+}) "=" v:((Quote s:StringValue Quote => s)|n:NummericValue) =>
                                        new HeaderFieldType
                                        {
                                            Key = k.Trim(),
                                            Value = v.Trim(),
                                        };
    
    syntax KeyValuePairs = kv:(HeaderField)* => MergeHeaders(kv);
    
    type HeaderFieldType
    {
        string Key;
        string Value;
    }
    
    
    autostart actor SinkToDynamicLogs(LogSink s)
    {
        //Parses the fields line
        process s accepts m:LogEntry where m is entry:DynamicLogEntry{}
        {
            var e = endpoint DynamicMsgEndpoint;
    		
    		DynamicFields l = ConstructDynamicMsg(entry.Params);
            dispatch e accepts l;
    	}	
    }
    
    map<string, string> MergeHeaders(array<HeaderFieldType> headFields)
    {
        map<string, string> headers = {}.ToOrdered();
        foreach (HeaderFieldType headerField in headFields)
        {
            if (!(headerField.Key in headers))
            {
                headers[headerField.Key] = headerField.Value;
            }
            else
            {
                headers[headerField.Key] += "\r\n" + headerField.Value;
            }
        }
        
        return headers;
    }
    
    
    DynamicFields ConstructDynamicMsg(string params)
    {
        DynamicFields m = new DynamicFields();
    	m.fields = KeyValuePairs(params);
    	
    	return m;
    }

    this seems to work for my log that looks like this:

    2016-09-21T13:24:29.4469278Z ProviderGuid="e13c0d23-ccbc-4e12-931b-d9cc2eee27e4" ProviderName="Microsoft-Windows-DotNETRuntime" EventName="ExceptionCatch/Start" ProcessID=8008 ThreadID=25192 Keywords="0000000000008000" Level="Informational" TaskName="ExceptionCatch" OpcodeName="Start" EntryEIP="117 494 896" MethodID="92 120 716" MethodName="instance void [Microsoft.ApplicationInsights.PersistenceChannel] Microsoft.ApplicationInsights.Channel.Storage+<LoadTransmissionFromFileAsync>d__14::MoveNext()" ClrInstanceID="12"
    2016-09-21T13:24:29.4470620Z ProviderGuid="e13c0d23-ccbc-4e12-931b-d9cc2eee27e4" ProviderName="Microsoft-Windows-DotNETRuntime" EventName="Exception/Start" ProcessID=8008 ThreadID=25192 Keywords="0000000200008000" Level="Error" TaskName="Exception" OpcodeName="Start" ExceptionType="System.UriFormatException" ExceptionMessage="Invalid URI: The format of the URI could not be determined." ExceptionEIP="0x700d663" ExceptionHRESULT="-2 146 233 033" ExceptionFlags="ReThrown, CLSCompliant" ClrInstanceID="12"
    2016-09-21T13:24:29.4471084Z ProviderGuid="e13c0d23-ccbc-4e12-931b-d9cc2eee27e4" ProviderName="Microsoft-Windows-DotNETRuntime" EventName="ExceptionCatch/Start" ProcessID=8008 ThreadID=25192 Keywords="0000000000008000" Level="Informational" TaskName="ExceptionCatch" OpcodeName="Start" EntryEIP="117 494 896" MethodID="92 120 716" MethodName="instance void [Microsoft.ApplicationInsights.PersistenceChannel] Microsoft.ApplicationInsights.Channel.Storage+<LoadTransmissionFromFileAsync>d__14::MoveNext()" ClrInstanceID="12"
    

    What I don't get, is how I can get rid of the two different messages in there (DynamicFields and DynamicLogEntry). Can I merge them somehow?

     
    Thursday, September 22, 2016 6:45 AM
  • Since you are parsing the raw line, I believe the new message is necessary as you are constructing more fields. IIS does the same thing.  If you want to have the same infromaiton in the DynamicFields message, you could copy the Params and/or timestamp fields.

    Paul

    Tuesday, October 4, 2016 2:38 PM