Nginx – How to parse an nginx msec field in logstash

logstashnginx

I'm trying to handle an nginx access log in logstash.

In order to get millisecond accuracy for my timestamps, I'm using the $msec variable. This means that timestamps will be of the form 1430832725.814, where the integer part is a UNIX timestamp, and the fraction part is in milliseconds. Unfortunately, I can't find a logstash date format which can parse UNIX timestamps.

Is there another way to get nginx to log with millisecond accuracy? Or a way to get the logstash date filter to parse UNIX timestamps?

Best Answer

The UNIX format literal can handle fractional seconds, so it will be able to parse an $msec field.

Related Solutions

How to parse audit.log using logstash

A quick search finds this on github

AUDIT type=%{WORD:audit_type} msg=audit\(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}\): user pid=%{NUMBER:audit_pid} uid=%{NUMBER:audit_uid} auid=%{NUMBER:audit_audid} subj=%{WORD:audit_subject} msg=%{GREEDYDATA:audit_message} 
AUDITLOGIN type=%{WORD:audit_type} msg=audit\(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}\): login pid=%{NUMBER:audit_pid} uid=%{NUMBER:audit_uid} old auid=%{NUMBER:old_auid} new auid=%{NUMBER:new_auid} old ses=%{NUMBER:old_ses} new ses=%{NUMBER:new_ses}

A cursory review suggests it's probably what you're looking for.

Nginx logging: Is there a way to log request time in milliseconds since Unix epoch

There are 2 ways to achieve this. The first one is dirty and I wont recommend it - it's the fastest way though:

if ($msec ~ "(.*)\.(.*)") {
    set $epoch_millis = "$1$2"
}

This is much cleaner but requires lua:

set_by_lua_block $epoch_millis { return string.gsub(ngx.var.msec, "%.", "") }

then just log $epoch_millis

Best Answer

Related Solutions

How to parse audit.log using logstash

Nginx logging: Is there a way to log request time in milliseconds since Unix epoch

Related Topic