Docker – Logstash issue matching a substring of custom field

dockerlogstashregex

I have the following Logstash setup.

Logs are piped to Logstash from Logspout via the logspout-logstash adapter, which adds a few fields to log messages, namely the docker.image field.

I am able to ingest the logs but am having trouble parsing them. I would like to make some filters based on the Docker image field, below I'm trying to parse and match just the nginx piece out of the full Docker image, which is similar to dockerhubuser/nginx:tag.

There is something wrong with my config though because the tag doesn't look like it is being created and the message field doesn't look like it is being parsed either.

Here's what I have in my config so far:

input {

    # Logspout UDP input
    udp {
        port => 5000
        type => logspout
        codec => json
    }
}

filter {

  # Nginx access logs
  if [docker.image] =~ /nginx/ {
    grok {
      match => [ "message", "%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:agent}" ]
      add_tag => [ "nginx" ]
    }
  }
}

Is the regex not matching? Is there a way to check to see if a log is hitting my filter?

Best Answer

I found the solution. To access nested fields in the logstash config I had to use [docker][image]. So the fixed config looks like this:

if [docker][image] =~ /nginx/ {
    grok {
      match => [ "message", "%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:agent}" ]
      add_tag => [ "nginx" ]
    }
  }
Related Topic