Nginx error log Grok pattern

groklogginglogstashnginxregex

I am having trouble getting the following nginx error log message to parse in the grok debugger. I have a feeling there is a stupid trick that I should use but can't figure out what it may be.

2015/03/20 23:35:52 [error] 8#0: *10241823 testing "/www" existence failed (2: No such file or directory) while logging request, client: 201.45.203.78, server: $domain, request: "GET /ritikapuri_"

Here is my Grok pattern so far:

(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage} client: %{IP:client}

This pattern gets me to the "server" section but I can't seem to get the rest to parse and it isn't clear to me why.

If I use another %{GREEDYDATA} pattern to grab the end of the log it sometimes wont' parse logs that don't match the above and give me a _grokparsefailure.

Would the best route be to use if statements to trap the different variations of log messages in nginx?

I have followed methods including this one but can't get them working.

Best Answer

I used @dr01's answer to improve the recipe for error logs in nginx 1.15 using the notice format - this answer will separate out the HTTP version and HTTP method and request.

(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:message}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion}))", host: %{GREEDYDATA:host}

sample string

2015/03/20 23:35:52 [error] 8#0: *10241823 testing "/www" existence failed (2: No such file or directory) while logging request, client: 201.45.203.78, server: $domain, request: "GET /dsfadsfe HTTP/1.1", host: "localhost:8080"

Output from grok debugger

{
  "timestamp": [
    [
      "2015/03/20 23:35:52"
    ]
  ],
  "severity": [
    [
      "error"
    ]
  ],
  "pid": [
    [
      "8"
    ]
  ],
  "threadid": [
    [
      "0"
    ]
  ],
  "connectionid": [
    [
      "10241823"
    ]
  ],
  "message": [
    [
      "testing "/www" existence failed (2: No such file or directory) while logging request"
    ]
  ],
  "client": [
    [
      "201.45.203.78"
    ]
  ],
  "server": [
    [
      "$domain"
    ]
  ],
  "verb": [
    [
      "GET"
    ]
  ],
  "request": [
    [
      "/dsfadsfe"
    ]
  ],
  "httpversion": [
    [
      "1.1"
    ]
  ],
  "host": [
    [
      ""localhost:8080""
    ]
  ]
}
Related Topic