PHP – Fix Line Breaks in Stack Trace in Nginx Error Logs for Logstash Analysis

elkfastcgilogstashnginxphp-fpm

I am using nginx with PHP-FPM and ELK as log file analysis.

When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file.

Problem is: Sometimes those error logs contains line breaks which logstash cannot handle, because line breaks are considered as a new log line. 

2019/04/17 19:23:00 [error] 8356#8356: *4403 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function wp_using_themes() in /htdocs/wp-includes/template-loader.php:7
Stack trace:
#0 /htdocs/wp-blog-header.php(19): require_once()
#1 /htdocs/index.php(17): require('/htdocs/wp-blog...')
#2 {main}
  thrown in /htdocs/wp-includes/template-loader.php on line 7" while reading response header from upstream, client: 123.123.123.123, server: foobar.de, request: "GET /2014/11/foobar/ HTTP/1.1", upstream: "fastcgi://unix:/run/php/php-fpm-foobar.sock:", host: "foobar.de"

How to I either handle those linebreaks with logstash or format those error messages to remove the line breaks?

Best Answer

Kudos to @USD-Matt thanks to his hint I now know where to look at and that's the solution:

There is a multiline feature in the ELK stack. But as I am using the filebeat module to postprocess the logs, I cannot just activate the multiline feature in logstash as mentioned above.

I have to enable it in filebeat itself (/etc/filebeat/filebeat.yml) as described here: https://www.elastic.co/guide/en/beats/filebeat/master/multiline-examples.html

But heads up: I am also using the filebeat module "nginx" which claims to handle the multiline issue itself. It doesn't. You can force it to do so by adding the above multiline settings to the module config file: /etc/filebeat/modules.d/nginx.yml

So I did, and that's how the particular part looks like now:

  error:
    enabled: true
    input:
      multiline.pattern: '^\d{4}\/\d{2}\/\d{2}'
      multiline.negate: true
      multiline.match: after

    var.paths:
      - /var/nginx/foobar_de/logs/error.log*
Related Topic