Sending logs to remote server with rsyslog

The problem was actually coming from logrotate.

Basically with my configuration, running unicorn, I don't need to use the copytruncate directive. (which is what causes problems here)

USR1 - Reopen all logs owned by the worker process. See Unicorn::Util.reopen_logs for what is considered a log. Log files are not reopened until it is done processing the current request, so multiple log lines for one request (as done by Rails) will not be split across multiple logs.

This started working properly after updating to this configuration:

/home/user/my_app/shared/log/*.log {
  daily
  missingok
  dateext
  rotate 30
  compress
  notifempty
  extension gz
  create 640 user user
  sharedscripts

  post-rotate
    # Telling Unicorn to reload files
    test -s /home/user/my_app/shared/pids/unicorn.pid && kill -USR1 "$(cat /home/user/my_app/shared/pids/unicorn.pid)"

    # Reloading rsyslog telling it that files have been rotated
    reload rsyslog 2>&1 || true
  endscript
}

Do you not have to parse out [program] first? I don't think the 'input' field does any sort of filtering at all, so you might need to start with %SYSLOGBASE http://logstash.net/docs/1.4.1/filters/grok

You could try instead perhaps:

if [message] =~ /nginx-access/ {

Which'll keyword match your message field. That'll at least tell you if that is what's happening here.