How to add AWS ELB access logs to logstash with S3 input

amazon s3amazon-elbelklogstash

I'm using an ELK stack for server monitoring. My application's access logs which are from AWS ELB are stored in AWS S3. I am trying to add them to logstash with the following input:

input {
  s3 {
    access_key_id => "my_id"
    secret_access_key => "my_key"
    bucket => "my_bucket"
    region => "region"
    prefix => "AWSLogs/828557649675/elasticloadbalancing/eu-west-1/**/**/**/*.log"
    type => "elb"
  }
}

But nothing is added to logstash, what am I doing wrong? I there another way to add them to logstash?
The filter and output parts seems OK so I don't post them.

Note: I am using logstash 2.1 version

Best Answer

Seems like the prefix should be, as the plugin does not support globs (*) :

 prefix => "AWSLogs/828557649675/elasticloadbalancing/eu-west-1/"

Full example:

input {
  s3 {
    access_key_id => "my_id"
    secret_access_key => "my_key"
    bucket => "my_bucket"
    region => "region"
    prefix => "AWSLogs/828557649675/elasticloadbalancing/eu-west-1/"
  }
}

Related Solutions

Logstash failing to parse syslog input

The syslog input use grok internally, your message is probably not following the syslog standard 100%.

The solution in this link worked for me: http://kartar.net/2014/09/when-logstash-and-syslog-go-wrong/

The key info from the link is:

Replace the existing syslog block in the Logstash configuration with:

input {
  tcp {
    port => 514
    type => syslog
  }
  udp {
    port => 514
    type => syslog
  }
}

Next, replace the parsing element of our syslog input plugin using a grok filter plugin.

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
  }
}

You can edit the filter matching ("grok") syntax now, to match your desired format. It's also possible to support multiple different syntaxes with creative use of if, else if, and else.

Logstash S3 input plugin re-scanning all bucket objects

This seems to be the default behaviour for this plugin, so it has to be managed using the plugin features.

Basically, you have to set up the plugin to backup-then-delete the objects with a prefix to the same bucket. In that way, Logstash will skip object when it polls the bucket after the next interval.

Sample config:

s3 {
   bucket => "s3-access-logs-eu-west-1"
   type => "s3-access"
   prefix => "2016-"
   region => "eu-west-1"
   sincedb_path => "/tmp/last-s3-file-s3-access-logs-eu-west-1"
   backup_add_prefix => "logstash-"
   backup_to_bucket => "s3-access-logs-eu-west-1"
   interval => 120
   delete => true
 }

This config will scan the bucket ever 120 seconds for objects starting with

2016-

It will process those objects, then back them up to the same bucket with prefix

logstash-

then delete them.

This means they won't be found at the next polling interval.

2 important notes:

You can't use backup_add_prefix by itself (the docs suggest you can). You can only use this parameter in conjunction with backup_to_bucket
Make sure the IAM account/role you are using to interface with S3 has Write permissions for the buckets you are using (other Logstash can't delete/rename objects).

Best Answer

Related Solutions

Logstash failing to parse syslog input

Logstash S3 input plugin re-scanning all bucket objects

Related Topic