Logstash / Elasticsearch – trasform fields to lowercase in output conf

elasticsearchlogstash

I have a standard ELK stack currently storing numerous log outputs. I'm trying to separate my indices to be source-specific.

As part of my FileBeats config, some standard fields are always generated as part of every message (and are location-specific), which I want to use as the basis for my my ES index:

output {
  elasticsearch {
    hosts => ["http://elasticsearch.mydomain.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{[fields][identifier]}"
  }
}

However, ES is rejecting some of the indices as the field contains uppercase characters – identifier has acceptable values like myfoo but also could be MyBar:

[logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {"reason"=>"Invalid index name [logstash-2017.06.02-MyBar], must be lowercase"}

The casing isn't essential and I can add a mutate filter to forcibly lowercase the fields in question, but I would prefer to store the identifier field with proper casing, yet use the lower-cased version for the index name.

Is there function which can be called in the elasticsearch output, to lower-case the field in question? Something like 

    index => "logstash-%{+YYYY.MM.dd}-%{[fields][identifier]}.lowercase()"

Best Answer

This can be done with a bit more mutate trickery.

  1. Create a new field using mutate, set to your identifier.
  2. In a second mutate, lowercase that new field.
  3. Use the new field in your output.

Like so:

filter {
  mutate {
    add_field => { "lc_identifier" => "%{fields}%{identifier}" }
  }
  mutate {
    lowercase => [ "lc_identifier" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch.example.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{[lc_identifier]}"
  }
}

You will end up with an lc_identifier field in your indices, but that shouldn't matter much.

Related Topic