I'm trying to configure logstash to send email alerts and log output in elasticsearch / kibana.
I have the logs successfully syncing via rsyslog, but I get the following error when I run
/opt/logstash-1.4.1/bin/logstash agent -f /opt/logstash-1.4.1/logstash.conf –configtest
Error: Expected one of #, {, ,, ] at line 23, column 12 (byte 387) after filter {
if [program] == "nginx-access" {
grok {
match => [ "message" , "%{IPORHOST:remote_addr} – %{USERNAME:remote_user} [%{HTTPDATE:time_local}] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}” ]
}
}
}
output {
stdout { }
elasticsearch {
embedded => false
host => "
Here is my logstash config file
input {
syslog {
type => syslog
port => 5544
}
}
filter {
if [program] == "nginx-access" {
grok {
match => [ "message" , "%{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[% {HTTPDATE:time_local}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}” ]
}
}
}
output {
stdout { }
elasticsearch {
embedded => false
host => "localhost"
cluster => "cluster01"
}
email {
from => "logstash.alert@nowhere.com"
match => [
"Error 504 Gateway Timeout", "status,504",
"Error 404 Not Found", "status,404"
]
subject => "%{matchName}"
to => "you@example.com"
via => "smtp"
body => "Here is the event line that occured: %{@message}"
htmlbody => "<h2>%{matchName}</h2><br/><br/><h3>Full Event</h3><br/><br/><div align='center'>%{@message}</div>"
}
}
I've checked line 23 which is referenced in the error and it looks fine….I've tried taking out the filter, and everything works…without changing that line.
Please help
Edit
I've now changed my config to this
input {
syslog {
type => syslog
port => 5544
}
}
filter {
grok {
type => "syslog"
match => ["syslog_program","nginx-access"]
match => [ "message","%{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}" ]
add_field => [ "nginx_response", "%{NUMBER:response}" ]
}
}
output {
stdout {}
elasticsearch {
embedded => false
host => "localhost"
cluster => "cluster01"
}
email {
match => [ "status", "status,304"]
to => "test@test.com"
from => "test@test.com"
options => [ "smtpIporHost", "",
"port", "",
"userName", "",
"password", "",
"starttls", "",
"authenticationType", ""
]
via => "smtp" # or pop or sendmail
subject => "Found %{IP:client} Alert on %{@source_host}"
body => "Here is the event line %{@message}"
htmlbody => "<h2>%{matchName}</h2><br/><br/><h3>Full Event</h3><br/><br/><div align='center'>%{@message}</div>"
}
}
This seems to work, in as much as I can see that it's now recognising things in logstash, and that there is an email plugin command there, but the match fails…..any ideas?
Thanks
Best Answer
Do you not have to parse out
[program]
first? I don't think the 'input' field does any sort of filtering at all, so you might need to start with%SYSLOGBASE
http://logstash.net/docs/1.4.1/filters/grokYou could try instead perhaps:
Which'll keyword match your
message
field. That'll at least tell you if that is what's happening here.