The AWS CLI now supports the --query
parameter which takes a JMESPath expressions.
This means you can sum the size values given by list-objects
using sum(Contents[].Size)
and count like length(Contents[])
.
This can be be run using the official AWS CLI as below and was introduced in Feb 2014
aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"
This is old, but I thought I would write this method which I use for low/medium traffic site (don't know if it will work well for heavy traffic site):
In Apache, I define a CustomLog format called graylog2_access
which formats the access log into a GELF format and then I send my log through Graylog2 by piping the log data through nc to send GELF messages to Graylog2's input.
Here is the custom format that it creates (human readable):
{
"version": "1.1",
"host": "%V",
"short_message": "%r",
"timestamp": %{%s}t,
"level": 6,
"_user_agent": "%{User-Agent}i",
"_source_ip": "%a",
"_duration_usec": %D,
"_duration_sec": %T,
"_request_size_byte": %O,
"_http_status": %s,
"_http_request_path": "%U",
"_http_request": "%U%q",
"_http_method": "%m",
"_http_referer": "%{Referer}i"
}
For the Apache config, here is a copy/paste version:
LogFormat "{ \"version\": \"1.1\", \"host\": \"%V\", \"short_message\": \"%r\", \"timestamp\": %{%s}t, \"level\": 6, \"_user_agent\": \"%{User-Agent}i\", \"_source_ip\": \"%a\", \"_duration_usec\": %D, \"_duration_sec\": %T, \"_request_size_byte\": %O, \"_http_status\": %s, \"_http_request_path\": \"%U\", \"_http_request\": \"%U%q\", \"_http_method\": \"%m\", \"_http_referer\": \"%{Referer}i\" }" graylog2_access
Then in your host configuration:
CustomLog "|nc -u graylogserver 12201" graylog2_access
Best Answer
Not having anything to configure would be a feature of cloud service...
You can try yourself by spinning instances of an ES domain and the official Graylog AMI, which is what I did.
The primary obstacle at the moment seems to be that AmazonES does not offer the TCP transport, and instead has only REST. It is not going to work without radical changes in Graylog's ES client.
A couple things you would need to line up between the two for this to theoretically work:
Elasticsearch itself has an EC2 discovery plugin, but it is not anticipated to be integrated into Graylog in the near term. Further, I don't think the EC2 features that discovery uses are exposed for the ES service, making it doubly unsupported.
Seems like you need to run your own cluster. At least there's graylog-ctl on the cloud images.