Statsd, Graphite and graphs

graphitestatsd

I've setup Graphite and statsd and both are running well. I'm using the example-client.py from graphite/examples to measure load values and it's OK.
I started doing tests with statsd and at first it seemed ok because it generated some graphs but now it doesn't look quite well.

First, this is my storage-schema.conf:

priority = 100
pattern = .*
retentions = 1m:395d

I'm using this command to send data to statsd:

echo 'ssh.invalid_users:1|c'| nc -w 1 -u localhost 8126

it executes, I click Update Graph in the Graphite web interface, it generates a line, hit again Update and the line disappears. Like this1 and this2
If I execute the previous command 5 times, the graph line will reach 2 and it will actually save it.
Again running the same command two times, graph line reaches 2 and disappears.
I can't find what I have misconfigured.

The intended use is this: 

tail -n 0 -f /var/log/auth.log|grep --line-buffered "Invalid user" | while read line; do echo "ssh.invalid_users:1|c" | nc -w 1 -u localhost 8126; done

EDIT:
On a fresh system I reinstalled using the latest versions of graphite, carbon, nodejs, statsd and it's acting the same.
While tail-ing /opt/graphite/storage/log/carbon-cache/carbon-cache-a/query.log I get:
cache query for "stats_counts.ssh.invalid_users" returned 0 values
cache query for "stats.ssh.invalid_users" returned 0 values
whenever I press update in webapp. I noticed that it will randomly say returned 1 values when drawing the lines, but will revert to returned 0 values and the lines disappear.

Best Answer

The problem is the storage-schema retention:
retentions = 1m:395d - which is taken from graphite wiki http://graphite.wikidot.com/installation

I had to use retentions = 10:2160,60:10080,600:262974 or something similar. This takes in consideration values saved every 10 seconds.

Also, although I restarted graphite after changing storage-schema.conf, I had to use a different metric name because the previous would retain the same behavior/retention (and I can reproduce this).
So instead of echo 'ssh.invalid_users:1|c', I had to use
echo 'ssh.invalid_userstest2:1|c'.

Related Topic