Sensu API not responding. Uchiwa: Connection error. Is the Sensu API running? Redis Issue

centos7monitoringrabbitmqredissensu

I have been setting up sensu on CentOS7.2 for the first time following the official docs over and over however I have not been able to get it to work:

I cannot seem to connect to the API. Uchiwa gives the error:

ALERT
Datacenter site1 returned:
Connection error. Is the Sensu API running?`

I have tried: curl -I http://localhost:4567/clients
…I do not get a response.

Here is /var/log/sensu/sensu-api.log
http://pastebin.com/wHEHE0bH

I have been creating a script to make the setup repeatable. Please see my script below which shows my config: http://pastebin.com/QEt5Msku
If you run the script on CentOS7 it should repeat this issue.

Fix:

Issue in answer below. After following a non official guide that successfully installed sensu on a fresh VM, I compared the two setup steps and after a couple of builds on fresh VMs swapping out the steps bit by bit I figured out replacing the repo with one from a non official guide that pointed to a slightly different URL fixed my problem (repo for CentOS6 but it works on 7, it just installs a non current verson 0.20.3).

echo '[sensu]
name=sensu-main
baseurl=http://repos.sensuapp.org/yum/el/6/x86_64/
gpgcheck=0
enabled=1' > /etc/yum.repos.d/sensu.repo

Best Answer

The problem is that sensu 0.23 resolves localhost to an ipv6 address but redis is not listening to the ipv6 address. The solution is to change "host": "localhost" to "host": "127.0.0.1". There is an issue on Sensus git page

For your information, you should have looked in the sesu-server logs too and posted them here.

Related Solutions

Freebsd – Redis connection issue

We reduced the error rate by 90% with the following redis command:

CONFIG SET save ""

This disables BGSAVE, which regulary stores all database changes on disk. The reason for the connect errors most likely come from a blocking fork() operation of the main redis process to start the BGSAVE process.

The redis.conf says:

# Redis may block too long on the fsync() call. Note that there is no fix for
# this currently, as even performing fsync in a different thread will block
# our synchronous write(2) call.

Also see how the mechanism is implemented with a simple fork() here. We think about using a dedicated redis server from our pool which will be responsible for the BGSAVE operations and just using the others for reading/writing.

From IRC chat, it seems like a couple of other companies ran into the same error. Bump was using a master/slave system as well. The slave does not accept connections and is only there to persist the data (see the discussion on hackernews here)

Hulu says the following: "To keep performance consistent on the shards, we disabled the writing to disk across all the shards, and we have a cron job that runs at 4am everyday doing a rolling “BGSAVE” command on each individual instance." (see here)

Edit:

It turns out that this was just a temporary fix. Load increased and we are back at the high error rates. Nevertheless I'm quite confident that a background operation (e.g. a fork, or a short-running background process) is causing the errors as the error messages always appear in blocks.

Edit2:

Since Redis is single-threaded, always keep an eye on long-running operations because they block everything else. An example is the keys * command. Avoid it and use scan instead

Uchiwa throwing “Error with Sensu API Sensu”

Another possibility is the location of your uchiwa.json file.

I had a similar issue and didn't know what was going on but it seemed to have issues when I had the uchiwa.json file created in /etc/sensu/conf.d but at the same time the default configuration created an uchiwa.json file in /etc/sensu. For some reason uchiwa wants the file to be in /etc/sensu and for now I am happy with that configuration as it finally has the api working correctly. Good luck to you.

Related Topic