Why is Uchiwa not able to monitor the datacenter on CentOS7

centos7rabbitmqredissensuuchiwa

This post combined with Install Sensu using YUM was used in order to install, configure and start Sensu and related services:

sudo yum install -y erlang && \
sudo rpm -Uvh http://www.rabbitmq.com/releases/rabbitmq-server/v3.5.0/rabbitmq-server-3.5.0-1.noarch.rpm && \
sudo yum install -y redis && \
echo '[sensu]
name=sensu
baseurl=http://sensu.global.ssl.fastly.net/yum/$basearch/
gpgcheck=0
enabled=1' | sudo tee /etc/yum.repos.d/sensu.repo && \
sudo yum install -y sensu && \
sudo yum install -y uchiwa && \
for s in rabbitmq-server redis.service sensu-server sensu-api uchiwa; do sudo systemctl restart $s; done && \
sudo rabbitmqctl add_vhost /sensu && \
sudo rabbitmqctl add_user sensu secret && \
sudo rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

Analysis

  • Navigating to :3000 shows the uchiwa dashboard and indicates Connection error. Is the Sensu API running?
  • /var/log/sensu/sensu-api.log indicates
{"timestamp":"2016-07-03T22:58:58.532905+0000","level":"warn","message":"config file does not exist or is not readable","file":"/etc/sensu/config.json"}
{"timestamp":"2016-07-03T22:58:58.533069+0000","level":"warn","message":"ignoring config file","file":"/etc/sensu/config.json"}
{"timestamp":"2016-07-03T22:58:58.533137+0000","level":"warn","message":"loading config files from directory","directory":"/etc/sensu/conf.d"}
{"timestamp":"2016-07-03T22:58:58.712175+0000","level":"info","message":"api listening","protocol":"http","bind":"0.0.0.0","port":4567}
  • The config.json does not seems to exist (/etc/sensu/config.json: No such file or directory)
  • Downloading the example config.json and restarting the sensu-api did not solve the issue
{
  "rabbitmq": {
    "host": "localhost",
    "vhost": "/sensu",
    "user": "sensu",
    "password": "secret"
  },
  "redis": {
    "host": "localhost",
    "port": 6379,
    "password": "secret"
  }
}
  • the redis port is listening:
LISTEN     0      128    127.0.0.1:6379                     *:*
  • the rabbitmq port as well:
LISTEN     0      100          *:4567                     *:*
  • Uchiwa cannot connect to the datacenters
{
  "sensu": [
    {
      "name": "Site 1",
      "host": "localhost",
      "port": 4567,
      "timeout": 10
    },
    {
      "name": "Site 2",
      "host": "localhost",
      "port": 4567,
      "ssl": false,
      "path": "",
      "user": "",
      "pass": "",
      "timeout": 10
    }
  ],
  "uchiwa": {
    "host": "0.0.0.0",
    "port": 3000,
    "refresh": 10
  }
}

{"timestamp":"2016-07-03T23:34:32.990067621Z","level":"warn","message":"GET http://localhost:4567/stashes returned: 500 Internal Server Error"}
{"timestamp":"2016-07-03T23:34:32.990102095Z","level":"warn","message":"Connection failed to the datacenter Site 1"}
{"timestamp":"2016-07-03T23:34:32.990115588Z","level":"info","message":"Updating the datacenter Site 2"}
{"timestamp":"2016-07-03T23:34:32.991462585Z","level":"warn","message":"GET http://localhost:4567/stashes returned: 500 Internal Server Error"}
{"timestamp":"2016-07-03T23:34:32.991492978Z","level":"warn","message":"Connection failed to the datacenter Site 2"}
  • navigating to <IPADDRESS>:4567 results in:
{"error":"redis and transport connections not initialized"}
  • redis replies
sudo redis-cli ping
PONG
  • redis password was set, but the issue persists
sudo redis-cli
127.0.0.1:6379> auth secret
OK

Best Answer

Concise

The sensu-client seems to require a running rabbitmq-server. As this message broker did not seem to be running neither the sensu-client was able to start, nor the Uchiwa dashboard was able to monitor it.

Verbose

If the following error:

{"error":"redis and transport connections not initialized"}

is shown and redis is listening, please check that rabbitmq is running and check the rabbitmq port using rabbitmq-status:

sudo systemctl status rabbitmq-server

Example

Today the above error was shown and when the rabbitmq port was checked it appeared that a certain port was not listening:

{listeners,[{clustering,25672,"::"},{amqp,5672,"::"}},

As SSL was configured, port 5671 should be configured in /etc/rabbitmq.config. Once the messagebroker server was restarted the ssl port 5671 started to listen and the issue was solved:

{listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{'amqp/ssl',5671,"::"}]},

my rabbitmq config is :

[
{rabbit, [ 
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
               {certfile,"/etc/rabbitmq/ssl/cert.pem"},
               {keyfile,"/etc/rabbitmq/ssl/key.pem"},
               {verify,verify_peer},
               {fail_if_no_peer_cert,true}]}
 ]}
  ].
Related Topic