Nginx – ELK stack: Can’t load Kibana 4

elasticsearchkibanalogstashnginxubuntu-14.04

I have installed Elasticsearch, Logstash, Kibana and Nginx following this guide, exept I used the latest version of everything.

https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04

The problem is that when accessing the webserver it says:

Kibana is loading. Give me a moment here. I'm loading a whole bunch of
code. Don't worry, all this good stuff will be cached up for next
time!

But it never loads the Kibana start page.

When I'm trying http://server-ip:5601 I get a

ERR_CONNECTION_REFUSED

But it seems to load with curl:

elk-stack:/opt/kibana/bin$ curl -i 0:5601
HTTP/1.1 200 OK
X-App-Name: kibana
Accept-Ranges: bytes
Cache-Control: public, max-age=0
Last-Modified: Tue, 21 Jul 2015 09:46:14 GMT
ETag: W/"6f9-14eb001a270"
Content-Type: text/html; charset=UTF-8
Content-Length: 1785
Vary: Accept-Encoding
Date: Wed, 22 Jul 2015 16:07:09 GMT
Connection: keep-alive

<!DOCTYPE html>
  <!--[if IE 8]>         <html class="no-js lt-ie9" lang="en"> <![endif]-->
  <!--[if gt IE 8]><!--> <html class="no-js" lang="en"> <!--<![endif]-->
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
    <meta name="viewport" content="width=device-width">
    <link rel="shortcut icon" href="styles/theme/elk.ico">
    <title>Kibana 4</title>
    <link rel="stylesheet" href="styles/main.css?_b=7489">

  </head>
  <body kibana ng-class="'application-' + activeApp.id">

    <div class="col-md-offset-4 col-md-4 page-header initial-load">
      <center>
        <img width="128" alt="Loading Kibana" src="images/initial_load.gif">
        <h1>
          <strong>Kibana</strong>
          <small id="cache-message">is loading. Give me a moment here. I'm loading a whole bunch of code. Don't worry, all this good stuff will be cached up for next time!</small>
        </h1>
      </center>
    </div>

    <script>
      window.KIBANA_VERSION='4.1.1';
      window.KIBANA_BUILD_NUM='7489';
      window.KIBANA_COMMIT_SHA='bd4fcb654a2fa062c9a99853ac5f776965ab7637';
    </script>

    <script src="bower_components/requirejs/require.js?_b=7489"></script>
    <script src="require.config.js?_b=7489"></script>
    <script>
      var showCacheMessage = location.href.indexOf('?embed') < 0 && location.href.indexOf('&embed') < 0;
      if (!showCacheMessage) document.getElementById('cache-message').style.display = 'none';

      if (window.KIBANA_BUILD_NUM.substr(0, 2) !== '@@') {
        // only cache bust if this is really the build number
        require.config({ urlArgs: '_b=' + window.KIBANA_BUILD_NUM });
      }

      require(['kibana'], function (kibana) { kibana.init(); });
    </script>

  </body>
</html>

Elasticsearch seems to work.

curl http://localhost:9200
{
  "status" : 200,
  "name" : "Martinex",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.0",
    "build_hash" : "929b9739cae115e73c346cb5f9a6f24ba735a743",
    "build_timestamp" : "2015-07-16T14:31:07Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Kibana runs as a service (se the script in the guide). If i stop it and start Kibana manualy i get:

elk-stack:/opt/kibana/bin$ sudo ./kibana
{"name":"Kibana","hostname":"elk-stack","pid":1414,"level":30,"msg":"Found kibana index","time":"2015-07-22T16:10:31.805Z","v":0}
{"name":"Kibana","hostname":"elk-stack","pid":1414,"level":30,"msg":"Listening on 127.0.0.1:5601","time":"2015-07-22T16:10:31.823Z","v":0}
{"name":"Kibana","hostname":"elk-stack","pid":1414,"level":30,"req":{"method":"GET","url":"/","headers":{"connection":"upgrade","host":"X.X.X.X","accept":"text/html, application/xhtml+xml, */*","accept-language":"sv-SE,sv;q=0.8,en-GB;q=0.5,en;q=0.3","user-agent":"Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko","accept-encoding":"gzip, deflate","if-modified-since":"Tue, 21 Jul 2015 09:46:14 GMT","if-none-match":"W/\"6f9-14eb001a270\"","authorization":"Basic YXRlYW9wOncwcms0bWU="},"remoteAddress":"127.0.0.1","remotePort":40809},"res":{"statusCode":304,"responseTime":8,"contentLength":0},"msg":"GET / 304 - 8ms","time":"2015-07-22T16:11:04.362Z","v":0}

I'm not realy sure if the Ngnix config is correct, it looks like this now:

server {
    listen 80 default_server;
    listen [::]:80 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;

            server_name elk-stack;

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;

    location / {
            proxy_pass http://localhost:5601;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;

            try_files $uri $uri/ =404;

Any ideas? I guess there is some logs somewhere, but I'm not sure which one to check.
More info of the installation:

Ubuntu 14.04.2
Kibana 4.1.1
Elasticsearch 1.7.0

EDIT: GregL ask for the netstat output:

    $ netstat -lnp
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      -
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
udp6       0      0 :::54328                :::*                                -
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     9748     -                   /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     7546     -                   @/com/ubuntu/upstart
unix  2      [ ACC ]     SEQPACKET  LISTENING     1667     -                   /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     7921     -                   /var/run/dbus/system_bus_socket

Here is the kibana.yml config:

port: 5601

host: "localhost"

elasticsearch_url: "http://localhost:9200"

elasticsearch_preserve_host: true

kibana_index: ".kibana"    

default_app_id: "discover"

request_timeout: 300000

shard_timeout: 0

verify_ssl: true

bundled_plugin_ids:
 - plugins/dashboard/index
 - plugins/discover/index
 - plugins/doc/index
 - plugins/kibana/index
 - plugins/markdown_vis/index
 - plugins/metric_vis/index
 - plugins/settings/index
 - plugins/table_vis/index
 - plugins/vis_types/index
 - plugins/visualize/index

In the elasticsearch.yml it is bound to localhost:

network.host: localhost

EDIT2: Netstat after all the changes.

    $ netstat -lnp
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:9300          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:9200          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
udp        0      0 0.0.0.0:54328           0.0.0.0:*                           -
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     8764     -                   /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     8013     -                   /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     1417     -                   @/com/ubuntu/upstart
unix  2      [ ACC ]     SEQPACKET  LISTENING     7856     -                   /run/udev/control

Best Answer

I have a feeling your problems are with the Nginx configuration.

Try commenting out host: "localhost" in kibana.yml, restarting the Kibana service and then trying server-ip:5601 again.

If that works, then you'll need to resolve the Nginx portion.

Changing the line proxy_pass http://localhost:5601 to proxy_pass http://<serverip>:5601 should fix the error about being unable to connect to the upstream server.