Elasticsearch nodes not joining cluster

elasticsearch

i have three nodes which should autoconfig into a cluster but for some reason
they dont seem to want to cluster up.

The 3 Hyper-V VM Servers are CENTOS 7 installed as compute with just the tool packages installed. the servers have consecutive IP's in the same subnet there is no firewall enabled – they can all ping each other both by DNS and IP. Ports on all servers seem to be open.

[root@manelasticshard01 ~]# netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 10.1.247.246:9200       0.0.0.0:*               LISTEN      993        26721      3913/java
tcp        0      0 10.1.247.246:9300       0.0.0.0:*               LISTEN      993        23402      3913/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          20731      1549/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          19136      2441/master
tcp6       0      0 :::22                   :::*                    LISTEN      0          20733      1549/sshd
udp        0      0 127.0.0.1:323           0.0.0.0:*                           994        16999      913/chronyd
udp6       0      0 ::1:323                 :::*                                994        17000      913/chronyd

same output for all aside for IP addresses

NODE 3

 "cluster_name" : "PACMAN",
  "nodes" : {
    "NlrIawYrR1aq79P6F_Q3zA" : {
      "name" : "node3",
      "transport_address" : "manelasticshard03/10.1.247.244:9300",
      "host" : "10.1.247.244",
      "ip" : "10.1.247.244",
      "version" : "2.2.0",
      "build" : "8ff36d1",
      "http_address" : "manelasticshard03/10.1.247.244:9200",
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 3968,
        "mlockall" : false
      }
    }
  }
}

NODE 2

"cluster_name" : "PACMAN",
  "nodes" : {
    "9Sv2k73BTFSgxMDRDdpRgA" : {
      "name" : "node2",
      "transport_address" : "manelasticshard02/10.1.247.245:9300",
      "host" : "10.1.247.245",
      "ip" : "10.1.247.245",
      "version" : "2.2.0",
      "build" : "8ff36d1",
      "http_address" : "manelasticshard02/10.1.247.245:9200",
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 3898,
        "mlockall" : false
      }
    }
  }
}

NODE 1

   "cluster_name" : "PACMAN",
      "nodes" : {
        "RborUe8CS_C7ynX_yFWJ6Q" : {
          "name" : "node1",
          "transport_address" : "manelasticshard01/10.1.247.246:9300",
          "host" : "10.1.247.246",
          "ip" : "10.1.247.246",
          "version" : "2.2.0",
          "build" : "8ff36d1",
          "http_address" : "manelasticshard01/10.1.247.246:9200",
          "process" : {
            "refresh_interval_in_millis" : 1000,
            "id" : 3913,
            "mlockall" : false
          }
        }
      }
    }

my config is currently as follows but i have pretty much been through every setting i can find.

cluster.name: PACMAN
node.name: "node3"
# node.master: false
# node.data: true
network.host: manelasticshard03
# http.port: 9230
discovery.zen.ping.unicast.hosts: [manelasticshard01, manelasticshard02]
discovery.zen.ping.multicast.enabled: false
# discovery.zen.ping.unicast.enabled: true
# discovery.zen.fd.ping_timeout: 30s
# gateway.expected_nodes: 3
# gateway.recover_after_time: 5m
# gateway.recover_after_nodes: 2
# discovery.zen.minimum_master_nodes: 2

Does anyone have any ideas as to why the nodes are not clustering, or of any other tests i can do to find out.

UPDATE – if i set discovery minimum master nodes to two in order to force the cluster to search for a second node i get the following output.

[root@manelasticshard01 ~]# curl http://manelasticshard01:9200/_cluster/health?pretty=true
{
  "error" : {
    "root_cause" : [ {
      "type" : "master_not_discovered_exception",
      "reason" : null
    } ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

Best Answer

Even though i had explicitly allowed the port range elasticsearch was using apparently RHEL/CENTOS firewall does not like elasticsearch.

Turning it off completely resolved the issue.

FAIL>

Related Topic