Best practice of adding node(s) to elasticsearch cluster

Even though i had explicitly allowed the port range elasticsearch was using apparently RHEL/CENTOS firewall does not like elasticsearch.

Turning it off completely resolved the issue.

FAIL>

We use HAproxy to balance between two redundant clusters. During normal operation each receives ~50% of traffic; each is provisioned to take 100% when necessary.

We experienced a fault recently based on a failure case we had not planned for: all client and master nodes stayed up, so our cluster was responsive to REST–but all data nodes were temporarily offline, all indices appeared red and empty, and queries against them returned 0 results. But with a 200, following REST convention.

Our simple HAproxy health check failed us in this case; it merely checked for 200s.

I am now investigating use of http-check expect ! string red with a URI that targets the index of interest directly. I haven't used the more advanced http-check features before.

A more expensive check, but, should correctly take the client nodes for a lobotomized cluster out of the pool.

UPDATE (2): I have switched us over to using

option httpchk get /_cat/indices/<index of interest>
http-check expect rstring \b(green|yellow)\b

and it indeed seems like a better test.

(Second revision: using explicit check for green or yellow instead of just not-red, belatedly thought about index entirely missing from _cat fiter..._