Centos – GlusterFS SSL not working

centoscentos7filesystemsglusterfshigh-availability

I have been trying for hours to get GlusterFS SSL workong on the management and I/O path to no avail, so I hope you can help me.

I have tried it the simple way:

openssl genrsa -out glusterfs.key 4096
openssl req -new -x509 -key glusterfs.key -subj "/CN=Anyone" -out glusterfs.pem

And then I concatenated the glusterfs.pem from all peers to glusterfs.ca and distributed glusterfs.ca to all nodes.
The following files exist:

/etc/ssl/glusterfs.key
/etc/ssl/glusterfs.pem
/etc/ssl/glusterfs.ca

I have created the file to enable SSL on the management path and restarted glusterd.

touch /var/lib/glusterd/secure-access
systemctl restart glusterd

My glusterd.log is now logging hundreds of entries (dozens per minute) showing an SSL error:

[2017-08-29 22:01:07.535725] E [socket.c:202:ssl_dump_error_stack] 0-socket.management:   error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
[2017-08-29 22:01:07.535760] E [socket.c:2519:socket_poller] 0-socket.management: server setup failed

The same error occurs in the volume log when I try to enable SSL on the I/O path and the same error also occurs when I try a more complex way of creating my glusterfs.* files. Example:

openssl genrsa 4096 > glusterfs.ca.key
openssl req -sha1 -new -x509 -nodes -days 3650 -key glusterfs.ca.key > glusterfs.ca
openssl req -sha1 -newkey rsa:4096 -days 1825 -nodes -keyout glusterfs.key > glusterfs-req.pem
openssl rsa -in glusterfs.key -out glusterfs.key
openssl x509 -sha1 -req -in glusterfs-req.pem -days 1825  -CA glusterfs.ca -CAkey glusterfs.ca.key -set_serial 01 > glusterfs.pem
openssl verify -CAfile glusterfs.ca glusterfs.pem

But this results in the same error as shown above, no matter what CN etc. I set and no matter whether I set the same or different CN on the peers.

I have also tried creating RSA keys with a key length of 2048 to no avail.

I am running out of ideas and I don't know what this error means in the context of GlusterFS:

SSL3_GET_RECORD:wrong version number

Disclaimer:
I am running CentOS 7 64-bit and GlusterFS 3.11.3 (latest version as of this writing).

Best Answer

After spending a few hours trying to figure out what is wrong with the certificates, I found that, in my case, the problem was a stupid mistake: I haven't restarted glusterd on one of the two servers that were hosting the bricks. Both servers were therefore configured exactly the same, but one of them was unaware of the new SSL configuration.

If you encounter the same issue, the first thing is to run service glusterd restart on all the nodes. If it doesn't work, the other answer, as well a bug report may help.