Mysql – MariaDB Galera SST Fails

galeramariadbMySQLmysql-clusterxtrabackup

I am running a 3 node MariaDB 10 Galera Cluster on CentOS 6. I have use this set-up before without issue else where but now when adding a 3rd node the SST fails with this error in the logs:

150212 18:03:02 [Warning] WSREP: 1.0 (server1.example.com): State transfer to 0.0 (server3.example.com) failed: -22 (Invalid argument)
150212 18:03:02 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.

Checking the logs of the donor node shows:

150212 18:03:01 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.0.3:4444/xtrabackup_sst' --auth 'wsrep_sst:wsrep_pass' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf'  --binlog 'mysql-bin' --gtid '8561e69a-9beb-11e4-825a-f2380deeda3b:18365'
150212 18:03:01 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.0.3:4444/xtrabackup_sst' --auth 'wsrep_sst:wsrep_pass' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf'  --binlog 'mysql-bin' --gtid '8561e69a-9beb-11e4-825a-f2380deeda3b:18365': 22 (Invalid argument)
150212 18:03:01 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.0.3:4444/xtrabackup_sst' --auth 'wsrep_sst:wsrep_pass' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf'  --binlog 'mysql-bin' --gtid '8561e69a-9beb-11e4-825a-f2380deeda3b:18365'

My config looks like so:

[server]
wsrep_provider="/usr/lib64/galera/libgalera_smm.so"
wsrep_cluster_address="gcomm://192.168.0.1,192.168.0.2,192.168.0.3"
wsrep_cluster_name='MY_CLUSTER'
wsrep_sst_auth=wsrep_sst:wsrep_pass
wsrep_sst_method=xtrabackup-v2
wsrep_sst_receive_address=192.168.0.3
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
innodb_file_per_table=1
query_cache_size=0
binlog_format=ROW
log-bin=mysql-bin

[mysqld]
datadir=/var/lib/mysql
tmpdir=/tmp
user=mysql
bind-address=0.0.0.0

Nodes only differ in their wsrep_sst_receive_address value.

Best Answer

Are you sure you are using the exact same version on all nodes? At some point there were version incompatible changes in SST methods, and those caused SST to fail between different versions.

There was a config option for wsrep_sst_method that allowed the two different versions to SST between themselves.

Related Topic