I've had run a Galera Cluster for a while, MariaDB version 10.0.20, Galera version 25.3.10, everything over Gentoo. That cluster worked fine, nodes restarted several times for maintenance and so on. When I started mysql on any node, it loaded wsrep_provider library, which started SST and then eventually joined the cluster.
Now the time come to do massive upgrade. I have chosen a "rolling upgrade" path described in http://galeracluster.com/documentation-webpages/upgrading.html#id1 , take the chosen node down and update software there.
For impatient, a short summary of what follows: mariadb starts and ignores any wsrep_ config options, and if I specify them by hand in running instance, it seems to hang.
Two questions:
– Why it silently ignores my configuration options? How to make it respect them?
– What to do to start replication, how to join a node into cluster?
Everything upraded fine. Now I have MariaDB 10.1.13, Galera 25.3.15. Configuration in /etc/mysql/my.cnf was merged. It now looks like this (only mysqld section):
[mysqld]
character-set-server = utf8
user = mysql
port = 3306
socket = /var/run/mysqld/mysqld.sock
pid-file = /var/run/mysqld/mysqld.pid
log-error = /var/log/mysql/mysqld.err
basedir = /usr
datadir = /var/lib/mysql
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 4M
table_open_cache = 400
sort_buffer_size = 512K
net_buffer_length = 16K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
log-bin
server-id = 1
tmpdir = /tmp/
innodb_buffer_pool_size = 128M
innodb_data_file_path = ibdata1:10M:autoextend:max:128M
innodb_log_file_size = 48M
innodb_log_buffer_size = 8M
innodb_log_files_in_group=2
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
innodb_file_per_table
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
bind-address=0.0.0.0
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="www_cluster"
wsrep_cluster_address="gcomm://192.168.4.28,192.168.5.28,192.168.5.29"
wsrep_sst_method=rsync
I triple-checked this file, to be sure my wsrep_* options are in [mysqld] section as they were. These options are exactly like they were before upgrade.
However, mysql service is started now as a normal standalone server, not clustered:
2016-04-04 16:30:46 139950286550848 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-index where used; This may cause repliction to break when this server acts as a master and has its hostname changed! Please use '--log-basename=www1' or '--log-bin=mysqld-bin' to avoid this problem.
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: The InnoDB memory heap is disabled
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Memory barrier is not used
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Compressed tables use zlib 1.2.8
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using Linux native AIO
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using generic crc32 instructions
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Completed initialization of buffer pool
2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Highest supported file format is Barracuda.
2016-04-04 16:30:47 139950286550848 [Note] InnoDB: 128 rollback segment(s) are active.
2016-04-04 16:30:47 139950286550848 [Note] InnoDB: Waiting for purge to start
2016-04-04 16:30:47 139950286550848 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.28-76.1 started; log sequence number 116616176
2016-04-04 16:30:47 139949849761536 [Note] InnoDB: Dumping buffer pool(s) not yet started
2016-04-04 16:30:47 139950286550848 [Note] Plugin 'FEEDBACK' is disabled.
2016-04-04 16:30:47 139950286550848 [Note] Server socket created on IP: '0.0.0.0'.
2016-04-04 16:30:47 139950286550848 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.13-MariaDB' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distribution
If I query wsrep_* variables it shows no provider loaded:
MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%';
+--------------------------+----------------------+
| Variable_name | Value |
+--------------------------+----------------------+
| wsrep_cluster_conf_id | 18446744073709551615 |
| wsrep_cluster_size | 0 |
| wsrep_cluster_state_uuid | |
| wsrep_cluster_status | Disconnected |
| wsrep_connected | OFF |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_index | 18446744073709551615 |
| wsrep_provider_name | |
| wsrep_provider_vendor | |
| wsrep_provider_version | |
| wsrep_ready | OFF |
| wsrep_thread_count | 0 |
+--------------------------+----------------------+
12 rows in set (0.00 sec)
It looks like it didn't respect a wsrep_provider and wsrep_cluster_address (and I believe all other wsrep_*) options.
I tried to set them globally in running mariadb instance by hand:
MariaDB [(none)]> set global wsrep_provider='/usr/lib/galera/libgalera_smm.so';
Query OK, 0 rows affected (0.02 sec)
MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%';
+------------------------------+-----------------------------------+
| Variable_name | Value |
+------------------------------+-----------------------------------+
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 0.000000 |
| wsrep_causal_reads | 0 |
| wsrep_cert_deps_distance | 0.000000 |
| wsrep_cert_index_size | 0 |
| wsrep_cert_interval | 0.000000 |
| wsrep_cluster_conf_id | 18446744073709551615 |
| wsrep_cluster_size | 0 |
| wsrep_cluster_state_uuid | |
| wsrep_cluster_status | Disconnected |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 0.000000 |
| wsrep_connected | OFF |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_incoming_addresses | |
| wsrep_last_committed | 18446744073709551615 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cached_downto | 18446744073709551615 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_commits | 0 |
| wsrep_local_index | 18446744073709551615 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_recv_queue_max | 0 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_send_queue_max | 0 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_state | 0 |
| wsrep_local_state_comment | Initialized |
| wsrep_local_state_uuid | |
| wsrep_protocol_version | 18446744073709551615 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 3.15(r8459459) |
| wsrep_ready | OFF |
| wsrep_received | 0 |
| wsrep_received_bytes | 0 |
| wsrep_repl_data_bytes | 0 |
| wsrep_repl_keys | 0 |
| wsrep_repl_keys_bytes | 0 |
| wsrep_repl_other_bytes | 0 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
| wsrep_thread_count | 0 |
+------------------------------+-----------------------------------+
52 rows in set (0.00 sec)
respective log entries:
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Stop replication
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Provider was not loaded, in stop replication
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Initial position: 913d310a-9360-11e4-9d31-922a1e5d98fe:63026
2016-04-04 16:45:19 139950170823424 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2016-04-04 16:45:19 139950170823424 [Note] WSREP: wsrep_load(): Galera 3.15(r8459459) by Codership Oy <info@codership.com> loaded successfully.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.4.28; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false;
2016-04-04 16:45:19 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
and fill up cluster addresses:
MariaDB [(none)]> set global wsrep_cluster_address='gcomm://192.168.4.28,192.168.5.28,192.168.5.29';
Query OK, 0 rows affected (3.55 sec)
MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%';
+------------------------------+-------------------------------------------------------+
| Variable_name | Value |
+------------------------------+-------------------------------------------------------+
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 0.000000 |
| wsrep_causal_reads | 0 |
| wsrep_cert_deps_distance | 0.000000 |
| wsrep_cert_index_size | 0 |
| wsrep_cert_interval | 0.000000 |
| wsrep_cluster_conf_id | 471 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | 913d310a-9360-11e4-9d31-922a1e5d98fe |
| wsrep_cluster_status | Primary |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 0.000000 |
| wsrep_connected | ON |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0.0456547/0.049131/0.0560754/0.00491047/3 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_gcomm_uuid | fd416235-fa6b-11e5-867c-72011d51ce8d |
| wsrep_incoming_addresses | 192.168.5.28:3306,192.168.5.29:3306,192.168.4.28:3306 |
| wsrep_last_committed | 18446744073709551615 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cached_downto | 18446744073709551615 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_commits | 0 |
| wsrep_local_index | 2 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_recv_queue_max | 1 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_state | 1 |
| wsrep_local_state_comment | Joining: receiving State Transfer |
| wsrep_local_state_uuid | |
| wsrep_protocol_version | 7 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 3.15(r8459459) |
| wsrep_ready | OFF |
| wsrep_received | 1 |
| wsrep_received_bytes | 266 |
| wsrep_repl_data_bytes | 0 |
| wsrep_repl_keys | 0 |
| wsrep_repl_keys_bytes | 0 |
| wsrep_repl_other_bytes | 0 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
| wsrep_thread_count | 2 |
+------------------------------+-------------------------------------------------------+
57 rows in set (0.00 sec)
respective log:
2016-04-04 16:45:19 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2016-04-04 16:49:00 139950170520320 [Note] WSREP: Stop replication
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Start replication
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2016-04-04 16:49:02 139950170520320 [Note] WSREP: protonet asio version 0
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Using CRC-32C for message checksums.
2016-04-04 16:49:02 139950170520320 [Note] WSREP: backend: asio
2016-04-04 16:49:02 139950170520320 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2016-04-04 16:49:02 139950170520320 [Note] WSREP: restore pc from disk failed
2016-04-04 16:49:02 139950170520320 [Note] WSREP: GMCast version 0
2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2016-04-04 16:49:02 139950170520320 [Note] WSREP: EVS version 0
2016-04-04 16:49:02 139950170520320 [Note] WSREP: gcomm: connecting to group 'www_cluster', peer '192.168.4.28:,192.168.5.28:,192.168.5.29:'
2016-04-04 16:49:02 139950170520320 [Warning] WSREP: (fd416235, 'tcp://0.0.0.0:4567') address 'tcp://192.168.4.28:4567' points to own listening address, blacklisting
2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2016-04-04 16:49:02 139950170520320 [Note] WSREP: declaring 4c4f4779 at tcp://192.168.5.28:4567 stable
2016-04-04 16:49:02 139950170520320 [Note] WSREP: declaring b75760e5 at tcp://192.168.5.29:4567 stable
2016-04-04 16:49:02 139950170520320 [Note] WSREP: Node 4c4f4779 state prim
2016-04-04 16:49:02 139950170520320 [Note] WSREP: view(view_id(PRIM,4c4f4779,710) memb {
4c4f4779,0
b75760e5,0
fd416235,0
} joined {
} left {
} partitioned {
})
2016-04-04 16:49:02 139950170520320 [Note] WSREP: save pc into disk
2016-04-04 16:49:03 139950170520320 [Note] WSREP: gcomm: connected
2016-04-04 16:49:03 139950170520320 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2016-04-04 16:49:03 139950170520320 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2016-04-04 16:49:03 139950170520320 [Note] WSREP: Opened channel 'www_cluster'
2016-04-04 16:49:03 139948892088064 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: sent state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 0 (www2)
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 1 (galera)
2016-04-04 16:49:03 139949710771968 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.55477S), skipping check
2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 2 ()
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 470,
members = 2/3 (joined/total),
act_id = 63180,
last_appl. = -1,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 913d310a-9360-11e4-9d31-922a1e5d98fe
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Flow-control interval: [28, 28]
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 63180)
2016-04-04 16:49:03 139950170217216 [Note] WSREP: State transfer required:
Group state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180
Local state: 00000000-0000-0000-0000-000000000000:-1
2016-04-04 16:49:03 139950170217216 [Note] WSREP: New cluster view: global state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180, view# 471: Primary, number of nodes: 3, my index: 2, protocol version 3
2016-04-04 16:49:03 139950170217216 [Warning] WSREP: Gap in state sequence. Need state transfer.
2016-04-04 16:49:03 139948883695360 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.4.28' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '6067' --binlog 'mysqld-bin' '
2016-04-04 16:49:03 139950170217216 [Note] WSREP: Prepared SST request: rsync|192.168.4.28:4444/rsync_sst
2016-04-04 16:49:03 139950170217216 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2016-04-04 16:49:03 139950170217216 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-04-04 16:49:03 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:49:03 139950170217216 [Note] WSREP: Assign initial position for certification: 63180, protocol version: 3
2016-04-04 16:49:03 139949719164672 [Note] WSREP: Service thread queue flushed.
2016-04-04 16:49:03 139950170217216 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (913d310a-9360-11e4-9d31-922a1e5d98fe): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Member 2.0 () requested state transfer from '*any*'. Selected 0.0 (www2)(SYNCED) as donor.
2016-04-04 16:49:03 139948892088064 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 63180)
2016-04-04 16:49:03 139950170217216 [Note] WSREP: Requesting state transfer: success, donor: 0
2016-04-04 16:49:05 139949710771968 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') turning message relay requesting off
And now node keep sitting in "Joining: receiving State Transfer".
Selected donor node (www2), however, says it had completed state transfer:
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping
160404 16:49:04 [Note] WSREP: declaring b75760e5 at tcp://192.168.5.29:4567 stable
160404 16:49:04 [Note] WSREP: declaring fd416235 at tcp://192.168.4.28:4567 stable
160404 16:49:04 [Note] WSREP: Node 4c4f4779 state prim
160404 16:49:04 [Note] WSREP: view(view_id(PRIM,4c4f4779,710) memb {
4c4f4779,0
b75760e5,0
fd416235,0
} joined {
} left {
} partitioned {
})
160404 16:49:04 [Note] WSREP: save pc into disk
160404 16:49:04 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
160404 16:49:04 [Note] WSREP: STATE_EXCHANGE: sent state UUID: feddde16-fa6b-11e5-86f6-fb86e3a56ec8
160404 16:49:04 [Note] WSREP: STATE EXCHANGE: sent state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8
160404 16:49:04 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 0 (www2)
160404 16:49:04 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 1 (galera)
160404 16:49:05 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 2 ()
160404 16:49:05 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 470,
members = 2/3 (joined/total),
act_id = 63180,
last_appl. = 63152,
protocols = 0/7/3 (gcs/repl/appl),
group UUID = 913d310a-9360-11e4-9d31-922a1e5d98fe
160404 16:49:05 [Note] WSREP: Flow-control interval: [28, 28]
160404 16:49:05 [Note] WSREP: New cluster view: global state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180, view# 471: Primary, number of nodes: 3, my index: 0, protocol version 3
160404 16:49:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160404 16:49:05 [Note] WSREP: REPL Protocols: 7 (3, 2)
160404 16:49:05 [Note] WSREP: Service thread queue flushed.
160404 16:49:05 [Note] WSREP: Assign initial position for certification: 63180, protocol version: 3
160404 16:49:05 [Note] WSREP: Service thread queue flushed.
160404 16:49:05 [Note] WSREP: Member 2.0 () requested state transfer from '*any*'. Selected 0.0 (www2)(SYNCED) as donor.
160404 16:49:05 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 63180)
160404 16:49:06 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
160404 16:49:06 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.4.28:4444/rsync_sst' --auth '(null)' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --binlog 'mysqld-bin' --gtid '913d310a-9360-11e4-9d31-922a1e5d98fe:63180''
160404 16:49:06 [Note] WSREP: sst_donor_thread signaled with 0
160404 16:49:06 [Note] WSREP: Flushing tables for SST...
160404 16:49:06 [Note] WSREP: Provider paused at 913d310a-9360-11e4-9d31-922a1e5d98fe:63180 (9555)
160404 16:49:06 [Note] WSREP: Tables flushed.
WSREP_SST: [INFO] Preparing binlog files for transfer: (20160404 16:49:07.005)
mysqld-bin.000033
160404 16:49:07 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') turning message relay requesting off
160404 16:51:39 [Note] WSREP: resuming provider at 9555
160404 16:51:39 [Note] WSREP: Provider resumed.
160404 16:51:40 [Note] WSREP: 0.0 (www2): State transfer to 2.0 () complete.
160404 16:51:40 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 63180)
160404 16:51:40 [Note] WSREP: Member 0.0 (www2) synced with group.
160404 16:51:40 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 63180)
160404 16:51:40 [Note] WSREP: Synchronized with group, ready for connections
160404 16:51:40 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
What to do now? How do I continue update node operation?
Best Answer
As I found, new option appeared in a new version of MariaDB: wsrep-on. It wasn't needed before.
I added that option to config file, it ceased ignoring my wsrep settings. It started a server for a long time (about five minutes), in this period I seen rsync activity. After that, node became useable.