Couldn't join MariaDB Galera Cluster node back to cluster after software update. wsrep_* options in my.cnf seem to be ignored
5,344
As I found, new option appeared in a new version of MariaDB: wsrep-on. It wasn't needed before.
I added that option to config file, it ceased ignoring my wsrep settings. It started a server for a long time (about five minutes), in this period I seen rsync activity. After that, node became useable.
Related videos on Youtube
18 : 44
LibreNMS High Availability - MariaDB Galera Cluster
01 : 02 : 12
Webinar replay: Top Five Tips to Drive MariaDB Galera Cluster Performance for Nextcloud
55 : 03
Galera Manager is now GA with ability to launch clusters on premise and in the AWS cloud
20 : 22
How to install and configuration MariaDB Galera Cluster on CentOS
57 : 03
Adding a new Node to MariaDB Galera Cluster
Author by
Nikita Kipriyanov
I'm against any war! I never voted for Putin. I never supported the annexation of Crimea.
Updated on September 18, 2022Comments
-
Nikita Kipriyanov over 1 year
I've had run a Galera Cluster for a while, MariaDB version 10.0.20, Galera version 25.3.10, everything over Gentoo. That cluster worked fine, nodes restarted several times for maintenance and so on. When I started mysql on any node, it loaded wsrep_provider library, which started SST and then eventually joined the cluster.
Now the time come to do massive upgrade. I have chosen a "rolling upgrade" path described in http://galeracluster.com/documentation-webpages/upgrading.html#id1 , take the chosen node down and update software there.
For impatient, a short summary of what follows: mariadb starts and ignores any wsrep_ config options, and if I specify them by hand in running instance, it seems to hang.
Two questions: - Why it silently ignores my configuration options? How to make it respect them? - What to do to start replication, how to join a node into cluster?
Everything upraded fine. Now I have MariaDB 10.1.13, Galera 25.3.15. Configuration in /etc/mysql/my.cnf was merged. It now looks like this (only mysqld section):
[mysqld] character-set-server = utf8 user = mysql port = 3306 socket = /var/run/mysqld/mysqld.sock pid-file = /var/run/mysqld/mysqld.pid log-error = /var/log/mysql/mysqld.err basedir = /usr datadir = /var/lib/mysql skip-external-locking key_buffer_size = 16M max_allowed_packet = 4M table_open_cache = 400 sort_buffer_size = 512K net_buffer_length = 16K read_buffer_size = 256K read_rnd_buffer_size = 512K myisam_sort_buffer_size = 8M lc_messages_dir = /usr/share/mysql lc_messages = en_US log-bin server-id = 1 tmpdir = /tmp/ innodb_buffer_pool_size = 128M innodb_data_file_path = ibdata1:10M:autoextend:max:128M innodb_log_file_size = 48M innodb_log_buffer_size = 8M innodb_log_files_in_group=2 innodb_flush_log_at_trx_commit = 1 innodb_lock_wait_timeout = 50 innodb_file_per_table binlog_format=ROW default-storage-engine=innodb innodb_autoinc_lock_mode=2 query_cache_size=0 query_cache_type=0 bind-address=0.0.0.0 wsrep_provider=/usr/lib/galera/libgalera_smm.so wsrep_cluster_name="www_cluster" wsrep_cluster_address="gcomm://192.168.4.28,192.168.5.28,192.168.5.29" wsrep_sst_method=rsyncI triple-checked this file, to be sure my wsrep_* options are in [mysqld] section as they were. These options are exactly like they were before upgrade.
However, mysql service is started now as a normal standalone server, not clustered:
2016-04-04 16:30:46 139950286550848 [Warning] No argument was provided to --log-bin and neither --log-basename or --log-bin-index where used; This may cause repliction to break when this server acts as a master and has its hostname changed! Please use '--log-basename=www1' or '--log-bin=mysqld-bin' to avoid this problem. 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: The InnoDB memory heap is disabled 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Memory barrier is not used 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Compressed tables use zlib 1.2.8 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using Linux native AIO 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Using generic crc32 instructions 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Completed initialization of buffer pool 2016-04-04 16:30:46 139950286550848 [Note] InnoDB: Highest supported file format is Barracuda. 2016-04-04 16:30:47 139950286550848 [Note] InnoDB: 128 rollback segment(s) are active. 2016-04-04 16:30:47 139950286550848 [Note] InnoDB: Waiting for purge to start 2016-04-04 16:30:47 139950286550848 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.28-76.1 started; log sequence number 116616176 2016-04-04 16:30:47 139949849761536 [Note] InnoDB: Dumping buffer pool(s) not yet started 2016-04-04 16:30:47 139950286550848 [Note] Plugin 'FEEDBACK' is disabled. 2016-04-04 16:30:47 139950286550848 [Note] Server socket created on IP: '0.0.0.0'. 2016-04-04 16:30:47 139950286550848 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.1.13-MariaDB' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distributionIf I query wsrep_* variables it shows no provider loaded:
MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%'; +--------------------------+----------------------+ | Variable_name | Value | +--------------------------+----------------------+ | wsrep_cluster_conf_id | 18446744073709551615 | | wsrep_cluster_size | 0 | | wsrep_cluster_state_uuid | | | wsrep_cluster_status | Disconnected | | wsrep_connected | OFF | | wsrep_local_bf_aborts | 0 | | wsrep_local_index | 18446744073709551615 | | wsrep_provider_name | | | wsrep_provider_vendor | | | wsrep_provider_version | | | wsrep_ready | OFF | | wsrep_thread_count | 0 | +--------------------------+----------------------+ 12 rows in set (0.00 sec)It looks like it didn't respect a wsrep_provider and wsrep_cluster_address (and I believe all other wsrep_*) options.
I tried to set them globally in running mariadb instance by hand:
MariaDB [(none)]> set global wsrep_provider='/usr/lib/galera/libgalera_smm.so'; Query OK, 0 rows affected (0.02 sec) MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%'; +------------------------------+-----------------------------------+ | Variable_name | Value | +------------------------------+-----------------------------------+ | wsrep_apply_oooe | 0.000000 | | wsrep_apply_oool | 0.000000 | | wsrep_apply_window | 0.000000 | | wsrep_causal_reads | 0 | | wsrep_cert_deps_distance | 0.000000 | | wsrep_cert_index_size | 0 | | wsrep_cert_interval | 0.000000 | | wsrep_cluster_conf_id | 18446744073709551615 | | wsrep_cluster_size | 0 | | wsrep_cluster_state_uuid | | | wsrep_cluster_status | Disconnected | | wsrep_commit_oooe | 0.000000 | | wsrep_commit_oool | 0.000000 | | wsrep_commit_window | 0.000000 | | wsrep_connected | OFF | | wsrep_flow_control_paused | 0.000000 | | wsrep_flow_control_paused_ns | 0 | | wsrep_flow_control_recv | 0 | | wsrep_flow_control_sent | 0 | | wsrep_incoming_addresses | | | wsrep_last_committed | 18446744073709551615 | | wsrep_local_bf_aborts | 0 | | wsrep_local_cached_downto | 18446744073709551615 | | wsrep_local_cert_failures | 0 | | wsrep_local_commits | 0 | | wsrep_local_index | 18446744073709551615 | | wsrep_local_recv_queue | 0 | | wsrep_local_recv_queue_avg | 0.000000 | | wsrep_local_recv_queue_max | 0 | | wsrep_local_recv_queue_min | 0 | | wsrep_local_replays | 0 | | wsrep_local_send_queue | 0 | | wsrep_local_send_queue_avg | 0.000000 | | wsrep_local_send_queue_max | 0 | | wsrep_local_send_queue_min | 0 | | wsrep_local_state | 0 | | wsrep_local_state_comment | Initialized | | wsrep_local_state_uuid | | | wsrep_protocol_version | 18446744073709551615 | | wsrep_provider_name | Galera | | wsrep_provider_vendor | Codership Oy <[email protected]> | | wsrep_provider_version | 3.15(r8459459) | | wsrep_ready | OFF | | wsrep_received | 0 | | wsrep_received_bytes | 0 | | wsrep_repl_data_bytes | 0 | | wsrep_repl_keys | 0 | | wsrep_repl_keys_bytes | 0 | | wsrep_repl_other_bytes | 0 | | wsrep_replicated | 0 | | wsrep_replicated_bytes | 0 | | wsrep_thread_count | 0 | +------------------------------+-----------------------------------+ 52 rows in set (0.00 sec)respective log entries:
2016-04-04 16:45:19 139950170823424 [Note] WSREP: Stop replication 2016-04-04 16:45:19 139950170823424 [Note] WSREP: Provider was not loaded, in stop replication 2016-04-04 16:45:19 139950170823424 [Note] WSREP: Initial position: 913d310a-9360-11e4-9d31-922a1e5d98fe:63026 2016-04-04 16:45:19 139950170823424 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' 2016-04-04 16:45:19 139950170823424 [Note] WSREP: wsrep_load(): Galera 3.15(r8459459) by Codership Oy <[email protected]> loaded successfully. 2016-04-04 16:45:19 139950170823424 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm. 2016-04-04 16:45:19 139950170823424 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 2016-04-04 16:45:19 139950170823424 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.4.28; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; 2016-04-04 16:45:19 139949719164672 [Note] WSREP: Service thread queue flushed. 2016-04-04 16:45:19 139950170823424 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1and fill up cluster addresses:
MariaDB [(none)]> set global wsrep_cluster_address='gcomm://192.168.4.28,192.168.5.28,192.168.5.29'; Query OK, 0 rows affected (3.55 sec) MariaDB [(none)]> SHOW STATUS LIKE 'wsrep%'; +------------------------------+-------------------------------------------------------+ | Variable_name | Value | +------------------------------+-------------------------------------------------------+ | wsrep_apply_oooe | 0.000000 | | wsrep_apply_oool | 0.000000 | | wsrep_apply_window | 0.000000 | | wsrep_causal_reads | 0 | | wsrep_cert_deps_distance | 0.000000 | | wsrep_cert_index_size | 0 | | wsrep_cert_interval | 0.000000 | | wsrep_cluster_conf_id | 471 | | wsrep_cluster_size | 3 | | wsrep_cluster_state_uuid | 913d310a-9360-11e4-9d31-922a1e5d98fe | | wsrep_cluster_status | Primary | | wsrep_commit_oooe | 0.000000 | | wsrep_commit_oool | 0.000000 | | wsrep_commit_window | 0.000000 | | wsrep_connected | ON | | wsrep_evs_delayed | | | wsrep_evs_evict_list | | | wsrep_evs_repl_latency | 0.0456547/0.049131/0.0560754/0.00491047/3 | | wsrep_evs_state | OPERATIONAL | | wsrep_flow_control_paused | 0.000000 | | wsrep_flow_control_paused_ns | 0 | | wsrep_flow_control_recv | 0 | | wsrep_flow_control_sent | 0 | | wsrep_gcomm_uuid | fd416235-fa6b-11e5-867c-72011d51ce8d | | wsrep_incoming_addresses | 192.168.5.28:3306,192.168.5.29:3306,192.168.4.28:3306 | | wsrep_last_committed | 18446744073709551615 | | wsrep_local_bf_aborts | 0 | | wsrep_local_cached_downto | 18446744073709551615 | | wsrep_local_cert_failures | 0 | | wsrep_local_commits | 0 | | wsrep_local_index | 2 | | wsrep_local_recv_queue | 0 | | wsrep_local_recv_queue_avg | 0.000000 | | wsrep_local_recv_queue_max | 1 | | wsrep_local_recv_queue_min | 0 | | wsrep_local_replays | 0 | | wsrep_local_send_queue | 0 | | wsrep_local_send_queue_avg | 0.000000 | | wsrep_local_send_queue_max | 1 | | wsrep_local_send_queue_min | 0 | | wsrep_local_state | 1 | | wsrep_local_state_comment | Joining: receiving State Transfer | | wsrep_local_state_uuid | | | wsrep_protocol_version | 7 | | wsrep_provider_name | Galera | | wsrep_provider_vendor | Codership Oy <[email protected]> | | wsrep_provider_version | 3.15(r8459459) | | wsrep_ready | OFF | | wsrep_received | 1 | | wsrep_received_bytes | 266 | | wsrep_repl_data_bytes | 0 | | wsrep_repl_keys | 0 | | wsrep_repl_keys_bytes | 0 | | wsrep_repl_other_bytes | 0 | | wsrep_replicated | 0 | | wsrep_replicated_bytes | 0 | | wsrep_thread_count | 2 | +------------------------------+-------------------------------------------------------+ 57 rows in set (0.00 sec)respective log:
2016-04-04 16:45:19 139949719164672 [Note] WSREP: Service thread queue flushed. 2016-04-04 16:45:19 139950170823424 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 2016-04-04 16:49:00 139950170520320 [Note] WSREP: Stop replication 2016-04-04 16:49:02 139950170520320 [Note] WSREP: Start replication 2016-04-04 16:49:02 139950170520320 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 2016-04-04 16:49:02 139950170520320 [Note] WSREP: protonet asio version 0 2016-04-04 16:49:02 139950170520320 [Note] WSREP: Using CRC-32C for message checksums. 2016-04-04 16:49:02 139950170520320 [Note] WSREP: backend: asio 2016-04-04 16:49:02 139950170520320 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) 2016-04-04 16:49:02 139950170520320 [Note] WSREP: restore pc from disk failed 2016-04-04 16:49:02 139950170520320 [Note] WSREP: GMCast version 0 2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2016-04-04 16:49:02 139950170520320 [Note] WSREP: EVS version 0 2016-04-04 16:49:02 139950170520320 [Note] WSREP: gcomm: connecting to group 'www_cluster', peer '192.168.4.28:,192.168.5.28:,192.168.5.29:' 2016-04-04 16:49:02 139950170520320 [Warning] WSREP: (fd416235, 'tcp://0.0.0.0:4567') address 'tcp://192.168.4.28:4567' points to own listening address, blacklisting 2016-04-04 16:49:02 139950170520320 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2016-04-04 16:49:02 139950170520320 [Note] WSREP: declaring 4c4f4779 at tcp://192.168.5.28:4567 stable 2016-04-04 16:49:02 139950170520320 [Note] WSREP: declaring b75760e5 at tcp://192.168.5.29:4567 stable 2016-04-04 16:49:02 139950170520320 [Note] WSREP: Node 4c4f4779 state prim 2016-04-04 16:49:02 139950170520320 [Note] WSREP: view(view_id(PRIM,4c4f4779,710) memb { 4c4f4779,0 b75760e5,0 fd416235,0 } joined { } left { } partitioned { }) 2016-04-04 16:49:02 139950170520320 [Note] WSREP: save pc into disk 2016-04-04 16:49:03 139950170520320 [Note] WSREP: gcomm: connected 2016-04-04 16:49:03 139950170520320 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2016-04-04 16:49:03 139950170520320 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2016-04-04 16:49:03 139950170520320 [Note] WSREP: Opened channel 'www_cluster' 2016-04-04 16:49:03 139948892088064 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3 2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: sent state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 0 (www2) 2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 1 (galera) 2016-04-04 16:49:03 139949710771968 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.55477S), skipping check 2016-04-04 16:49:03 139948892088064 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 2 () 2016-04-04 16:49:03 139948892088064 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf_id = 470, members = 2/3 (joined/total), act_id = 63180, last_appl. = -1, protocols = 0/7/3 (gcs/repl/appl), group UUID = 913d310a-9360-11e4-9d31-922a1e5d98fe 2016-04-04 16:49:03 139948892088064 [Note] WSREP: Flow-control interval: [28, 28] 2016-04-04 16:49:03 139948892088064 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 63180) 2016-04-04 16:49:03 139950170217216 [Note] WSREP: State transfer required: Group state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180 Local state: 00000000-0000-0000-0000-000000000000:-1 2016-04-04 16:49:03 139950170217216 [Note] WSREP: New cluster view: global state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180, view# 471: Primary, number of nodes: 3, my index: 2, protocol version 3 2016-04-04 16:49:03 139950170217216 [Warning] WSREP: Gap in state sequence. Need state transfer. 2016-04-04 16:49:03 139948883695360 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.4.28' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '6067' --binlog 'mysqld-bin' ' 2016-04-04 16:49:03 139950170217216 [Note] WSREP: Prepared SST request: rsync|192.168.4.28:4444/rsync_sst 2016-04-04 16:49:03 139950170217216 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2016-04-04 16:49:03 139950170217216 [Note] WSREP: REPL Protocols: 7 (3, 2) 2016-04-04 16:49:03 139949719164672 [Note] WSREP: Service thread queue flushed. 2016-04-04 16:49:03 139950170217216 [Note] WSREP: Assign initial position for certification: 63180, protocol version: 3 2016-04-04 16:49:03 139949719164672 [Note] WSREP: Service thread queue flushed. 2016-04-04 16:49:03 139950170217216 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (913d310a-9360-11e4-9d31-922a1e5d98fe): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable. 2016-04-04 16:49:03 139948892088064 [Note] WSREP: Member 2.0 () requested state transfer from '*any*'. Selected 0.0 (www2)(SYNCED) as donor. 2016-04-04 16:49:03 139948892088064 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 63180) 2016-04-04 16:49:03 139950170217216 [Note] WSREP: Requesting state transfer: success, donor: 0 2016-04-04 16:49:05 139949710771968 [Note] WSREP: (fd416235, 'tcp://0.0.0.0:4567') turning message relay requesting offAnd now node keep sitting in "Joining: receiving State Transfer".
Selected donor node (www2), however, says it had completed state transfer:
160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping 160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping 160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping 160404 16:49:04 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') address 'tcp://192.168.5.28:4567' pointing to uuid 4c4f4779 is blacklisted, skipping 160404 16:49:04 [Note] WSREP: declaring b75760e5 at tcp://192.168.5.29:4567 stable 160404 16:49:04 [Note] WSREP: declaring fd416235 at tcp://192.168.4.28:4567 stable 160404 16:49:04 [Note] WSREP: Node 4c4f4779 state prim 160404 16:49:04 [Note] WSREP: view(view_id(PRIM,4c4f4779,710) memb { 4c4f4779,0 b75760e5,0 fd416235,0 } joined { } left { } partitioned { }) 160404 16:49:04 [Note] WSREP: save pc into disk 160404 16:49:04 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 160404 16:49:04 [Note] WSREP: STATE_EXCHANGE: sent state UUID: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 160404 16:49:04 [Note] WSREP: STATE EXCHANGE: sent state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 160404 16:49:04 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 0 (www2) 160404 16:49:04 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 1 (galera) 160404 16:49:05 [Note] WSREP: STATE EXCHANGE: got state msg: feddde16-fa6b-11e5-86f6-fb86e3a56ec8 from 2 () 160404 16:49:05 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf_id = 470, members = 2/3 (joined/total), act_id = 63180, last_appl. = 63152, protocols = 0/7/3 (gcs/repl/appl), group UUID = 913d310a-9360-11e4-9d31-922a1e5d98fe 160404 16:49:05 [Note] WSREP: Flow-control interval: [28, 28] 160404 16:49:05 [Note] WSREP: New cluster view: global state: 913d310a-9360-11e4-9d31-922a1e5d98fe:63180, view# 471: Primary, number of nodes: 3, my index: 0, protocol version 3 160404 16:49:05 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 160404 16:49:05 [Note] WSREP: REPL Protocols: 7 (3, 2) 160404 16:49:05 [Note] WSREP: Service thread queue flushed. 160404 16:49:05 [Note] WSREP: Assign initial position for certification: 63180, protocol version: 3 160404 16:49:05 [Note] WSREP: Service thread queue flushed. 160404 16:49:05 [Note] WSREP: Member 2.0 () requested state transfer from '*any*'. Selected 0.0 (www2)(SYNCED) as donor. 160404 16:49:05 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 63180) 160404 16:49:06 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 160404 16:49:06 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.4.28:4444/rsync_sst' --auth '(null)' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --binlog 'mysqld-bin' --gtid '913d310a-9360-11e4-9d31-922a1e5d98fe:63180'' 160404 16:49:06 [Note] WSREP: sst_donor_thread signaled with 0 160404 16:49:06 [Note] WSREP: Flushing tables for SST... 160404 16:49:06 [Note] WSREP: Provider paused at 913d310a-9360-11e4-9d31-922a1e5d98fe:63180 (9555) 160404 16:49:06 [Note] WSREP: Tables flushed. WSREP_SST: [INFO] Preparing binlog files for transfer: (20160404 16:49:07.005) mysqld-bin.000033 160404 16:49:07 [Note] WSREP: (4c4f4779, 'tcp://0.0.0.0:4567') turning message relay requesting off 160404 16:51:39 [Note] WSREP: resuming provider at 9555 160404 16:51:39 [Note] WSREP: Provider resumed. 160404 16:51:40 [Note] WSREP: 0.0 (www2): State transfer to 2.0 () complete. 160404 16:51:40 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 63180) 160404 16:51:40 [Note] WSREP: Member 0.0 (www2) synced with group. 160404 16:51:40 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 63180) 160404 16:51:40 [Note] WSREP: Synchronized with group, ready for connections 160404 16:51:40 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.What to do now? How do I continue update node operation?