专职DBA-MySQL高可用集群复制方案-MGR

基本环境：
------------------------------------------------------------
主机名  ip地址(bond0)  ip地址(bond1)  端口号   数据目录
------------------------------------------------------------
mgr01  10.0.0.31     192.168.10.31  3306   /data/mysql/3306/
mgr02  10.0.0.32     192.168.10.32  3306   /data/mysql/3306/
mgr03  10.0.0.33     192.168.10.33  3306   /data/mysql/3306/
------------------------------------------------------------

MySQL下载地址：
https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.18-linux-glibc2.12-x86_64.tar.xz
MD5: 686b454f7f7f0b0bf4814929fcd0fa81

操作系统安装请参考《专职DBA-Linux实验环境安装部署》
安装MySQL请参考《专职DBA-互联网大厂MySQL标准化安装》
解压MySQL8.0二进制包：
# xz -d mysql-8.0.18-linux-glibc2.12-x86_64.tar.xz
# tar -xf mysql-8.0.18-linux-glibc2.12-x86_64.tar -C /u01/app/mysql/
配置好先不要初始化MySQL，克隆出两台主机mgr02,mgr03（并修改主机名和ip地址）

my.cnf配置文件参考（MGR大多是独占服务器，配置文件可以放在/etc/my.cnf）
周老师还是喜欢把配置文件放在老地方/data/mysql/3306/conf/my.cnf
配置文件参考：https://www.cnblogs.com/zhouwanchun/p/11366260.html
my.cnf配置文件注意的地方：
server_id = 313306  #每个节点的server_id不能一样
binlog_format = row  #日志格式必须为ROW
log_bin = /data/mysql/3306/binlog/mysql-bin
gtid_mode = on
enforce_gtid_consistency = on
# MySQL Group Replication
master_info_repository = table
relay_log_info_repository = table
binlog_checksum = none
log_slave_updates = on
transaction_write_set_extraction = XXHASH64
loose-group_replication_group_name = "3ff8dc74-e421-43eb-add3-abf2671886c5"
loose-group_replication_start_on_boot = off
loose-group_replication_local_address = "10.0.0.31:33061"
loose-group_replication_group_seeds = "10.0.0.31:33061,10.0.0.32:33061,10.0.0.33:33061"
loose-group_replication_bootstrap_group = off

字符集采用utf8mb4
[root@mgr01 ~]# grep "utf8mb4" /data/mysql/3306/conf/my.cnf
default-character-set = utf8mb4
default-character-set = utf8mb4
character_set_server                    = utf8mb4                                       # latin1

特别说明：
server_id 需要更改
loose-group_replication_local_address = 更改为本地节点的ip
loose-group_replication_group_name = 可以使用uuidgen命令生成

[root@mgr01 ~]# uuidgen
3ff8dc74-e421-43eb-add3-abf2671886c5

[root@mgr01 ~]# vim /etc/hosts
10.0.0.14   proxysql
10.0.0.31   mgr01
10.0.0.32   mgr02
10.0.0.33   mgr03


关机，现在克隆mgr02,mgr03
[root@mgr01 ~]# init 0
记得修改主机名、ip地址、server_id、loose-group_replication_local_address


初始化MySQL8.0的数据库
分别在mgr01,mgr02,mgr03上执行初始化，并且确保初始化没有报错信息。
[root@mgr01 ~]# mysqld --defaults-file=/data/mysql/3306/conf/my.cnf --initialize-insecure
[root@mgr01 ~]# cat /data/mysql/3306/log/error.log

[root@mgr02 ~]# mysqld --defaults-file=/data/mysql/3306/conf/my.cnf --initialize-insecure
[root@mgr02 ~]# cat /data/mysql/3306/log/error.log

[root@mgr03 ~]# mysqld --defaults-file=/data/mysql/3306/conf/my.cnf --initialize-insecure
[root@mgr03 ~]# cat /data/mysql/3306/log/error.log


启动第一个节点数据库mgr01
[root@mgr01 ~]# mysqld --defaults-file=/data/mysql/3306/conf/my.cnf &
[root@mgr01 ~]# ps -ef | grep mysqld
[root@mgr01 ~]# netstat -lnp | grep mysqld

[root@mgr01 ~]# mysql -S /data/mysql/3306/sock/mysql.sock
[root@localhost:mysql.sock] [(none)]> set sql_log_bin = off;
[root@localhost:mysql.sock] [(none)]> set global read_only = off;
[root@localhost:mysql.sock] [(none)]> set global super_read_only = off;

设置密码
[root@localhost:mysql.sock] [(none)]> alter user user() identified by '123';

创建复制用户mgr
[root@localhost:mysql.sock] [(none)]> create user 'mgr'@'10.0.0.%' identified by '123';
[root@localhost:mysql.sock] [(none)]> grant replication slave on *.* to 'mgr'@'10.0.0.%';
[root@localhost:mysql.sock] [(none)]> set sql_log_bin = on;

开启复制
[root@localhost:mysql.sock] [(none)]> change master to master_user='mgr',master_password='123' for channel 'group_replication_recovery';

加载GR插件
[root@localhost:mysql.sock] [(none)]> install plugin group_replication soname 'group_replication.so';

[root@localhost:mysql.sock] [(none)]> show plugins;
[root@localhost:mysql.sock] [(none)]> select * from mysql.plugin;
+-------------------+----------------------+
| name              | dl                   |
+-------------------+----------------------+
| group_replication | group_replication.so |
+-------------------+----------------------+
1 row in set (0.00 sec)

启用第一个节点的Group Replication复制程序
[root@localhost:mysql.sock] [(none)]> set global group_replication_bootstrap_group = on; #只在第一个节点使用
[root@localhost:mysql.sock] [(none)]> start group_replication;

检查复制组是否创建并已加入新成员
[root@localhost:mysql.sock] [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
1 row in set (0.00 sec)

[root@localhost:mysql.sock] [(none)]> set global group_replication_bootstrap_group = off;
off掉是为了故障切换更快一些


注：前面的用户密码修改和创建用户操作必须设置binlog不记录，执行后再打开，否则会引起START GROUP_REPLICATION执行报错：
ERROR 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log.
解决方案是：
根据提示打开group_replication_allow_local_disjoint_gtids_join选项，
MySQL命令行执行：
mysql> set global group_replication_allow_local_disjoint_gtids_join=ON;
然后再执行:
mysql> start group_replication;



启动第二个节点数据库mgr02
[root@mgr02 ~]# mysqld --defaults-file=/data/mysql/3306/conf/my.cnf &
[root@mgr02 ~]# ps -ef | grep mysqld
[root@mgr02 ~]# netstat -lnp | grep mysqld

[root@mgr02 ~]# mysql -S /data/mysql/3306/sock/mysql.sock
[root@localhost:mysql.sock] [(none)]> set sql_log_bin = off;
[root@localhost:mysql.sock] [(none)]> set global read_only = off;
[root@localhost:mysql.sock] [(none)]> set global super_read_only = off;
[root@localhost:mysql.sock] [(none)]> alter user user() identified by '123';
[root@localhost:mysql.sock] [(none)]> create user 'mgr'@'10.0.0.%' identified by '123';
[root@localhost:mysql.sock] [(none)]> grant replication slave on *.* to 'mgr'@'10.0.0.%';
[root@localhost:mysql.sock] [(none)]> set sql_log_bin= on;

[root@localhost:mysql.sock] [(none)]> change master to master_user='mgr',master_password='123' for channel 'group_replication_recovery';
[root@localhost:mysql.sock] [(none)]> install plugin group_replication soname 'group_replication.so';
[root@localhost:mysql.sock] [(none)]> select * from mysql.plugin;
+-------------------+----------------------+
| name              | dl                   |
+-------------------+----------------------+
| group_replication | group_replication.so |
+-------------------+----------------------+
1 row in set (0.00 sec)


启用第二个节点的Group Replication
set global group_replication_bootstrap_group = on;  #不需要执行
[root@localhost:mysql.sock] [(none)]> start group_replication;

确认第二个节点启动成功
[root@localhost:mysql.sock] [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bb0eed1f-7031-11ea-b8be-000c292c6112 | mgr02       |        3306 | RECOVERING   | SECONDARY   | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
2 rows in set (0.00 sec)


启动第三个节点数据库mgr03
[root@mgr03 ~]# mysqld --defaults-file=/data/mysql/3306/conf/my.cnf &
[root@mgr03 ~]# ps -ef | grep mysqld
[root@mgr03 ~]# netstat -lnp | grep mysqld

[root@mgr03 ~]# mysql -S /data/mysql/3306/sock/mysql.sock
[root@localhost:mysql.sock] [(none)]> set sql_log_bin = off;
[root@localhost:mysql.sock] [(none)]> set global read_only = off;
[root@localhost:mysql.sock] [(none)]> set global super_read_only = off;
[root@localhost:mysql.sock] [(none)]> alter user user() identified by '123';
[root@localhost:mysql.sock] [(none)]> create user 'mgr'@'10.0.0.%' identified by '123';
[root@localhost:mysql.sock] [(none)]> grant replication slave on *.* to 'mgr'@'10.0.0.%';
[root@localhost:mysql.sock] [(none)]> set sql_log_bin= off;


[root@localhost:mysql.sock] [(none)]> change master to master_user='mgr',master_password='123' for channel 'group_replication_recovery';
[root@localhost:mysql.sock] [(none)]> install plugin group_replication soname 'group_replication.so';
[root@localhost:mysql.sock] [(none)]> select * from mysql.plugin;
+-------------------+----------------------+
| name              | dl                   |
+-------------------+----------------------+
| group_replication | group_replication.so |
+-------------------+----------------------+
1 row in set (0.00 sec)

启用第三个节点的Group Replication
set global group_replication_bootstrap_group = on;  #不需要执行
[root@localhost:mysql.sock] [(none)]> start group_replication;

确认第三个节点启动成功
[root@localhost:mysql.sock] [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bb0eed1f-7031-11ea-b8be-000c292c6112 | mgr02       |        3306 | RECOVERING   | SECONDARY   | 8.0.18         |
| group_replication_applier | bcf37e71-7031-11ea-89f0-000c2989dd62 | mgr03       |        3306 | RECOVERING   | SECONDARY   | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
3 rows in set (0.00 sec)

确认复制账号是不是可用（使用有安装MySQL8.0的环境连接测试）
# mysql -umgr -p -h10.0.0.31 -P3306
# mysql -umgr -p -h10.0.0.32 -P3306
# mysql -umgr -p -h10.0.0.33 -P3306
mysql> select @@hostname;
mysql> select @@server_id;
目前环境安装完毕。可以看到mgr01是当前的主节点，可以进行读写操作。其他节点只能读。
MySQL5.7的密码认证方式是：
mysql> show variables like "%default_authentication_plugin%";
+-------------------------------+-----------------------+
| Variable_name                 | Value                 |
+-------------------------------+-----------------------+
| default_authentication_plugin | mysql_native_password |
+-------------------------------+-----------------------+
1 row in set, 1 warning (0.02 sec)

mysql> select user,host,plugin from mysql.user;
+---------------+-----------+-----------------------+
| user          | host      | plugin                |
+---------------+-----------+-----------------------+
| root          | localhost | mysql_native_password |
| mysql.session | localhost | mysql_native_password |
| mysql.sys     | localhost | mysql_native_password |
+---------------+-----------+-----------------------+
3 rows in set (0.00 sec)

MySQL8.0的密码认证方式是：
[root@localhost:mysql.sock] [(none)]> show variables like "%default_authentication_plugin%";
+-------------------------------+-----------------------+
| Variable_name                 | Value                 |
+-------------------------------+-----------------------+
| default_authentication_plugin | caching_sha2_password |
+-------------------------------+-----------------------+
1 row in set (0.03 sec)

[root@localhost:mysql.sock] [(none)]> select user,host,plugin from mysql.user;
+------------------+-----------+-----------------------+
| user             | host      | plugin                |
+------------------+-----------+-----------------------+
| mgr              | 10.0.0.%  | caching_sha2_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session    | localhost | caching_sha2_password |
| mysql.sys        | localhost | caching_sha2_password |
| root             | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
5 rows in set (0.01 sec)
如果你想改：alter user 'mgr'@'10.0.0.%' identified with mysql_native_password by '123';
然后就可以使用低版本连接MySQL8.0的数据库了。


环境确认
[root@localhost:mysql.sock] [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bb0eed1f-7031-11ea-b8be-000c292c6112 | mgr02       |        3306 | ONLINE       | SECONDARY   | 8.0.18         |
| group_replication_applier | bcf37e71-7031-11ea-89f0-000c2989dd62 | mgr03       |        3306 | ONLINE       | SECONDARY   | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
3 rows in set (0.00 sec)

在single-master中Secondary节点，会自动设置super_read_only=1;和read_only=1;
Secondary:
mgr02 [(none)]> select @@super_read_only,@@read_only;
+-------------------+-------------+
| @@super_read_only | @@read_only |
+-------------------+-------------+
|                 1 |           1 |
+-------------------+-------------+
1 row in set (0.00 sec)

mgr03 [(none)]> select @@super_read_only,@@read_only;
+-------------------+-------------+
| @@super_read_only | @@read_only |
+-------------------+-------------+
|                 1 |           1 |
+-------------------+-------------+
1 row in set (0.00 sec)

mgr02 [(none)]> show global variables like "%read_only%";
+-----------------------+-------+
| Variable_name         | Value |
+-----------------------+-------+
| innodb_read_only      | OFF   |
| read_only             | ON    |
| super_read_only       | ON    |
| transaction_read_only | OFF   |
+-----------------------+-------+
4 rows in set (0.00 sec)

mgr03 [(none)]> show global variables like "%read_only%";
+-----------------------+-------+
| Variable_name         | Value |
+-----------------------+-------+
| innodb_read_only      | OFF   |
| read_only             | ON    |
| super_read_only       | ON    |
| transaction_read_only | OFF   |
+-----------------------+-------+
4 rows in set (0.01 sec)


Primary:
mgr01 [(none)]> select @@super_read_only,@@read_only;
+-------------------+-------------+
| @@super_read_only | @@read_only |
+-------------------+-------------+
|                 0 |           0 |
+-------------------+-------------+
1 row in set (0.00 sec)

mgr01 [(none)]> show global variables like "%read_only%";
+-----------------------+-------+
| Variable_name         | Value |
+-----------------------+-------+
| innodb_read_only      | OFF   |
| read_only             | OFF   |
| super_read_only       | OFF   |
| transaction_read_only | OFF   |
+-----------------------+-------+
4 rows in set (0.00 sec)


测试MGR
mgr02 [(none)]> create database app01;
ERROR 1290 (HY000): The MySQL server is running with the --super-read-only option so it cannot execute this statement

mgr03 [(none)]> create database app01;
ERROR 1290 (HY000): The MySQL server is running with the --super-read-only option so it cannot execute this statement


mgr01 [(none)]> create database app01;
Query OK, 1 row affected (0.02 sec)

mgr01 [(none)]> use app01;
Database changed

mgr01 [app01]> create table t1(id int not null primary key auto_increment,name varchar(64) not null);
Query OK, 0 rows affected (0.05 sec)

mgr01 [app01]> insert into t1(name) values('zhouwanchun');
mgr01 [app01]> insert into t1(name) values('zhouwanchun');
mgr01 [app01]> insert into t1(name) values('zhouwanchun');

mgr01 [app01]> select * from t1;
+----+-------------+
| id | name        |
+----+-------------+
|  1 | zhouwanchun |
|  2 | zhouwanchun |
|  3 | zhouwanchun |
+----+-------------+
3 rows in set (0.00 sec)

mgr02 [(none)]> select * from app01.t1;
+----+-------------+
| id | name        |
+----+-------------+
|  1 | zhouwanchun |
|  2 | zhouwanchun |
|  3 | zhouwanchun |
+----+-------------+
3 rows in set (0.00 sec)

mgr03 [(none)]> select * from app01.t1;
+----+-------------+
| id | name        |
+----+-------------+
|  1 | zhouwanchun |
|  2 | zhouwanchun |
|  3 | zhouwanchun |
+----+-------------+
3 rows in set (0.00 sec)


创建业务账号：（Primary）
mgr01 [(none)]> create user 'app01'@'10.0.0.%' identified by '123';
mgr01 [(none)]> grant all privileges on *.* to 'app01'@'10.0.0.%';

mgr02 [(none)]> select user,host,plugin from mysql.user;
+------------------+-----------+-----------------------+
| user             | host      | plugin                |
+------------------+-----------+-----------------------+
| app01            | 10.0.0.%  | caching_sha2_password |
| mgr              | 10.0.0.%  | caching_sha2_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session    | localhost | caching_sha2_password |
| mysql.sys        | localhost | caching_sha2_password |
| root             | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
6 rows in set (0.00 sec)

mgr03 [(none)]> select user,host,plugin from mysql.user;
+------------------+-----------+-----------------------+
| user             | host      | plugin                |
+------------------+-----------+-----------------------+
| app01            | 10.0.0.%  | caching_sha2_password |
| mgr              | 10.0.0.%  | caching_sha2_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session    | localhost | caching_sha2_password |
| mysql.sys        | localhost | caching_sha2_password |
| root             | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
6 rows in set (0.01 sec)


由单写变成多写
MGR single-master到multi-master转换
准备工作：
原来的三个节点全部关闭
[root@mgr01 ~]# mysqladmin -S /data/mysql/3306/sock/mysql.sock -p shutdown
[root@mgr02 ~]# mysqladmin -S /data/mysql/3306/sock/mysql.sock -p shutdown
[root@mgr03 ~]# mysqladmin -S /data/mysql/3306/sock/mysql.sock -p shutdown

在三个节点my.cnf配置文件中添加：
# vim /data/mysql/3306/conf/my.cnf
# multi-master
loose-group_replication_single_primary_mode = off
loose-group_replication_enforce_update_everywhere_checks = on


分别启动三个节点：
[root@mgr01 ~]# mysqld --defaults-file=/data/mysql/3306/conf/my.cnf &
[root@mgr02 ~]# mysqld --defaults-file=/data/mysql/3306/conf/my.cnf &
[root@mgr03 ~]# mysqld --defaults-file=/data/mysql/3306/conf/my.cnf &


分别测试一下复制账号是不是可用：
# mysql -umgr -p -h10.0.0.31 -P3306
# mysql -umgr -p -h10.0.0.32 -P3306
# mysql -umgr -p -h10.0.0.33 -P3306
mysql> select @@hostname;


第一个节点mgr01（不是必须选择mgr01，也可以用其他节点）
[root@mgr01 ~]# mysql -S /data/mysql/3306/sock/mysql.sock -p
mgr01 [(none)]> set global group_replication_bootstrap_group = on;
mgr01 [(none)]> start group_replication;
mgr01 [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
1 row in set (0.02 sec)

mgr01 [(none)]> set global group_replication_bootstrap_group = off;


其他节点（mgr02,mgr03）：
[root@mgr02 ~]# mysql -S /data/mysql/3306/sock/mysql.sock -p
mgr02 [(none)]> start group_replication;
mgr02 [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bb0eed1f-7031-11ea-b8be-000c292c6112 | mgr02       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
2 rows in set (0.00 sec)


[root@mgr03 ~]# mysql -S /data/mysql/3306/sock/mysql.sock -p
mgr03 [(none)]> start group_replication;
mgr03 [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bb0eed1f-7031-11ea-b8be-000c292c6112 | mgr02       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bcf37e71-7031-11ea-89f0-000c2989dd62 | mgr03       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
3 rows in set (0.02 sec)


最终状态：
可以看到三个节点都是Primary的状态
mgr01 [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bb0eed1f-7031-11ea-b8be-000c292c6112 | mgr02       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bcf37e71-7031-11ea-89f0-000c2989dd62 | mgr03       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
3 rows in set (0.00 sec)

三个节点都是读写状态
mgr01 [(none)]> select @@super_read_only,@@read_only;
+-------------------+-------------+
| @@super_read_only | @@read_only |
+-------------------+-------------+
|                 0 |           0 |
+-------------------+-------------+
1 row in set (0.00 sec)

mgr02 [(none)]> select @@super_read_only,@@read_only;
+-------------------+-------------+
| @@super_read_only | @@read_only |
+-------------------+-------------+
|                 0 |           0 |
+-------------------+-------------+
1 row in set (0.00 sec)

mgr03 [(none)]> select @@super_read_only,@@read_only;
+-------------------+-------------+
| @@super_read_only | @@read_only |
+-------------------+-------------+
|                 0 |           0 |
+-------------------+-------------+
1 row in set (0.00 sec)


MGR集群中节点加入
对于一个运行很久的集群，如果需要加入MGR集群成为复制节点，新节点除了需要执行以下：
set sql_log_bin = off;
set global read_only = off;
set global super_read_only = off;
alter user user() identified by '123';
create user 'mgr'@'10.0.0.%' identified by '123';
grant replication slave on *.* to 'mgr'@'10.0.0.%';
set sql_log_bin = on;
change master to master_user='mgr',master_password='123' for channel 'group_replication_recovery';
install plugin group_replication soname 'group_replication.so';
使用：start group_replication;已经行不通了。
可以考虑使用mysqldump,Xtrabackup做备份，利用备份把新节点恢复出来。
最重要的设置：gtid_purged，参考《基于gtid复制利用备份建一个新从库方法》
set global gtid_purged = 'xxxxxx';
然后就可以直接：start group_replication;
select * from performance_schema.replication_group_members;

使用Xtrabackup备份恢复的节点不需要执行绿色部分的SQL。
使用mysqldump/mydumper备份恢复的节点需要执行绿色部分的SQL。
mysqldump备份自动包含gtid_purged


1.MGR工作的两种模型：
(1).基于single-master环境
在Group Replication配置中，默认的模式是：single-master模式。
在single-master模式中，只有一个节点是可以进行写操作，其他节点是开启着read_only模式。
当主节点挂起，其他节点中会选举出来一个新的new-master，这里面看起来还是主从结构的概念。
在single-master模式中可以利用下面SQL发现谁是主节点：
mgr01 [(none)]> select VARIABLE_NAME,VARIABLE_VALUE from performance_schema.global_status where VARIABLE_NAME='group_replication_primary_member';
mgr01 [(none)]> select VARIABLE_VALUE from performance_schema.global_status where VARIABLE_NAME='group_replication_primary_member';
+--------------------------------------+
| VARIABLE_VALUE                       |
+--------------------------------------+
| 25b47e38-b2ee-11e9-85fe-000c297dcffc |
+--------------------------------------+
1 row in set (0.01 sec)

[root@mgr01 ~]# cat /data/mysql/3306/data/auto.cnf
[auto]
server-uuid=25b47e38-b2ee-11e9-85fe-000c297dcffc

mgr01 [(none)]> show global variables like "%server_uuid%";
+---------------+--------------------------------------+
| Variable_name | Value                                |
+---------------+--------------------------------------+
| server_uuid   | 25b47e38-b2ee-11e9-85fe-000c297dcffc |
+---------------+--------------------------------------+
1 row in set (0.01 sec)

或者
mgr01 [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bb0eed1f-7031-11ea-b8be-000c292c6112 | mgr02       |        3306 | ONLINE       | SECONDARY   | 8.0.18         |
| group_replication_applier | bcf37e71-7031-11ea-89f0-000c2989dd62 | mgr03       |        3306 | ONLINE       | SECONDARY   | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
3 rows in set (0.00 sec)


(2).基于multi-master环境
该结构不是默认模式，只需要在三个节点配置文件[mysqld]中部分添加：
# vim /data/mysql/3306/my.cnf
# multi-master
loose-group_replication_single_primary_mode = off
loose-group_replication_enforce_update_everywhere_checks = on
在该结构中需要注意，在使用中，不能同时在不同节点对同一行数据进行update操作。
如果进行，客户端会收到报错。在使用这个结构时，需要在应用层引入控制。

特别注意：在multi-master结构中group_replication_primary_member为空。
mgr01 [(none)]> select VARIABLE_NAME,VARIABLE_VALUE from performance_schema.global_status where VARIABLE_NAME='group_replication_primary_member';
+----------------------------------+----------------+
| VARIABLE_NAME                    | VARIABLE_VALUE |
+----------------------------------+----------------+
| group_replication_primary_member |                |
+----------------------------------+----------------+
1 row in set (0.00 sec)


2.集群重启
集群重启分成两种情况。
(1).滚动重启
节点启动后，执行：
start group_replication;
确认状态即可。

(2).全部关闭后（集体掉电）
第一个节点启动（single-master & multi-master都一样）
set global group_replication_bootstrap_group = on;
start group_replication;

其他节点启动：
start group_replication;


3.故障处理
节点在start group_replication;后处于RECOVERING状态。
mgr01 [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bb0eed1f-7031-11ea-b8be-000c292c6112 | mgr02       |        3306 | RECOVERING   | SECONDARY   | 8.0.18         |
| group_replication_applier | bcf37e71-7031-11ea-89f0-000c2989dd62 | mgr03       |        3306 | RECOVERING   | SECONDARY   | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
3 rows in set (0.00 sec)

mgr01 [(none)]> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************
                                      CHANNEL_NAME: group_replication_applier
                                        GROUP_NAME: 3ff8dc74-e421-43eb-add3-abf2671886c5
                                       SOURCE_UUID: 3ff8dc74-e421-43eb-add3-abf2671886c5
                                         THREAD_ID: NULL
                                     SERVICE_STATE: ON
                         COUNT_RECEIVED_HEARTBEATS: 0
                          LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00.000000
                          RECEIVED_TRANSACTION_SET: 3ff8dc74-e421-43eb-add3-abf2671886c5:1-15
                                 LAST_ERROR_NUMBER: 0
                                LAST_ERROR_MESSAGE: 
                              LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00.000000
                           LAST_QUEUED_TRANSACTION: 3ff8dc74-e421-43eb-add3-abf2671886c5:15
 LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
LAST_QUEUED_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
     LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP: 2019-07-31 05:44:53.625881
       LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP: 2019-07-31 05:44:53.626141
                              QUEUEING_TRANSACTION: 
    QUEUEING_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
   QUEUEING_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
        QUEUEING_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
1 row in set (0.00 sec)

mgr02 [(none)]> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************
                                      CHANNEL_NAME: group_replication_applier
                                        GROUP_NAME: 3ff8dc74-e421-43eb-add3-abf2671886c5
                                       SOURCE_UUID: 3ff8dc74-e421-43eb-add3-abf2671886c5
                                         THREAD_ID: NULL
                                     SERVICE_STATE: ON
                         COUNT_RECEIVED_HEARTBEATS: 0
                          LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00.000000
                          RECEIVED_TRANSACTION_SET: 3ff8dc74-e421-43eb-add3-abf2671886c5:1-15
                                 LAST_ERROR_NUMBER: 0
                                LAST_ERROR_MESSAGE: 
                              LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00.000000
                           LAST_QUEUED_TRANSACTION: 
 LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
LAST_QUEUED_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
     LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
       LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
                              QUEUEING_TRANSACTION: 
    QUEUEING_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
   QUEUEING_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
        QUEUEING_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
*************************** 2. row ***************************
                                      CHANNEL_NAME: group_replication_recovery
                                        GROUP_NAME: 
                                       SOURCE_UUID: 
                                         THREAD_ID: NULL
                                     SERVICE_STATE: OFF
                         COUNT_RECEIVED_HEARTBEATS: 0
                          LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00.000000
                          RECEIVED_TRANSACTION_SET: 
                                 LAST_ERROR_NUMBER: 2061
                                LAST_ERROR_MESSAGE: error connecting to master 'mgr@mgr01:3306' - retry-time: 60  retries: 1
                              LAST_ERROR_TIMESTAMP: 2019-07-31 06:19:44.073179
                           LAST_QUEUED_TRANSACTION: 
 LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
LAST_QUEUED_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
     LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
       LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
                              QUEUEING_TRANSACTION: 
    QUEUEING_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
   QUEUEING_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
        QUEUEING_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
2 rows in set (0.00 sec)

mgr03 [(none)]> select * from performance_schema.replication_connection_status\G
*************************** 1. row ***************************
                                      CHANNEL_NAME: group_replication_applier
                                        GROUP_NAME: 3ff8dc74-e421-43eb-add3-abf2671886c5
                                       SOURCE_UUID: 3ff8dc74-e421-43eb-add3-abf2671886c5
                                         THREAD_ID: NULL
                                     SERVICE_STATE: ON
                         COUNT_RECEIVED_HEARTBEATS: 0
                          LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00.000000
                          RECEIVED_TRANSACTION_SET: 3ff8dc74-e421-43eb-add3-abf2671886c5:1-15
                                 LAST_ERROR_NUMBER: 0
                                LAST_ERROR_MESSAGE: 
                              LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00.000000
                           LAST_QUEUED_TRANSACTION: 
 LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
LAST_QUEUED_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
     LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
       LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
                              QUEUEING_TRANSACTION: 
    QUEUEING_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
   QUEUEING_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
        QUEUEING_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
*************************** 2. row ***************************
                                      CHANNEL_NAME: group_replication_recovery
                                        GROUP_NAME: 
                                       SOURCE_UUID: 
                                         THREAD_ID: NULL
                                     SERVICE_STATE: OFF
                         COUNT_RECEIVED_HEARTBEATS: 0
                          LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00.000000
                          RECEIVED_TRANSACTION_SET: 
                                 LAST_ERROR_NUMBER: 2061
                                LAST_ERROR_MESSAGE: error connecting to master 'mgr@mgr01:3306' - retry-time: 60  retries: 1
                              LAST_ERROR_TIMESTAMP: 2019-07-31 06:20:47.614480
                           LAST_QUEUED_TRANSACTION: 
 LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
LAST_QUEUED_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
     LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
       LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
                              QUEUEING_TRANSACTION: 
    QUEUEING_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
   QUEUEING_TRANSACTION_IMMEDIATE_COMMIT_TIMESTAMP: 0000-00-00 00:00:00.000000
        QUEUEING_TRANSACTION_START_QUEUE_TIMESTAMP: 0000-00-00 00:00:00.000000
2 rows in set (0.01 sec)


解决办法，使用mgr账号在客户端去连接登录一下：
# mysql -umgr -p -h10.0.0.31 -P3306
# mysql -umgr -p -h10.0.0.32 -P3306
# mysql -umgr -p -h10.0.0.33 -P3306
stop group_replication;
start group_replication;


然后再来查看MGR状态
mgr01 [(none)]> select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 957071cc-7031-11ea-81ba-000c2984a5b2 | mgr01       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bb0eed1f-7031-11ea-b8be-000c292c6112 | mgr02       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
| group_replication_applier | bcf37e71-7031-11ea-89f0-000c2989dd62 | mgr03       |        3306 | ONLINE       | PRIMARY     | 8.0.18         |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
3 rows in set (0.00 sec)