redis集群搭建 - 码农教程

笔记内容：redis集群 笔记内容：2018-1-04

21.22 redis集群介绍
21.23/21.24 redis集群搭建配置
21.25 redis集群操作

21.22 redis集群介绍

redis cluster（redis集群）是redis3.0之后的版本才支持的架构，和其他集群一样，redis集群也是为了解决单台服务器不够用的情况。redis cluster是一个分布式的集群方案，它支持横向扩展。redis的集群方案不单止官方的redis cluster，还有第三方的Codis集群方案。Codis类似于LVS做负载均衡一样，它用的是路由的技术。而官方的redis cluster则是把所有的节点都互联，实现数据共享。

redis cluster集群的特点：

多个redis节点网络互联，数据共享
所有的节点都是一主一从（可以是多个从），其中从不提供服务，仅作为备用
不支持同时处理多个键（如mset/mget），因为redis需要把键均匀分布在各个节点上，并发量很高的情况下同时创建键值会降低性能并导致不可预测的行为。
支持在线增加、删除节点
客户端可以连任何一个主节点进行读写

21.23/21.24 redis集群搭建配置

场景：

由于资源有限，所以只使用两台机器做这个实验，两台机器上分别开启三个Redis服务(不同的端口)
A机器的IP：192.168.77.130
B机器的IP：192.168.77.128
A机器上三个端口7000,7002,7004，全部为主
B机器上三个端口7001,7003,7005，全部为从
redis版本：4.0.1
两台机器的防火墙都是关闭的

两台机器上都要编译安装redis，然后编辑并复制3个不同的redis.conf，分别设置不同的端口号、dir等参数，还需要增加cluster相关参数，然后分别启动6个redis服务

配置文件的内容可以在以下网址获得： https://coding.net/u/aminglinux/p/yuanke_centos7/git/tree/master/21NOSQL

1.A机器上编辑配置文件：

[root@localhost ~]# vim /etc/redis_7000.conf
port 7000
bind 192.168.77.130  # IP改为你自己的
daemonize yes
pidfile /var/run/redis_7000.pid
dir /data/redis_data/7000
cluster-enabled yes
cluster-config-file nodes_7000.conf
cluster-node-timeout 10100
appendonly yes
[root@localhost ~]# vim /etc/redis_7001.conf
port 7001
bind 192.168.77.130
daemonize yes
pidfile /var/run/redis_7001.pid
dir /data/redis_data/7001
cluster-enabled yes
cluster-config-file nodes_7001.conf
cluster-node-timeout 10100
appendonly yes
[root@localhost ~]# vim /etc/redis_7002.conf
port 7002
bind 192.168.77.130
daemonize yes
pidfile /var/run/redis_7002.pid
dir /data/redis_data/7002
cluster-enabled yes
cluster-config-file nodes_7002.conf
cluster-node-timeout 10100
appendonly yes

2.B机器上编辑配置文件：

[root@localhost ~]# vim /etc/redis_7003.conf
port 7003
bind 192.168.77.128
daemonize yes
pidfile /var/run/redis_7003.pid
dir /data/redis_data/7003
cluster-enabled yes
cluster-config-file nodes_7003.conf
cluster-node-timeout 10100
appendonly yes
[root@localhost ~]# vim /etc/redis_7004.conf
port 7004
bind 192.168.77.128
daemonize yes
pidfile /var/run/redis_7004.pid
dir /data/redis_data/7004
cluster-enabled yes
cluster-config-file nodes_7004.conf
cluster-node-timeout 10100
appendonly yes
[root@localhost ~]# vim /etc/redis_7005.conf
port 7005
bind 192.168.77.128
daemonize yes
pidfile /var/run/redis_7005.pid
dir /data/redis_data/7005
cluster-enabled yes
cluster-config-file nodes_7005.conf
cluster-node-timeout 10100
appendonly yes

3.启动A机器上的redis服务，并检查进程和端口：

[root@localhost ~]# mkdir /data/redis_data/{7000,7001,7002}
[root@localhost ~]# redis-server /etc/redis_7000.conf 
3321:C 04 Jan 19:06:50.351 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3321:C 04 Jan 19:06:50.351 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=3321, just started
3321:C 04 Jan 19:06:50.351 # Configuration loaded
[root@localhost ~]# redis-server /etc/redis_7001.conf 
3326:C 04 Jan 19:06:54.636 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3326:C 04 Jan 19:06:54.636 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=3326, just started
3326:C 04 Jan 19:06:54.636 # Configuration loaded
[root@localhost ~]# redis-server /etc/redis_7002.conf 
3331:C 04 Jan 19:06:57.955 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
3331:C 04 Jan 19:06:57.955 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=3331, just started
3331:C 04 Jan 19:06:57.955 # Configuration loaded
[root@localhost ~]# netstat -lntp |grep 700.
tcp        0      0 192.168.77.130:17000    0.0.0.0:*               LISTEN      3322/redis-server 1 
tcp        0      0 192.168.77.130:17001    0.0.0.0:*               LISTEN      3327/redis-server 1 
tcp        0      0 192.168.77.130:17002    0.0.0.0:*               LISTEN      3332/redis-server 1 
tcp        0      0 192.168.77.130:7000     0.0.0.0:*               LISTEN      3322/redis-server 1 
tcp        0      0 192.168.77.130:7001     0.0.0.0:*               LISTEN      3327/redis-server 1 
tcp        0      0 192.168.77.130:7002     0.0.0.0:*               LISTEN      3332/redis-server 1 
[root@localhost ~]# ps aux |grep redis
root       3322  0.1  0.4 145248  7564 ?        Ssl  19:06   0:00 redis-server 192.168.77.130:7000 [cluster]
root       3327  0.1  0.4 145248  7564 ?        Ssl  19:06   0:00 redis-server 192.168.77.130:7001 [cluster]
root       3332  0.1  0.4 145248  7564 ?        Ssl  19:06   0:00 redis-server 192.168.77.130:7002 [cluster]
root       3365  0.0  0.0 112668   972 pts/0    S+   19:09   0:00 grep --color=auto redis
[root@localhost ~]#

4.启动B机器上的redis服务，并检查进程和端口：

[root@localhost ~]# mkdir /data/redis_data/{7003,7004,7005}
[root@localhost ~]# redis-server /etc/redis_7003.conf
4842:C 04 Jan 11:14:41.314 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4842:C 04 Jan 11:14:41.314 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=4842, just started
4842:C 04 Jan 11:14:41.314 # Configuration loaded
[root@localhost ~]# redis-server /etc/redis_7004.conf
4847:C 04 Jan 11:14:43.907 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4847:C 04 Jan 11:14:43.907 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=4847, just started
4847:C 04 Jan 11:14:43.907 # Configuration loaded
[root@localhost ~]# redis-server /etc/redis_7005.conf
4852:C 04 Jan 11:14:46.184 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4852:C 04 Jan 11:14:46.184 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=4852, just started
4852:C 04 Jan 11:14:46.184 # Configuration loaded
[root@localhost ~]# netstat -lntp |grep 700.
tcp        0      0 192.168.77.128:17003    0.0.0.0:*               LISTEN      4843/redis-server 1 
tcp        0      0 192.168.77.128:17004    0.0.0.0:*               LISTEN      4848/redis-server 1 
tcp        0      0 192.168.77.128:17005    0.0.0.0:*               LISTEN      4853/redis-server 1 
tcp        0      0 192.168.77.128:7003     0.0.0.0:*               LISTEN      4843/redis-server 1 
tcp        0      0 192.168.77.128:7004     0.0.0.0:*               LISTEN      4848/redis-server 1 
tcp        0      0 192.168.77.128:7005     0.0.0.0:*               LISTEN      4853/redis-server 1 
[root@localhost ~]# ps aux |grep redis
root       4843  0.1  0.1 145244  7572 ?        Ssl  11:14   0:00 redis-server 192.168.77.128:7003 [cluster]
root       4848  0.1  0.1 145244  7568 ?        Ssl  11:14   0:00 redis-server 192.168.77.128:7004 [cluster]
root       4853  0.1  0.1 145244  7568 ?        Ssl  11:14   0:00 redis-server 192.168.77.128:7005 [cluster]
root       4911  0.0  0.0 112660   964 pts/0    S+   11:17   0:00 grep --color=auto redis
[root@localhost ~]#

5.在A机器上安装ruby2.2：

安装好依赖：

[root@localhost ~]# yum -y groupinstall "Development Tools"
[root@localhost ~]# yum -y install gdbm-devel libdb4-devel libffi-devel libyaml libyaml-devel ncurses-devel openssl-devel readline-devel tcl-deve
[root@localhost ~]# cd /root/
[root@localhost ~]# mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}

下载源码包，并把源码包打包成rpm包进行安装：

[root@localhost ~]# wget http://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.3.tar.gz
[root@localhost ~]# mv ruby-2.2.3.tar.gz rpmbuild/SOURCES/
[root@localhost ~]# wget https://raw.githubusercontent.com/tjinjin/automate-ruby-rpm/master/ruby22x.spec -P rpmbuild/SPECS
[root@localhost ~]# rpmbuild -bb rpmbuild/SPECS/ruby22x.spec  # 这一步需要等待一会
[root@localhost ~]# ls rpmbuild/RPMS/x86_64/ruby-2.2.3-1.el7.centos.x86_64.rpm  # 检查是否打包成了rpm包
rpmbuild/RPMS/x86_64/ruby-2.2.3-1.el7.centos.x86_64.rpm
[root@localhost ~]# yum -y localinstall rpmbuild/RPMS/x86_64/ruby-2.2.3-1.el7.centos.x86_64.rpm
[root@localhost ~]# ruby -v  # 检查是否安装完成
ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-linux]
[root@localhost ~]# gem install redis  # 安装与redis相关的包
Fetching: redis-4.0.1.gem (100%)
Successfully installed redis-4.0.1
Parsing documentation for redis-4.0.1
Installing ri documentation for redis-4.0.1
Done installing documentation for redis after 1 seconds
1 gem installed
[root@localhost ~]#

6.拷贝可执行文件（命令）：

[root@localhost ~]# cp /usr/local/src/redis-4.0.1/src/redis-trib.rb  /usr/bin/

7.创建replicas：

[root@localhost ~]# redis-trib.rb create --replicas 1 192.168.77.130:7000 192.168.77.130:7001 192.168.77.130:7002 192.168.77.128:7003 192.168.77.128:7004 192.168.77.128:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:  # 这里决定了哪3个作为master
192.168.77.130:7000
192.168.77.128:7003
192.168.77.130:7001  # 以下打印了主从关系
Adding replica 192.168.77.128:7004 to 192.168.77.130:7000
Adding replica 192.168.77.130:7002 to 192.168.77.128:7003
Adding replica 192.168.77.128:7005 to 192.168.77.130:7001
M: 03fc051e6ba17eee1efa6188e961455958b154e0 192.168.77.130:7000
   slots:0-5460 (5461 slots) master
M: ef2f571de1d8d1bfa5d96acab297ad857c09d33c 192.168.77.130:7001
   slots:10923-16383 (5461 slots) master
S: e95229afb8b6f22ff4414aaf1173574e39a12526 192.168.77.130:7002
   replicates c9cef49da3e8acbc47d9b3151b04589e349031b1
M: c9cef49da3e8acbc47d9b3151b04589e349031b1 192.168.77.128:7003
   slots:5461-10922 (5462 slots) master
S: 824b56352f1b464918702aba2597e9bf46eb70dc 192.168.77.128:7004
   replicates 03fc051e6ba17eee1efa6188e961455958b154e0
S: d75dd3c61389200941c6987062a79ba8db576741 192.168.77.128:7005
   replicates ef2f571de1d8d1bfa5d96acab297ad857c09d33c
Can I set the above configuration? (type 'yes' to accept): yes  # 会询问是否可以这样配置
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 192.168.77.130:7000)
M: 03fc051e6ba17eee1efa6188e961455958b154e0 192.168.77.130:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: e95229afb8b6f22ff4414aaf1173574e39a12526 192.168.77.130:7002
   slots: (0 slots) slave
   replicates c9cef49da3e8acbc47d9b3151b04589e349031b1
M: ef2f571de1d8d1bfa5d96acab297ad857c09d33c 192.168.77.130:7001
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: c9cef49da3e8acbc47d9b3151b04589e349031b1 192.168.77.128:7003
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: d75dd3c61389200941c6987062a79ba8db576741 192.168.77.128:7005
   slots: (0 slots) slave
   replicates ef2f571de1d8d1bfa5d96acab297ad857c09d33c
S: 824b56352f1b464918702aba2597e9bf46eb70dc 192.168.77.128:7004
   slots: (0 slots) slave
   replicates 03fc051e6ba17eee1efa6188e961455958b154e0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@localhost ~]# echo $?
0
[root@localhost ~]#

打印了两个OK表示集群成功。

21.25 redis集群操作

以集群的方式登录redis，并创建一个key：

[root@localhost ~]# redis-cli -c -h 192.168.77.130 -p 7000  # 加上-c选项表示以集群的方式登录
192.168.77.130:7000> set key1 123
-> Redirected to slot [9189] located at 192.168.77.128:7003  # 可以看到，这个key1被重定向到了192.168.77.128:7003这台redis机器上
OK
92.168.77.128:7003> set key2 abc
-> Redirected to slot [4998] located at 192.168.77.130:7000
OK
192.168.77.130:7000> set key3 test  # 没有重定向提示信息的就表示存储到了本地上
OK
192.168.77.130:7000>

如果不加-c的话，只会登录目标机器，操作的也只是目标机器，而不会像集群那样重定向数据。任意一个节点都可以创建key，或者查看key。

检测集群状态：

[root@localhost ~]# redis-trib.rb check  192.168.77.130:7000
>>> Performing Cluster Check (using node 192.168.77.130:7000)
M: 03fc051e6ba17eee1efa6188e961455958b154e0 192.168.77.130:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: e95229afb8b6f22ff4414aaf1173574e39a12526 192.168.77.130:7002
   slots: (0 slots) slave
   replicates c9cef49da3e8acbc47d9b3151b04589e349031b1
M: ef2f571de1d8d1bfa5d96acab297ad857c09d33c 192.168.77.130:7001
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: c9cef49da3e8acbc47d9b3151b04589e349031b1 192.168.77.128:7003
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: d75dd3c61389200941c6987062a79ba8db576741 192.168.77.128:7005
   slots: (0 slots) slave
   replicates ef2f571de1d8d1bfa5d96acab297ad857c09d33c
S: 824b56352f1b464918702aba2597e9bf46eb70dc 192.168.77.128:7004
   slots: (0 slots) slave
   replicates 03fc051e6ba17eee1efa6188e961455958b154e0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@localhost ~]#

cluster nodes命令可以列出节点：

[root@localhost ~]# redis-cli -c -h 192.168.77.130 -p 7000
192.168.77.130:7000> cluster nodes
e95229afb8b6f22ff4414aaf1173574e39a12526 192.168.77.130:7002@17002 slave c9cef49da3e8acbc47d9b3151b04589e349031b1 0 1515075493469 4 connected
ef2f571de1d8d1bfa5d96acab297ad857c09d33c 192.168.77.130:7001@17001 master - 0 1515075494478 2 connected 10923-16383
c9cef49da3e8acbc47d9b3151b04589e349031b1 192.168.77.128:7003@17003 master - 0 1515075496493 4 connected 5461-10922
d75dd3c61389200941c6987062a79ba8db576741 192.168.77.128:7005@17005 slave ef2f571de1d8d1bfa5d96acab297ad857c09d33c 0 1515075495486 6 connected
03fc051e6ba17eee1efa6188e961455958b154e0 192.168.77.130:7000@17000 myself,master - 0 1515075489000 1 connected 0-5460
824b56352f1b464918702aba2597e9bf46eb70dc 192.168.77.128:7004@17004 slave 03fc051e6ba17eee1efa6188e961455958b154e0 0 1515075493000 5 connected
192.168.77.130:7000>

cluster info命令可以查看集群信息：

192.168.77.130:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1076
cluster_stats_messages_pong_sent:1033
cluster_stats_messages_sent:2109
cluster_stats_messages_ping_received:1028
cluster_stats_messages_pong_received:1076
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:2109
192.168.77.130:7000>

cluster meet ip port 命令可以添加节点，例如我现在在B机器上新增加了一个redis服务，端口为7006，现在我要把这个新增的redis服务加入到集群节点中来：

192.168.77.130:7000> cluster meet 192.168.77.128 7006
OK
192.168.77.130:7000> cluster nodes
deca81cf98ac8206419096f8b86a7a6e925078ad 192.168.77.128:7006@17006 master - 0 1515075895000 0 connected
            .....以下省略......

可以看到7006成功添加到了节点中，身份是master。

使用以上方式添加的节点，都会是master身份，而 cluster replicate node_id 命令可以将当前节点设置为指定节点的从（slave）：

[root@localhost ~]# redis-cli -c -h 192.168.77.128 -p 7006
192.168.77.128:7006> cluster replicate ef2f571de1d8d1bfa5d96acab297ad857c09d33c
OK
192.168.77.128:7006> CLUSTER NODES
                                .....以上省略......
deca81cf98ac8206419096f8b86a7a6e925078ad 192.168.77.128:7006@17006 myself,slave ef2f571de1d8d1bfa5d96acab297ad857c09d33c 0 1515047943000 0 connected
192.168.77.128:7006>

可以看到成功把7006指定为 ef2f571de1d8d1bfa5d96acab297ad857c09d33c 这个节点的slave，这个节点对应的端口是7001。

cluster forget node_id 命令可以移除某个节点，例如我把7006移除掉：

[root@localhost ~]# redis-cli -c -h 192.168.77.130 -p 7001
192.168.77.130:7001> cluster forget deca81cf98ac8206419096f8b86a7a6e925078ad
OK
192.168.77.130:7001>

要注意的是，这个命令不能移除master身份的节点，而且当前登录的节点（myself）也不能移除，只能移除非当前登录的slave节点。如果想要移除master节点，只能先把这个master指定为slave之后再移除。

cluster saveconfig命令可以把当前集群的操作保存到配置文件里：

192.168.77.130:7001> CLUSTER SAVECONFIG 
OK
192.168.77.130:7001> quit
[root@localhost ~]# cat /data/redis_data/7001/nodes_7001.conf 
d75dd3c61389200941c6987062a79ba8db576741 192.168.77.128:7005@17005 slave ef2f571de1d8d1bfa5d96acab297ad857c09d33c 0 1515076783944 2 connected
c9cef49da3e8acbc47d9b3151b04589e349031b1 192.168.77.128:7003@17003 master - 0 1515076785957 4 connected 5461-10922
03fc051e6ba17eee1efa6188e961455958b154e0 192.168.77.130:7000@17000 master - 0 1515076784951 1 connected 0-5460
deca81cf98ac8206419096f8b86a7a6e925078ad 192.168.77.128:7006@17006 slave ef2f571de1d8d1bfa5d96acab297ad857c09d33c 0 1515076785153 2 connected
e95229afb8b6f22ff4414aaf1173574e39a12526 192.168.77.130:7002@17002 slave c9cef49da3e8acbc47d9b3151b04589e349031b1 0 1515076787970 4 connected
ef2f571de1d8d1bfa5d96acab297ad857c09d33c 192.168.77.130:7001@17001 myself,master - 0 1515076786000 2 connected 10923-16383
824b56352f1b464918702aba2597e9bf46eb70dc 192.168.77.128:7004@17004 slave 03fc051e6ba17eee1efa6188e961455958b154e0 0 1515076786964 5 connected
vars currentEpoch 6 lastVoteEpoch 0
[root@localhost ~]#