Centos下部署DRBD+NFS+Keepalived高可用环境记录
时间:2022-04-23
本文章向大家介绍Centos下部署DRBD+NFS+Keepalived高可用环境记录,主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项,具有一定的参考价值,需要的朋友可以参考一下。
使用NFS服务器(比如图片业务),一台为主,一台为备。通常主到备的数据同步是通过rsync来做(可以结合inotify做实时同步)。由于NFS服务是存在单点的,出于对业务在线率和数据安全的保障,可以采用"DRBD+NFS+Keepalived"架构来完成高可用方案部署。之前介绍了DRBD详细解说及配置过程记录,废话不多说了,基于之前的那篇文档的机器配置信息,以下记录部署过程:
思路:
1)在两台机器上安装keepalived,VIP为192.168.1.200
2)将DRBD的挂载目录/data作为NFS的挂载目录。远程客户机使用vip地址挂载NFS
3)当Primary主机发生宕机或NFS挂了的故障时,Secondary主机提权升级为DRBD的主节点,并且VIP资源也会转移过来。
当Primary主机的故障恢复时,会再次变为DRBD的主节点,并重新夺回VIP资源。从而实现故障转移
-----------------------------------------------------------------------------------------------------------
Primary和Secondary两台主机的DRBD环境部署,参见http://www.cnblogs.com/kevingrace/p/5740940.html
Primary主机(192.168.1.151)默认作为DRBD的主节点,DRBD挂载目录是/data
Secondary主机(192.168.1.152)是DRBD的备份节点
在Primary主机上查看DRBD状态,如下,可知Primary主机是DRBD的主节点
[root@Primary ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:r0 Connected Primary/Secondary UpToDate/UpToDate C /data ext4
如下,DRBD已完成挂载,挂载目录是/data
[root@Primary ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
156G 36G 112G 25% /
tmpfs 2.9G 0 2.9G 0% /dev/shm
/dev/vda1 190M 98M 83M 55% /boot
/dev/drbd0 9.8G 23M 9.2G 1% /data
DRBD数据如下
[root@Primary ~]# cd /data
[root@Primary data]# ll
total 16
-rw-r--r--. 1 root root 9 May 25 09:33 test3
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo1
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo2
-----------------------------------------------------------------------------------------------------------
在Primary和Secondary两台主机上安装NFS(可以参考:http://www.cnblogs.com/kevingrace/p/6084604.html)
[root@Primary ~]# yum install rpcbind nfs-utils
[root@Primary ~]# vim /etc/exports
/data 192.168.1.0/24(rw,sync,no_root_squash)
[root@Primary ~]# /etc/init.d/rpcbind start
[root@Primary ~]# /etc/init.d/nfs start
---------------------------------------------------------------------------------------------------------
关闭两台主机的iptables防火墙
防火墙最好关闭,否则可能导致客户机挂载nfs时会失败!
若开启防火墙,需要在iptables中开放nfs相关端口机以及VRRP组播地址
[root@Primary ~]# /etc/init.d/iptables stop
两台机器上的selinux一定要关闭!!!!!!!!!!
否则下面在keepalived.conf里配置的notify_master.sh等脚本执行失败!这是曾经踩过的坑!
[root@Primary ~]# setenforce 0 //临时关闭。永久关闭的话,还需要在/etc/sysconfig/selinux 文件里将SELINUX改为disabled
[root@Primary ~]# getenforce
Permissive
-----------------------------------------------------------------------------------------------------------
在两台主机上安装Keepalived,配合keepalived实现自动fail-over
安装Keepalived
[root@Primary ~]# yum install -y openssl-devel popt-devel
[root@Primary ~]# cd /usr/local/src/
[root@Primary src]# wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz
[root@Primary src]# tar -zvxf keepalived-1.3.5.tar.gz
[root@Primary src]# cd keepalived-1.3.5
[root@Primary keepalived-1.3.5]# ./configure --prefix=/usr/local/keepalived
[root@Primary keepalived-1.3.5]# make && make install
[root@Primary keepalived-1.3.5]# cp /usr/local/src/keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/rc.d/init.d/
[root@Primary keepalived-1.3.5]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@Primary keepalived-1.3.5]# mkdir /etc/keepalived/
[root@Primary keepalived-1.3.5]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[root@Primary keepalived-1.3.5]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
[root@Primary keepalived-1.3.5]# echo "/etc/init.d/keepalived start" >> /etc/rc.local
[root@Primary keepalived-1.3.5]# chmod +x /etc/rc.d/init.d/keepalived #添加执行权限
[root@Primary keepalived-1.3.5]# chkconfig keepalived on #设置开机启动
[root@Primary keepalived-1.3.5]# service keepalived start #启动
[root@Primary keepalived-1.3.5]# service keepalived stop #关闭
[root@Primary keepalived-1.3.5]# service keepalived restart #重启
-----------Primary主机的keepalived.conf配置
[root@Primary ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-bak
[root@Primary ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DRBD_HA_MASTER
}
vrrp_script chk_nfs {
script "/etc/keepalived/check_nfs.sh"
interval 5
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
track_script {
chk_nfs
}
notify_stop /etc/keepalived/notify_stop.sh
notify_master /etc/keepalived/notify_master.sh
virtual_ipaddress {
192.168.1.200
}
}
启动keepalived服务
[root@Primary data]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@Primary data]# ps -ef|grep keepalived
root 30937 1 0 11:49 ? 00:00:00 keepalived -D
root 30939 30937 0 11:49 ? 00:00:00 keepalived -D
root 30940 30937 0 11:49 ? 00:00:00 keepalived -D
root 31123 10364 0 11:50 pts/1 00:00:00 grep --color keepalived
查看VIP
[root@Primary data]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:35:d1:d6 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.151/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.200/32 scope global eth0
inet6 fe80::f816:3eff:fe35:d1d6/64 scope link
valid_lft forever preferred_lft forever
-----------Secondary主机的keepalived.conf配置
[root@Secondary ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf-bak
[root@Secondary ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DRBD_HA_BACKUP
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
notify_master /etc/keepalived/notify_master.sh //当此机器为keepalived的master角色时执行这个脚本
notify_backup /etc/keepalived/notify_backup.sh //当此机器为keepalived的backup角色时执行这个脚本
virtual_ipaddress {
192.168.1.200
}
}
启动keepalived服务
[root@Secondary ~]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]
[root@Secondary ~]# ps -ef|grep keepalived
root 17128 1 0 11:50 ? 00:00:00 keepalived -D
root 17129 17128 0 11:50 ? 00:00:00 keepalived -D
root 17131 17128 0 11:50 ? 00:00:00 keepalived -D
root 17219 29939 0 11:50 pts/1 00:00:00 grep --color keepalived
-------------四个脚本配置---------------
1)此脚本只在Primary机器上配置
[root@Primary ~]# vim /etc/keepalived/check_nfs.sh
#!/bin/sh
###检查nfs可用性:进程和是否能够挂载
/sbin/service nfs status &>/dev/null
if [ $? -ne 0 ];then
###如果服务状态不正常,先尝试重启服务
/sbin/service nfs restart
/sbin/service nfs status &>/dev/null
if [ $? -ne 0 ];then
###若重启nfs服务后,仍不正常
###卸载drbd设备
umount /dev/drbd0
###将drbd主降级为备
drbdadm secondary r0
#关闭keepalived
/sbin/service keepalived stop
fi
fi
[root@Primary ~]# chmod 755 /etc/keepalived/check_nfs.sh
2)此脚本只在Primary机器上配置
[root@Primary ~]# mkdir /etc/keepalived/logs
[root@Primary ~]# vim /etc/keepalived/notify_stop.sh
#!/bin/bash
time=`date "+%F %H:%M:%S"`
echo -e "$time ------notify_stop------n" >> /etc/keepalived/logs/notify_stop.log
/sbin/service nfs stop &>> /etc/keepalived/logs/notify_stop.log
/bin/umount /data &>> /etc/keepalived/logs/notify_stop.log
/sbin/drbdadm secondary r0 &>> /etc/keepalived/logs/notify_stop.log
echo -e "n" >> /etc/keepalived/logs/notify_stop.log
[root@Primary ~]# chmod 755 /etc/keepalived/notify_stop.sh
3)此脚本在两台机器上都要配置
[root@Primary ~]# vim /etc/keepalived/notify_master.sh
#!/bin/bash
time=`date "+%F %H:%M:%S"`
echo -e "$time ------notify_master------n" >> /etc/keepalived/logs/notify_master.log
/sbin/drbdadm primary r0 &>> /etc/keepalived/logs/notify_master.log
/bin/mount /dev/drbd0 /data &>> /etc/keepalived/logs/notify_master.log
/sbin/service nfs restart &>> /etc/keepalived/logs/notify_master.log
echo -e "n" >> /etc/keepalived/logs/notify_master.log
[root@Primary ~]# chmod 755 /etc/keepalived/notify_master.sh
4)此脚本只在Secondary机器上配置
[root@Secondary ~]# mkdir /etc/keepalived/logs
[root@Secondary ~]# vim /etc/keepalived/notify_backup.sh
#!/bin/bash
time=`date "+%F %H:%M:%S"`
echo -e "$time ------notify_backup------n" >> /etc/keepalived/logs/notify_backup.log
/sbin/service nfs stop &>> /etc/keepalived/logs/notify_backup.log
/bin/umount /dev/drbd0 &>> /etc/keepalived/logs/notify_backup.log
/sbin/drbdadm secondary r0 &>> /etc/keepalived/logs/notify_backup.log
echo -e "n" >> /etc/keepalived/logs/notify_backup.log
[root@Secondary ~]# chmod 755 /etc/keepalived/notify_backup.sh
-----------------------------------------------------------------------------------------------------------
在远程客户机上挂载NFS
客户端只需要安装rpcbind程序,并确认服务正常
[root@huanqiu ~]# yum install rpcbind nfs-utils
[root@huanqiu ~]# /etc/init.d/rpcbind start
挂载NFS
[root@huanqiu ~]# mount -t nfs 192.168.1.200:/data /web
如下查看,发现已经成功挂载了NFS
[root@huanqiu ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
107G 15G 87G 14% /
tmpfs 2.9G 0 2.9G 0% /dev/shm
/dev/vda1 190M 67M 113M 38% /boot
192.168.1.200:/data 9.8G 23M 9.2G 1% /web
[root@huanqiu ~]# cd /web/
[root@huanqiu web]# ll
total 16
-rw-r--r--. 1 root root 9 May 25 09:33 test3
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo1
-rw-r--r--. 1 root root 5 May 25 09:34 wangshibo2
-----------------------------------------------------------------------------------------------------------
接着进行fail-over(故障)自动切换测试:
1)
先关闭Primary主机上的keepalived服务。就会发现VIP资源已经转移到Secondary主机上了。
同时,Primary主机的nfs也会主动关闭,同时Secondary会升级为DRBD的主节点
[root@Primary ~]# /etc/init.d/keepalived stop
Stopping keepalived: [ OK ]
[root@Primary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:35:d1:d6 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.151/24 brd 192.168.1.255 scope global eth0
inet6 fe80::f816:3eff:fe35:d1d6/64 scope link
valid_lft forever preferred_lft forever
查看系统日志,也能看到VIP资源转移信息
[root@Primary ~]# tail -1000 /var/log/messages
........
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:50:03 localhost Keepalived_vrrp[30940]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:51 localhost Keepalived[30937]: Stopping
May 25 11:58:51 localhost Keepalived_vrrp[30940]: VRRP_Instance(VI_1) sent 0 priority
May 25 11:58:51 localhost Keepalived_vrrp[30940]: VRRP_Instance(VI_1) removing protocol VIPs.
[root@Primary ~]# ps -ef|grep nfs
root 588 10364 0 12:13 pts/1 00:00:00 grep --color nfs
[root@Primary ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
156G 36G 112G 25% /
tmpfs 2.9G 0 2.9G 0% /dev/shm
/dev/vda1 190M 98M 83M 55% /boot
[root@Primary ~]# /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil@Build64R6, 2014-11-24 14:51:37
m:res cs ro ds p mounted fstype
0:r0 Connected Secondary/Secondary UpToDate/UpToDate C
登录到Secondary备份机器上,发现VIP资源已经转移过来
[root@Secondary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:4c:7e:88 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.152/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.200/32 scope global eth0
inet6 fe80::f816:3eff:fe4c:7e88/64 scope link
valid_lft forever preferred_lft forever
[root@Secondary ~]# tail -1000 /var/log/messages
........
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:53 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:58 localhost Keepalived_vrrp[17131]: Sending gratuitous ARP on eth0 for 192.168.1.200
May 25 11:58:58 localhost Keepalived_vrrp[17131]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.1.200
[root@Secondary ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:4c:7e:88 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.152/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.200/32 scope global eth0
inet6 fe80::f816:3eff:fe4c:7e88/64 scope link
valid_lft forever preferred_lft forever
[root@Secondary ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
156G 13G 135G 9% /
tmpfs 2.9G 0 2.9G 0% /dev/shm
/dev/vda1 190M 89M 92M 50% /boot
/dev/drbd0 9.8G 23M 9.2G 1% /data
当Primary机器的keepalived服务恢复启动后,VIP资源又会强制夺回来(可以查看/var/log/message系统日志)
并且Primary还会再次变为DRBD的主节点
2)
关闭Primary主机的nfs服务。根据监控脚本,会主动去启动nfs,只要当启动失败时,才会强制由DRBD的主节点降为备份节点,并关闭keepalived。
从而跟上面流程一样实现故障转移
结论:
在上面的主从故障切换过程中,对于客户端来说,挂载NFS不影响使用,只是会有一点的延迟。
这也验证了drbd提供的数据一致性功能(包括文件的打开和修改状态等),在客户端看来,真个切换过程就是"一次nfs重启"(主nfs停,备nfs启)。
- MySQL源码安装总结(r12笔记第12天)
- 全站启用SSL之后,如何兼容不支持https抓取的搜索引擎?
- sandbox和MHA快速测试(r12笔记第32天)
- 分享一个支持https的CDN及启用SSL后续问题汇总
- 分分钟搭建MySQL一主多从环境(r12笔记第31天)
- 相同update语句在MySQL,Oracle的不同表现(r12笔记第30天)
- 升级Nginx1.9.5以上版本,开启博客网站http2.0时代
- Oracle Data Guard压缩归档测试(二)(r12笔记第27天)
- Oracle Data Guard压缩归档效果对比(r12笔记第26天)
- PHP7.0正式版编译安装升级及WordPress问题解决分享
- MySQL自增列的重复值问题(r12笔记第25天)
- 借助腾讯云CDN开启全站https及问题解决分享
- 分分钟搭建MySQL Group Replication测试环境(二)(r12笔记第41天)
- php5编译安装常见错误和解决办法集锦
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法
- laravel 时间格式转时间戳的例子
- php新建文件的方法实例
- 解决PHP curl或file_get_contents下载图片损坏或无法打开的问题
- PHP迭代器和生成器用法实例分析
- php中关于换行的实例写法
- laravel 获取当前url的别名方法
- php中文语义分析实现方法示例
- laravel 中某一字段自增、自减的例子
- php统计数组不同元素的个数的实例方法
- PHP实现单条sql执行多个数据的insert语句方法
- PHP的JSON封装、转变及输出操作示例
- php 策略模式原理与应用深入理解
- 解决在Laravel 中处理OPTIONS请求的问题
- PHP使用观察者模式处理异常信息的方法详解
- php判断目录存在的简单方法