说说Keepalived的脑裂

1. 工作场景

Keepalived提供了Loadbalancing和High-Availability的功能, 本文说的是其为2个Mycat节点提供HA功能的场景.

2. 关键配置如下, 为主备非抢占模式.

! mycat01, 192.168.4.196

global_defs {

    # 一个keepalived.conf, 对应一个router_id

    router_id mycat01

}

vrrp_instance VI_1 {

    state BACKUP

    nopreempt

    interface eth0

    # 一个组播内, 各节点该值相同

    virtual_router_id 196

    priority 200

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass Zf4aqy

    }

    virtual_ipaddress {

        192.168.4.200

    }

}    

! mycat02, 192.168.4.195

global_defs {

    router_id mycat02

}

vrrp_instance VI_1 {

    state BACKUP

    nopreempt

    interface eth0

    virtual_router_id 196

    priority 200

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass Zf4aqy

    }

    virtual_ipaddress {

        192.168.4.200

    }

}    

3. Keepalived提供HA的工作原理

Keepalived的HA功能, 是通过VRRP(Virtual Router Redundancy Protocol, 虚拟路由冗余协议)来实现的. 其用IP组播的方式(默认组播地址: 224.0.0.18), 实现服务节点间的通信, 通过一种竞选机制来将路由任务交给某台VRRP路由器. 工作时主节点发送VRRP协议报文, 备节点接收报文, 若一段时间(默认3个报文发送时间)备节点接收不到主节点发送的报文, 就会启动接管程序接管主节点的资源. 备节点可以有多个, 通过优先级竞选.

4. Keepalived发生脑裂的情况

4.1 根据上述工作原理知道, 备节点接收不到报文时, 如两者间的网络不通了, 备节点就会启动接管程序接管主节点的资源, 对外提供服务, 表现形式就是备节点上出现了虚拟IP, 此时主节点也是持有虚拟IP的.

问题重现如下, 此时mycat01为主节点, mycat02为备节点, 在mycat02上drop掉组播数据包.

# iptables -A INPUT -m pkttype --pkt-type multicast -j DROP

然后观察message日志, 可见其也变成了主节点.

# tail -f /var/log/messages

Mar 17 00:13:58 mycat02 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team

Mar 17 00:14:00 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Mar 17 00:14:01 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Mar 17 00:14:01 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Mar 17 00:14:01 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.4.200

Mar 17 00:14:01 mycat02 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.4.200 added

Mar 17 00:14:06 mycat02 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.4.200

在mycat01上抓包, 看到mycat02也在向组播地址发送数据包.

# tcpdump -nnei eth0 | grep -i 'vrid 196'

00:13:58.735738 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

00:13:59.736268 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

00:14:00.736800 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

00:14:00.955581 fa:16:3e:5d:23:10 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.195 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

00:14:01.956340 fa:16:3e:f4:18:40 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.4.196 > 224.0.0.18: VRRPv2, Advertisement, vrid 196, prio 202, authtype simple, intvl 1s, length 20

当把iptables停掉后, mycat01和mycat02扔分别持有虚拟IP 192.168.4.200, 这也说明了主备非抢占模式的一个弊端, 主备抢占模式是不会这样的, 下面是其主要配置.

! mycat01, 192.168.4.196

global_defs {

    router_id mycat01

}

vrrp_instance VI_1 {

    state MASTER

    virtual_router_id 196

    priority 200

}    

! mycat02, 192.168.4.195

global_defs {

    router_id mycat02

}

vrrp_instance VI_1 {

    state BACKUP

    virtual_router_id 196

    priority 150

}    

4.2 同一个组播内, virtual_router_id配置不相同, 也会发生脑裂现象, 这里就不在赘述了.

说到最后, 如何避免脑裂现象呢, 其实可以看到, 只要有虚拟IP存在, 就不可能完全规避这个问题. 也就是没有虚拟IP了, 就没有脑裂了, 那么节点又如何向外提供服务呢? 引入ZooKeeper或Consul的服务发现机制, 这些又是另外的事情了...

若感兴趣可关注订阅号”数据库最佳实践”(DBBestPractice).