Cloud Native应用交付
  • 首页
  • 关于本站
  • 个人介绍
  • Downloads
  • Repo
    • Github
    • Container
  • F5
    • F5 Python SDK
    • F5-container
    • F5-LBaaS
  • 社交
    • 联系我
    • 微信/微博
    • 公众号
    • 打赏赞助
行至水穷处 坐看云起时
☁️We are in new App Mesh era: imesh.club ☁️
  1. 首页
  2. 容器/k8s
  3. 正文

Etcd Cluster配置

2017年06月27日 9747点热度 1人点赞 0条评论

环境:

host1(docker1): 172.16.199.17/24

host2(docker2):172.16.199.27/24

host3(docker2):172.16.199.37/24

目的:

配置etcd集群服务,为后续flannel网络及kubernetes(k8s)集群提供基础服务

配置步骤:

主机1安装配置:

下载etcd服务(当前版本v3.2.1)

1
wget https://github.com/coreos/etcd/releases/download/v3.2.1/etcd-v3.2.1-linux-amd64.tar.gz

下载完毕后,解压,将etcd及etcdctl文件拷贝到PATH可以搜索到的目录,例如/usr/bin

进入systmed的服务目录,创建etcd service使得以后可以通过systemd托管服务启动

1
[root@docker1 etcd-v3.2.1-linux-amd64]# cd /usr/lib/systemd/system/

创建名为 etcd.service文件,内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
 
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=root
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" --listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" --advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" --initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" --initial-cluster=\"${ETCD_INITIAL_CLUSTER}\" --initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\" "
Restart=on-failure
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target

创建 /var/lib/etcd目录

创建/etc/etcd/etcd.conf 目录及文件,文件内容如下:

1
2
3
4
5
6
7
8
9
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/etcd01"
ETCD_LISTEN_PEER_URLS="http://172.16.199.17:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.199.17:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.199.17:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.199.17:2379"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster1"
ETCD_INITIAL_CLUSTER="etcd01=http://172.16.199.17:2380,etcd02=http://172.16.199.27:2380"

主机2安装配置:

方法同主机1,在etcd.conf配置文件中将ETCD_NAME,DIR以及相关IP修改为etcd02和172.16.199.27,TOKEN保持不变

启动etcd服务:

在两台主机上分别执行

systemctl daemon-reload

systemctl start etcd.service

检查/var/log/messages,以及systemctl命令本身无错误输出

systemctl enable etcd.service

检验集群服务:

1
2
3
4
5
6
7
8
[root@docker1 system]# etcdctl cluster-health
member 63e5296d334e3fa is healthy: got healthy result from http://172.16.199.17:2379
member 9558b2437a9b849a is healthy: got healthy result from http://172.16.199.27:2379
 
[root@docker1 system]# etcdctl member list
63e5296d334e3fa: name=etcd01 peerURLs=http://172.16.199.17:2380 clientURLs=http://172.16.199.17:2379 isLeader=true
9558b2437a9b849a: name=etcd02 peerURLs=http://172.16.199.27:2380 clientURLs=http://172.16.199.27:2379 isLeader=false
[root@docker1 system]# etcdctl cluster heath

1
2
3
4
5
6
7
[root@docker2 etcd-v3.2.1-linux-amd64]# etcdctl cluster-health
member 63e5296d334e3fa is healthy: got healthy result from http://172.16.199.17:2379
member 9558b2437a9b849a is healthy: got healthy result from http://172.16.199.27:2379
cluster is healthy
[root@docker2 etcd-v3.2.1-linux-amd64]#  etcdctl member list
63e5296d334e3fa: name=etcd01 peerURLs=http://172.16.199.17:2380 clientURLs=http://172.16.199.17:2379 isLeader=true
9558b2437a9b849a: name=etcd02 peerURLs=http://172.16.199.27:2380 clientURLs=http://172.16.199.27:2379 isLeader=false

测试etcd:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@docker1 system]# etcdctl mkdir /testdir
[root@docker1 system]# etcdctl ls
/testdir
 
[root@docker1 system]# etcdctl set /testdir/key1 value1
value1
[root@docker1 system]# etcdctl get /testdir/key1
value1
 
[root@docker1 system]# etcdctl -o extended get /testdir/key1
Key: /testdir/key1
Created-Index: 7
Modified-Index: 7
TTL: 0
Index: 7
 
value1
 
 
[root@docker1 system]# etcdctl ls --recursive
/testdir
/testdir/key1

1
2
3
4
5
6
7
8
[root@docker2 member]# etcdctl ls
/testdir
[root@docker2 member]# etcdctl ls --recursive
/testdir
/testdir/key1
 
[root@docker2 member]# etcdctl get /testdir/key1
value1

集群冗余测试:

停止host1 etcd服务,host2无法选举成为leader,这是因为etcd集群实际上至少需要三个节点:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
[root@docker2 member]# tail -f /var/log/messages
Jun 27 21:44:45 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 430
Jun 27 21:44:46 docker2 etcd: 9558b2437a9b849a is starting a new election at term 430
Jun 27 21:44:46 docker2 etcd: 9558b2437a9b849a became candidate at term 431
Jun 27 21:44:46 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 431
Jun 27 21:44:46 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 431
Jun 27 21:44:47 docker2 etcd: health check for peer 63e5296d334e3fa could not connect: dial tcp 172.16.199.17:2380: getsockopt: connection refused
Jun 27 21:44:48 docker2 etcd: 9558b2437a9b849a is starting a new election at term 431
Jun 27 21:44:48 docker2 etcd: 9558b2437a9b849a became candidate at term 432
Jun 27 21:44:48 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 432
Jun 27 21:44:48 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 432
Jun 27 21:44:49 docker2 etcd: 9558b2437a9b849a is starting a new election at term 432
Jun 27 21:44:49 docker2 etcd: 9558b2437a9b849a became candidate at term 433
Jun 27 21:44:49 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 433
Jun 27 21:44:49 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 433
Jun 27 21:44:51 docker2 etcd: 9558b2437a9b849a is starting a new election at term 433
Jun 27 21:44:51 docker2 etcd: 9558b2437a9b849a became candidate at term 434
Jun 27 21:44:51 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 434
Jun 27 21:44:51 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 434
Jun 27 21:44:52 docker2 etcd: health check for peer 63e5296d334e3fa could not connect: dial tcp 172.16.199.17:2380: getsockopt: connection refused
Jun 27 21:44:52 docker2 etcd: 9558b2437a9b849a is starting a new election at term 434
Jun 27 21:44:52 docker2 etcd: 9558b2437a9b849a became candidate at term 435
Jun 27 21:44:52 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 435
Jun 27 21:44:52 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 435
Jun 27 21:44:54 docker2 etcd: 9558b2437a9b849a is starting a new election at term 435
Jun 27 21:44:54 docker2 etcd: 9558b2437a9b849a became candidate at term 436
Jun 27 21:44:54 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 436
Jun 27 21:44:54 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 436
Jun 27 21:44:55 docker2 etcd: 9558b2437a9b849a is starting a new election at term 436
Jun 27 21:44:55 docker2 etcd: 9558b2437a9b849a became candidate at term 437
Jun 27 21:44:55 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 437
Jun 27 21:44:55 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 437
Jun 27 21:44:56 docker2 etcd: 9558b2437a9b849a is starting a new election at term 437
Jun 27 21:44:56 docker2 etcd: 9558b2437a9b849a became candidate at term 438
Jun 27 21:44:56 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 438
Jun 27 21:44:56 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 438
Jun 27 21:44:57 docker2 etcd: health check for peer 63e5296d334e3fa could not connect: dial tcp 172.16.199.17:2380: getsockopt: connection refused
Jun 27 21:44:57 docker2 etcd: 9558b2437a9b849a is starting a new election at term 438
Jun 27 21:44:57 docker2 etcd: 9558b2437a9b849a became candidate at term 439
Jun 27 21:44:57 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 439
Jun 27 21:44:57 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 439
Jun 27 21:44:58 docker2 etcd: 9558b2437a9b849a is starting a new election at term 439
Jun 27 21:44:58 docker2 etcd: 9558b2437a9b849a became candidate at term 440
Jun 27 21:44:58 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 440
Jun 27 21:44:58 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 440
Jun 27 21:44:59 docker2 etcd: 9558b2437a9b849a is starting a new election at term 440
Jun 27 21:44:59 docker2 etcd: 9558b2437a9b849a became candidate at term 441
Jun 27 21:44:59 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 441
Jun 27 21:44:59 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 441
Jun 27 21:45:01 docker2 etcd: 9558b2437a9b849a is starting a new election at term 441
Jun 27 21:45:01 docker2 etcd: 9558b2437a9b849a became candidate at term 442
Jun 27 21:45:01 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 442
Jun 27 21:45:01 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 442
Jun 27 21:45:02 docker2 etcd: health check for peer 63e5296d334e3fa could not connect: dial tcp 172.16.199.17:2380: getsockopt: connection refused

恢复host1 etcd服务:

1
2
3
4
5
6
7
8
9
10
11
12
13
Jun 27 21:45:02 docker2 etcd: peer 63e5296d334e3fa became active
Jun 27 21:45:02 docker2 etcd: established a TCP streaming connection with peer 63e5296d334e3fa (stream MsgApp v2 writer)
Jun 27 21:45:02 docker2 etcd: established a TCP streaming connection with peer 63e5296d334e3fa (stream Message writer)
Jun 27 21:45:02 docker2 etcd: established a TCP streaming connection with peer 63e5296d334e3fa (stream MsgApp v2 reader)
Jun 27 21:45:02 docker2 etcd: established a TCP streaming connection with peer 63e5296d334e3fa (stream Message reader)
Jun 27 21:45:02 docker2 etcd: 9558b2437a9b849a is starting a new election at term 442
Jun 27 21:45:02 docker2 etcd: 9558b2437a9b849a became candidate at term 443
Jun 27 21:45:02 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 9558b2437a9b849a at term 443
Jun 27 21:45:02 docker2 etcd: 9558b2437a9b849a [logterm: 61, index: 17] sent MsgVote request to 63e5296d334e3fa at term 443
Jun 27 21:45:02 docker2 etcd: 9558b2437a9b849a received MsgVoteResp from 63e5296d334e3fa at term 443
Jun 27 21:45:02 docker2 etcd: 9558b2437a9b849a [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
Jun 27 21:45:02 docker2 etcd: 9558b2437a9b849a became leader at term 443
Jun 27 21:45:02 docker2 etcd: raft.node: 9558b2437a9b849a elected leader 9558b2437a9b849a at term 443

增加第三节点:

修改已有两个节点的etcd.conf中集群节点配置项

1
ETCD_INITIAL_CLUSTER="etcd01=http://172.16.199.17:2380,etcd02=http://172.16.199.27:2380"

安装第三节点,配置文件如下:

1
2
3
4
5
6
7
8
9
10
[root@docker3 system]# cat /etc/etcd/etcd.conf
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/etcd03"
ETCD_LISTEN_PEER_URLS="http://172.16.199.37:2380"
ETCD_LISTEN_CLIENT_URLS="http://172.16.199.37:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.199.37:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.199.37:2379"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster1"
ETCD_INITIAL_CLUSTER="etcd01=http://172.16.199.17:2380,etcd02=http://172.16.199.27:2380,etcd03=http://172.16.199.37:2380"

注意由于集群在两个节点模式下已经测试启动过,系统已经生成的cluster id是关于两个节点的,直接修改上述配置并启动第三节点,第三节点会无法加入集群,并报以下问题:

1
2
3
Jun 27 22:17:06 docker3 etcd: request sent was ignored (cluster ID mismatch: peer[63e5296d334e3fa]=93ce6eea0aad94f1, local=de36ca03b996fb38)
Jun 27 22:17:06 docker3 etcd: request sent was ignored (cluster ID mismatch: peer[63e5296d334e3fa]=93ce6eea0aad94f1, local=de36ca03b996fb38)
Jun 27 22:17:06 docker3 etcd: request sent was ignored (cluster ID mismatch: peer[9558b2437a9b849a]=93ce6eea0aad94f1, local=de36ca03b996fb38)

此时需要删除host1,host2上/var/lib/etcd/**/下的member文件夹,然后重启所有etcd服务,最终结果如下:

1
2
3
4
[root@docker1 etcd01]# etcdctl member list
63e5296d334e3fa: name=etcd01 peerURLs=http://172.16.199.17:2380 clientURLs=http://172.16.199.17:2379 isLeader=true
58f1fbf6c5601b97: name=etcd03 peerURLs=http://172.16.199.37:2380 clientURLs=http://172.16.199.37:2379 isLeader=false
9558b2437a9b849a: name=etcd02 peerURLs=http://172.16.199.27:2380 clientURLs=http://172.16.199.27:2379 isLeader=false

上述新增第三节点的正确做法,应该在启动第三节点之前,在现有节点上add member

etcdctl member add new-member-name http://new-member-ip:2380

三节点冗余测试:

关闭host1上etcd服务,并在其它host上查看发现host2变成了leader:

1
2
3
4
[root@docker3 etcd03]#  etcdctl member list
63e5296d334e3fa: name=etcd01 peerURLs=http://172.16.199.17:2380 clientURLs=http://172.16.199.17:2379 isLeader=false
58f1fbf6c5601b97: name=etcd03 peerURLs=http://172.16.199.37:2380 clientURLs=http://172.16.199.37:2379 isLeader=false
9558b2437a9b849a: name=etcd02 peerURLs=http://172.16.199.27:2380 clientURLs=http://172.16.199.27:2379 isLeader=true

1
2
3
4
5
6
[root@docker3 etcd03]# etcdctl cluster-health
failed to check the health of member 63e5296d334e3fa on http://172.16.199.17:2379: Get http://172.16.199.17:2379/health: dial tcp 172.16.199.17:2379: getsockopt: connection refused
member 63e5296d334e3fa is unreachable: [http://172.16.199.17:2379] are all unreachable
member 58f1fbf6c5601b97 is healthy: got healthy result from http://172.16.199.37:2379
member 9558b2437a9b849a is healthy: got healthy result from http://172.16.199.27:2379
cluster is healthy

查看停止host1 etcd服务之前创建的key,可以获得:

1
2
[root@docker3 etcd03]#  etcdctl get /testdir/key1
value1

创建新key(在非leader上执行):

1
2
3
4
[root@docker3 etcd03]#  etcdctl set /testdir/key2 value2
value2
[root@docker3 etcd03]#  etcdctl get /testdir/key2
value2

重新启动host1 etcd, 可以看到后创建的key2可以在host1上被查到。

注意:各个主机应该关闭firewalld服务,否则会出现类似提示无法获取某个节点健康状态的提示:

1
2
3
4
5
[root@docker1 etcd01]# etcdctl cluster-health
member 63e5296d334e3fa is healthy: got healthy result from http://172.16.199.17:2379
failed to check the health of member 58f1fbf6c5601b97 on http://172.16.199.37:2379: Get http://172.16.199.37:2379/health: dial tcp 172.16.199.37:2379: i/o timeout
member 58f1fbf6c5601b97 is unreachable: [http://172.16.199.37:2379] are all unreachable
member 9558b2437a9b849a is healthy: got healthy result from http://172.16.199.27:2379

附录:

Active Peers Majority Failure Tolerance
1 peers 1 peers None
3 peers 2 peers 1 peer
4 peers 3 peers 1 peer
5 peers 3 peers 2 peers
6 peers 4 peers 2 peers
7 peers 4 peers 3 peers
8 peers 5 peers 3 peers
9 peers 5 peers 4 peers

http://www.infoq.com/cn/articles/etcd-interpretation-application-scenario-implement-principle/

本作品采用 知识共享署名 4.0 国际许可协议 进行许可
标签: etcd etcd cluster
最后更新:2017年07月14日

纳米

http://linjing.io

打赏 点赞
< 上一篇
下一篇 >

文章评论

取消回复

纳米

http://linjing.io

☁️迈向Cloud Native ADC ☁️

认证获得:
Kubernetes: CKA #664
Microsoft: MCSE MCDBA
Cisco: CCNP
Juniper: JNCIS
F5:
F5 Certified Solution Expert, Security
F5 Certified Technology Specialist, LTM/GTM/APM/ASM
F5 Certified BIG-IP Administrator
  • 点击查看本博技术要素列表
  • 分类目录
    • Avi Networks (3)
    • Cisco ACI (1)
    • CISCO资源 (21)
    • F5 with ELK (8)
    • F5-Tech tips (38)
    • F5技术 (203)
    • Juniper (4)
    • Linux (7)
    • Nginx (18)
    • SDN (4)
    • ServiceMesh (19)
    • WEB编程 (8)
    • WINDOWS相关 (7)
    • 业界文章 (18)
    • 交换机技术 (20)
    • 化云为雨/Openstack (35)
    • 协议原理 (52)
    • 容器/k8s (64)
    • 我的工作 (19)
    • 我的生活 (70)
    • 网站技术 (19)
    • 路由器技术 (80)
    • 项目案例 (28)
    文章归档
    标签聚合
    F5 k8s openstack nginx istio DNS envoy gtm docker network flannel api irule bigip neutron cc kubernetes ELK vxlan BGP dhcp VPN IPSec lbaas ingress ingress controller nginx plus sidecar IPSec VPN NAT sql
    最新 热点 随机
    最新 热点 随机
    Say hello for 2021 二进制flannel部署,非cni网络模式下与k8s CIS结合方案 又是一年国庆 Service Account Token Volume Projection Istio ingressgateway 静态TLS证书加载与SDS发现方式配置区别 Istio里Gateway的port定义与实际ingressgateway的listener端口关系及规则 Helm 3 部署NGINX Ingress Controller 应用交付老兵眼中的Envoy, 云原生时代下的思考 Istio sidecar iptables以及流量控制分析 Istio 熔断策略及envoy配置
    Say hello for 2021
    华夏基金经理观市 [原创]理解CHAP验证过程及单双向验证 IBR(Intelligent Browser Rerferencing)总结 终于解决PIX与checkpoint互联VPN的怪问题 Apache HTTP Server 与 Tomcat 的三种连接方式 【转】openstack 的vlan配置方式 整理一些F5不常用的命令,但有时却很有用。【更新中】 一个DHCP问题的分析过程 ATM 里封装类型中的SNAP意思[821考试] 支持 edns client subnet dig下载
    链接表
    • Jimmy Song‘s Blog
    • SDNap
    • SDNlab
    • SDN论坛
    • Service Mesh社区
    • 三斗室
    • 个人profile

    COPYRIGHT © 2020 Cloud Native应用交付. ALL RIGHTS RESERVED.

    THEME KRATOS MADE BY VTROIS

    京ICP备14048088号-1

    京公网安备 11010502041506号

    [ Placeholder content for popup link ] WordPress Download Manager - Best Download Management Plugin