环境
k8s版本是1.10.6，当前已经使用手工配置的kubeconfig通过证书方式实现kubelet与API的通信，配置如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
[root@k8s-node1 kubelet]# pwd /var/lib/kubelet [root@k8s-node1 kubelet]# cat kubeconfig apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQyakNDQXNLZ0F3SUJBZ0lVYVprVFlZci9lNTYrVEI1QjI1eHplbzFzS1Bzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2N6RU9NQXdHQTFVRUJoTUZRMmhwYm1FeEVUQVBCZ05WQkFnVENGTm9ZVzVuYUdGcE1SRXdEd1lEVlFRSApFd2hUYUdGdVoyaGhhVEVUTUJFR0ExVUVDaE1LUzNWaVpYSnVaWFJsY3pFUk1BOEdBMVVFQ3hNSVUyaGhibWRvCllXa3hFekFSQmdOVkJBTVRDa3QxWW1WeWJtVjBaWE13SGhjTk1UZ3dPREF6TURnek1UQXdXaGNOTWpNd09EQXkKTURnek1UQXdXakJ6TVE0d0RBWURWUVFHRXdWRGFHbHVZVEVSTUE4R0ExVUVDQk1JVTJoaGJtZG9ZV2t4RVRBUApCZ05WQkFjVENGTm9ZVzVuYUdGcE1STXdFUVlEVlFRS0V3cExkV0psY201bGRHVnpNUkV3RHdZRFZRUUxFd2hUCmFHRnVaMmhoYVRFVE1CRUdBMVVFQXhNS1MzVmlaWEp1WlhSbGN6Q0NBU0l3RFFZSktvWklodmNOQVFFQkJRQUQKZ2dFUEFEQ0NBUW9DZ2dFQkFLeUF2NUc0czBEN243S0srcjFTdW9wc0lvcU41YWg1U0UyRGoyNndMMzlFYmkzVwpVUmo4S2tOZWRMU01wMThRTys2WjdxVXM0SkFYWE9FbzV5Zy9HdlZuREtoK2xmZEpzRjdOa3lsOEQvYWRyUmJqClltSU90cEVnTEQrSU9YZEoyVS85MUdCbHIwZUNNM2Y5WjhiRDIwc2wzK0dsK1QwWHFHa0lQWGhCVjQyTjIrbzQKVzIrd1R6TVpaOVc3c0J3bEh2eXJpNk5mYzRaRC9CcFZRN0l5QTRINm44VUp3UmNVTnZRR2luR09Scm5zYVVNbQo0T1BYYlFlUEZPVkZEeDNNeHF3NjljLzUwdnF1RlIyemg4Q0s5dE4vZmhiUUErUksrQWRTYXdmYndNelgrdEtTCmxkMThKNTBvQ3pjSEF4aHhzbnl2VDZYQzBZalVBTkhzN2tVcHZSMENBd0VBQWFObU1HUXdEZ1lEVlIwUEFRSC8KQkFRREFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUl3SFFZRFZSME9CQllFRkhwQkMxaEVST1RJNFZkVgpKT29tT3B3QlVsWHJNQjhHQTFVZEl3UVlNQmFBRkhwQkMxaEVST1RJNFZkVkpPb21PcHdCVWxYck1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUJhdmJYaXMxbFY4eTlqU1cwSTZ0b1lYZzBLWFVneUZNSGpsTWhjTnZWa1hCN0oKdkRkRW5RZ0ljc0YzVWtrNDBwM0tEZzFidmhZdWZjVUFKMFp4OTF1d0U3a0NWd2VXeDhQNlQ4OHNTcHhKS0YzSQo3RzY3ZVVUYmMrWkpBTU5tTEE2bHB2QWZBeGJvdzR3ZzZCSG1PazZGeDRCaHZjQTBQUHJEd3A0NFdhZE9Fb0dtClJjQWZnQzRic1Nibm1xMFUxSTVyblB6NjlVdDRLVVF1Z1U3bVk1LzVscVFKYmp2WWVrVEMyR2w4VEVnVkN2NkEKNFk0ZU1IS0RSZHBuUFlLRllPNk9mcDEwcjFUa3htNlAvU3U3QktzL3gvMHAzVjV1V2JSOS9iYmkzeXNrTE5QVgo4Yk8raTgzakNkelozdTNmVnZLcnFhVG83NjRIbkh1RzNOSlBMVWxKCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://172.16.150.100:6443 name: kubernetes-training contexts: - context: cluster: kubernetes-training user: system:node:k8s-node1 name: default current-context: default kind: Config preferences: {} users: - name: system:node:k8s-node1 user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVJVENDQXdtZ0F3SUJBZ0lVUjdiQmhMamZ5UitxUmlQS1FpTUdEbWt6cVpRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2N6RU9NQXdHQTFVRUJoTUZRMmhwYm1FeEVUQVBCZ05WQkFnVENGTm9ZVzVuYUdGcE1SRXdEd1lEVlFRSApFd2hUYUdGdVoyaGhhVEVUTUJFR0ExVUVDaE1LUzNWaVpYSnVaWFJsY3pFUk1BOEdBMVVFQ3hNSVUyaGhibWRvCllXa3hFekFSQmdOVkJBTVRDa3QxWW1WeWJtVjBaWE13SGhjTk1UZ3dPREF6TURnek1qQXdXaGNOTWpnd056TXgKTURnek1qQXdXakNCZ2pFT01Bd0dBMVVFQmhNRlEyaHBibUV4RVRBUEJnTlZCQWdUQ0ZOb1lXNW5hR0ZwTVJFdwpEd1lEVlFRSEV3aFRhR0Z1WjJoaGFURVZNQk1HQTFVRUNoTU1jM2x6ZEdWdE9tNXZaR1Z6TVJNd0VRWURWUVFMCkV3cExkV0psY201bGRHVnpNUjR3SEFZRFZRUURFeFZ6ZVhOMFpXMDZibTlrWlRwck9ITXRibTlrWlRFd2dnRWkKTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFEb0src2FhMUtyY0U3Y3hibVdRdDhOUDM1YgpRSTZhclJjUWNpeTZycGdCd1lNZEFYaVpLTURnSHlzdllaQmh0V1RXRCtLYTI5QzByQTd6bHUyTzg4enBWb2xSCjE0ZHo1RU9ZWElhdzFlVnBEd1d5bGptOUtsQnlIUnFOOGh3NkkzZjNDWkJaOWlEbWR5NkxxNVBkNm8rNnlSRnUKS1JrMUtSdENJOWl3alNUbEtJZGJ1TUpsNkRBenY4Q0gvQ29YeUlCNjZMSndWSUhaZHpXMFlCWHM3dUxJWFN4VgpOVkQ0V0ZiYnczVm5OZXRWWW1pSjNiZFFEUVZjQlFvTW9BcUpQcUdLTHdTS2pQV0dYc0tkaHJ3Yjh5S0czL2hoClkwdmZySll0WmVkSmZseGhvL0JybDgyRXQwTTBOT1ZHU204T1RBZkJHb2plaThzcVJ0ZnFTRXVqQ2lUMUFnTUIKQUFHamdad3dnWmt3RGdZRFZSMFBBUUgvQkFRREFnV2dNQjBHQTFVZEpRUVdNQlFHQ0NzR0FRVUZCd01CQmdncgpCZ0VGQlFjREFqQU1CZ05WSFJNQkFmOEVBakFBTUIwR0ExVWREZ1FXQkJUOTFLbmRWd1JYMFdoT0NIWXRJSzJ5ClRBZzVHVEFmQmdOVkhTTUVHREFXZ0JSNlFRdFlSRVRreU9GWFZTVHFKanFjQVZKVjZ6QWFCZ05WSFJFRUV6QVIKZ2dsck9ITXRibTlrWlRHSEJLd1FsbVF3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUJVUzYwNUhGaW9zWGk5NQpaWTVtenhXM2R2R3d5Y2NCUzJ5MlYrTGVSWFBiaGZkRzZKcCtsOS9pNFVCRzZFa1hsYTlrVWpDMjBWNUV1UHVYCnZBcGdDQmgwc01EQXN3UEMweHh4Q3NWL0xWRjI1TTh1Vi9rMU42Ny84aU5uNGx3U2tIU055WkxsVXVqNTA0TlcKeFAxVmszT3dwdTNoOVdCb29aazFYUHNQRlhyTEFwSkk4SUhqenpWUDJteVA5SnVVd2dBOSs1SytUVlJ4Y0VubQprMUhBVURSTERka3daYkVlSllVamF3bjhoWG1QL2hGdFdOa1lGZFZDQ0NZNWFiRHBZd3h1alFGU1paajBYa3hMCkFZUnE3c3NSYVN5enpzenIvUjRSRUw1LzUrMmRFSjNjYWxMWjJhZWtBc003aU4xS0E1emdVNmRoR2FTZnRUb2cKMEFVL3ZIZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBNkN2ckdtdFNxM0JPM01XNWxrTGZEVDkrVzBDT21xMFhFSElzdXE2WUFjR0RIUUY0Cm1TakE0QjhyTDJHUVliVmsxZy9pbXR2UXRLd084NWJ0anZQTTZWYUpVZGVIYytSRG1GeUdzTlhsYVE4RnNwWTUKdlNwUWNoMGFqZkljT2lOMzl3bVFXZllnNW5jdWk2dVQzZXFQdXNrUmJpa1pOU2tiUWlQWXNJMGs1U2lIVzdqQwpaZWd3TTcvQWgvd3FGOGlBZXVpeWNGU0IyWGMxdEdBVjdPN2l5RjBzVlRWUStGaFcyOE4xWnpYclZXSm9pZDIzClVBMEZYQVVLREtBS2lUNmhpaThFaW96MWhsN0NuWWE4Ry9NaWh0LzRZV05MMzZ5V0xXWG5TWDVjWWFQd2E1Zk4KaExkRE5EVGxSa3B2RGt3SHdScUkzb3ZMS2tiWDZraExvd29rOVFJREFRQUJBb0lCQUZXVE9vajJYZlpLNHdhRQo2cTd6M3ZzbkdBL1NiNzJiWnhEc3hJZmFxclNhK3hUSjRnV3grOFdnTjI2emEyQllWOTUrNUd0bjhqbGw0ZXp2CkNJalVYQWIxbW8vTC9XVStFWndySG9vM2pPTjRZZ1U5VWZMamtkczkxdmZTOFl5TVhqOTBDdnpkdDZxNjZ6RVkKQU55ZkJITTRnWEExUGRGV1lYR3pmTDgyaHc3ZmVaR2ltTUdRei80clBySG0zVHNRYXdtdEVzWkNoOElDTXlkcgp1VExBYlAydWF4QVJZTXlKU3B5MDBjOHJXQXRFOWRUYzNMQ1dzRkZRY0hEYk90NHg4MjBrUnhYYXRNcWRlSm0zCnk0SGxOakdJeDhGWnlyVjJ1S3NJK3RyQjVTSmpGTEtQOUtRSlJVWmxaNm5NRy9xbnNRZG5qYlFJTnduZStEbE4KQ1hzMGJ0RUNnWUVBL0Rob2JGZERQVHRVQnJBZC9lMW9lSGhYM0pLQytydXdLK1hRVEJmOXc5MUhHRkwyK1pwTwpUbGYwTlVyM1BkMkhnVmtPMUQ3TnFzZS9rS2VWbU5JVVNHUkZLQksyaHBzV0FSYVJ2R0N2dkpLMmwxZ3lOL0NMClRaQUJTc3dRRzNKYUpoU0UvaW0wMzE4dmlJeVBZZ1p6OFJLcVl3bU81ZmtCdzBXQndkM0tneDhDZ1lFQTY2YVkKN1VPK2FON3pxa1lHbjNMdVpwclZZSWp3Qnl5WCtNcDJMWGVVVkU4TS82SjZNRGhVMGZxTm1qRUw4WnI4bDZWRgpPZjhHbi9reVZsTFE3Q0EralRiSno2UzErZHo1YzFMN042dDhkSzFBamFhell6YWc4WENuSlRSQmNvZURNcHJaCmx5VWNKbDVXL0ZZb0JRZ2s4bkpTUUJPVVNLbU4rSjdJOTZWT3lXc0NnWUJ4K0ZOSFVZOHlwaEpzQ3owaU54c1UKQk4wKzhIMTdLMENLTllpUU9lOFRpSFBOeS8rSi9tNFAvTEovT21iS2I4WUJidXJDZ2JhUnl6MEc4cE1vd1JVbApPUm4rNTZYTzRWbHFHSGEyTnNKeVphRllkaVFMSHdqemZYVDRCWW5wdlJ4N1JKNXI4eENtTnU3VnR4NDdYMHNZCnppUnNLQytJUDZMS1NDaEFJQ2FZWlFLQmdBVytDSW4zbGFNK2s1ZDVuMTJ4VWtmanpBRlBOUEluendnNy9zYzYKMjcxMk02M2pDckUwSGpVRHN3R240aytmNURuWnlDY1NwTm12Rzkvb29yaXVUSGZxZHQ1OWVBVy9zY0J5ckRaKwpFa2xSazNvZ0xzempIdWJqUk45dU40NFA2VTJKWW5ZQmJIOHUyMVhJejY3WHRQYkhmR09kVm9heXd3Zms3cWJDCk8zcERBb0dBRW1NUGd6MHVERjlqUFNIbk00djJiQzgveVY5RHZwS0U1ZVhoVnVDblNteXZMU09xRGNsYXpVYnYKOEFZalpzUm9iV0x5YkJoc2RnV2hrZ3YvMGo1Y3JRRGRvYjhoREVjK0Z4dVU3UHRTZkhJVHBCZjl3M0xiMFFadApwOEF5MDFXZlRhanpMQjhyWHZML0RjT1hHbkFMeHptZ2gzbmZmRzlaUkhlS2lKSE55dlE9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg== |
|
1 2 3 4 5 6 7 8 9 10 11 |
[Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=docker.service Requires=docker.service [Service] ExecStart=/usr/local/bin/kubelet --allow-privileged=true --anonymous-auth=false --authorization-mode=Webhook --client-ca-file=/var/lib/kubernetes/ca.pem --cluster-dns=10.250.0.53 --cluster-domain=cluster.local --image-pull-progress-deadline=2m --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --register-node=true --runtime-request-timeout=15m --pod-infra-container-image=cargo.caicloud.io/caicloud/pause-amd64:3.0 --tls-cert-file=/var/lib/kubelet/k8s-node1.pem --tls-private-key-file=/var/lib/kubelet/k8s-node1-key.pem --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target |
在测试环境中，master节点本身也是一个node节点，通过修改相关配置实现master上这个node节点的kubelet实现自动的TLS bootstrapping.
Node TLS bootstrapping原理
(首先注意这是Node的TLS的bootstrapping，还不是node token bootstrapping）
ndoe是靠节点上的kubelet与API通信实现node向集群中的注册的。因此kubelet需要和API通信，而API又需要认证和授权，纯手工的做法是用集群的CA签发一个证书给kubelet，通过将此证书与key配置在kubeconfig文件里，最后通过设置kubelet启动参数传入该文件实现实现认证和授权。这个证书的特点是：
CN= system:node:k8s-master （node后的名字每个node都不一样）
O=system:nodes
kubeconfig中的user则设置为与CN一致的名字.
这是因为API设置了基于node的授权 --authorization-mode=Node,RBAC
但是这样给每台机器手工这么设置太麻烦了，而且证书如果有效期短，到期前还得手工再处理一遍。所以就有了node boostrapping。那么什么node bootstrapping呢，其原理就是：
-给kubelnet启动时候预先加载一个bootstrap-kubeconfig配置，这个配置总体上类似于上面手工设置的kubeconfig，但是不同的是这个kubeconfig没有配置kubelet客户端证书，而是配置了一个token。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
[root@k8s-master kubelet]# cat /etc/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=docker.service Requires=docker.service [Service] ExecStart=/usr/local/bin/kubelet \ --allow-privileged=true \ --anonymous-auth=false \ --authorization-mode=Webhook \ <<<控制kubelet如何对向kubelet服务发起请求的进行授权-例如APIserver主动向kubelet的请求通信 ，webhook利用借助APIserver的授权信息，因此，需要在k8s内部配置相关role并授权相关权限给提交上来的用户（ 用户是通过APIserver配置里的kubelet-client-certificate所配置的的证书中的CN名称）相关配置见附录 --client-ca-file=/var/lib/kubernetes/ca.pem \ <<<验证APIserver主动向kubelet通信时出示的证书 --cluster-dns=10.250.0.53 \ --cluster-domain=cluster.local \ --image-pull-progress-deadline=2m \ --bootstrap-kubeconfig=/var/lib/kubelet/bootstrap.kubeconfig \ <<<<如果--kubeconfig配置的文件不存在则使用bootstrap-kubeconfig文件;反之直接使用kubeconfig里的配置与API通信 --cert-dir=/var/lib/kubelet/ \ <<<<<自动获取到证书和可以存放的目录路径 --kubeconfig=/var/lib/kubelet/kubeconfig \ <<<<<kubelet真正与API通信需要的配置。自动获取证书后这文件会被系统自动写入配置 --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \ --network-plugin=cni \ --register-node=true \ --runtime-request-timeout=15m \ --pod-infra-container-image=cargo.caicloud.io/caicloud/pause-amd64:3.0 \ --tls-cert-file=/var/lib/kubelet/k8s-master.pem \ <<<<<<kubelet作为SSL服务端时用的证书 --tls-private-key-file=/var/lib/kubelet/k8s-master-key.pem \ --pod-manifest-path=/etc/staticpod/ \ --v=2 Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target |
(上述示例输出为实际测试最终的配置)
-API上则提前配置了基于token的认证--token-auth-file=/var/lib/kubernetes/token.csv ， 这样kubelet向API出示其token，API根据token.csv里的文件知道了对应的用户名以及该用户属于的组。API server并不需要打开 --enable-bootstrap-token-auth 功能。
-该token对应的用户（token.csv里设置的，这里是kubelet-bootstrap）是绑定到特定的clusterrole的 以便获得权限，如下：
|
1 2 3 4 5 6 7 8 9 |
[root@k8s-master ~]# kubectl get clusterrolebinding -o wide NAME AGE ROLE USERS GROUPS SERVICEACCOUNTS canal-calico 4d ClusterRole/calico kube-system/canal canal-flannel 4d ClusterRole/flannel kube-system/canal cluster-admin 4d ClusterRole/cluster-admin system:masters dashboard-gui-rolebind 3d ClusterRole/cluster-admin kube-system/dashboard-gui kubelet-auto-bootstrap 2h ClusterRole/system:certificates.k8s.io:certificatesigningrequests:nodeclient kubelet-bootstrap kubelet-auto-bootstrap-renew-cert 2h ClusterRole/system:certificates.k8s.io:certificatesigningrequests:selfnodeclient kubelet-bootstrap kubelet-bootstrap-create-csr 2h ClusterRole/system:node-bootstrapper kubelet-bootstrap |
system:node-bootstrapper 实现证书的申请
system:certificate.k8s.io.certificatesigningrequests:nodeclient 实现自动批准证书申请system:certificate.k8s.io.certificatesigningrequests:selfnodeclient 实现自动批准node到期前renew的证书的申请
-kube contoller是执行具体证书的签发人，因此在kube controller的启动参数里需要包含以下配置
|
1 2 |
--cluster-signing-cert-file=/var/lib/kubernetes/ca.pem --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem |
这里的signing证书必须是集群的CA所信任的，也就是说它可以是集群CA下的中间CA，这里我们直接配置集群本身的CA
具体操作步骤
- 产生一个token.csv 文件
1head -c 16 /dev/urandom | od -An -t x | tr -d ' '
将上述命令产生的token值，编辑写入一个token.csv文件,并将文件放置在某个目录，一般和API所用的证书目录一致
12[root@k8s-master ~]# cat /var/lib/kubernetes/token.csv00d9f28b73ef3df793e35712949ea6e5,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
注意上述内容的username是kubelet-bootstrap，组是system:kubelet-bootstrap组 - 修改API启动配置
1234567891011121314151617181920212223242526272829303132333435363738394041424344[root@k8s-master kubelet]# cat /etc/systemd/system/kube-apiserver.service[Unit]Description=Kubernetes API ServerDocumentation=https://github.com/kubernetes/kubernetes[Service]ExecStart=/usr/local/bin/kube-apiserver \--enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRestriction,PodPreset,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota \--advertise-address=172.16.150.100 \--allow-privileged=true \--anonymous-auth=true \--audit-log-maxage=30 \--audit-log-maxbackup=3 \--audit-log-maxsize=100 \--audit-log-path=/var/log/audit.log \ <<<必须同时配置--audit-policy-file指明策略文件才能生效--authorization-mode=Node,RBAC \ <<<<<<授权模式里包含Node--enable-bootstrap-token-auth \ <<<<<<<<<<TLS bootstrapping情形下，可忽略，该配置是给node token bootstrapping--token-auth-file=/var/lib/kubernetes/token.csv \ <<<<<<<<<指定token验证的文件--bind-address=0.0.0.0 \--client-ca-file=/var/lib/kubernetes/ca.pem \--enable-swagger-ui=true \--etcd-cafile=/var/lib/kubernetes/ca.pem \--etcd-certfile=/var/lib/kubernetes/kubernetes.pem \--etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \--etcd-servers=http://127.0.0.1:2379 \--event-ttl=1h \--experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \--insecure-bind-address=127.0.0.1 \--kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \--kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \ <<<<主动向kubelet发起通信时用的证书证书CN名字是kubernetes--kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \--kubelet-https=true \--runtime-config=api/all,rbac.authorization.k8s.io/v1alpha1,settings.k8s.io/v1alpha1=true \--service-account-key-file=/var/lib/kubernetes/ca-key.pem \--service-cluster-ip-range=10.250.0.0/24 \--service-node-port-range=30000-32767 \--tls-ca-file=/var/lib/kubernetes/ca.pem \--tls-cert-file=/var/lib/kubernetes/kubernetes.pem \--tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \--v=2Restart=on-failureRestartSec=5[Install]WantedBy=multi-user.target
- 修改kube controller配置
12345678910111213141516171819202122[root@k8s-master kubelet]# cat /etc/systemd/system/kube-controller-manager.service[Unit]Description=Kubernetes Controller ManagerDocumentation=https://github.com/kubernetes/kubernetes[Service]ExecStart=/usr/local/bin/kube-controller-manager \--address=0.0.0.0 \--allocate-node-cidrs=true \--cluster-cidr=10.244.0.0/16 \--cluster-name=kubernetes \--cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \ <<<<<<<<<签发用的证书--cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \ <<<<<<<<<<<对应的私钥--leader-elect=true \--master=http://127.0.0.1:8080 \--root-ca-file=/var/lib/kubernetes/ca.pem \--service-account-private-key-file=/var/lib/kubernetes/ca-key.pem \--service-cluster-ip-range=10.250.0.0/24 \--v=2Restart=on-failureRestartSec=5[Install]WantedBy=multi-user.target
- 执行角色绑定
123kubectl create clusterrolebinding kubelet-auto-bootstrap --user=kubelet-bootstrap --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclientkubectl create clusterrolebinding kubelet-auto-bootstrap-renew-cert --user=kubelet-bootstrap --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclientkubectl create clusterrolebinding kubelet-bootstrap-create-csr --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
- 产生bootstrap-kubeconfig配置文件
1234kubectl config set-cluster kubernetes --certificate-authority=/var/lib/kubernetes/ca.pem --embed-certs=true --server=https://172.16.150.100:6443 --kubeconfig=bootstrap.kubeconfigkubectl config set-credentials kubelet-bootstrap --token=00d9f28b73ef3df793e35712949ea6e5 --kubeconfig=bootstrap.kubeconfigkubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=bootstrap.kubeconfigkubectl config use-context default --kubeconfig=bootstrap.kubeconfig - 修改kubelet启动配置参数（见上文），需要注意的是--feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true配置，这是一个非正式功能，是用于对证书在到期前进行自动roate的，roate的效果如下，获取新证书（带日期标记，并被软连接）
12-rw-------. 1 root root 1545 Aug 10 16:52 kubelet-server-2018-08-10-16-52-43.pemlrwxrwxrwx. 1 root root 55 Aug 10 16:52 kubelet-server-current.pem -> /var/lib/kubelet/kubelet-server-2018-08-10-16-52-43.pem
效果验证
|
1 2 3 |
[root@k8s-master ~]# kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-02FibRya-at4QZjG6ss1ralRgqoWoIPKSWp8HGtcPx4 39m kubelet-bootstrap Approved,Issued |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
-rw-------. 1 root root 227 Aug 10 14:00 kubelet-client.key -rw-r--r--. 1 root root 1062 Aug 10 14:29 kubelet-client.crt 同时还会自动产生kubeconfig这个最终配置文件 [root@k8s-master kubelet]# openssl x509 -in kubelet-client.crt -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 30:80:a2:d1:38:7b:96:3f:7e:1e:3b:96:43:61:28:d8:26:f9:cb:6f Signature Algorithm: sha256WithRSAEncryption Issuer: C=China, ST=Shanghai, L=Shanghai, O=Kubernetes, OU=Shanghai, CN=Kubernetes Validity Not Before: Aug 10 05:55:00 2018 GMT Not After : Aug 10 05:55:00 2019 GMT Subject: O=system:nodes, CN=system:node:k8s-master Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:a6:17:f1:13:93:a3:db:2c:bb:0a:08:e9:6e:33: d8:54:16:69:a9:c1:13:53:24:b2:fa:e0:27:8e:b4: f0:00:dc:bf:f6:02:0b:2d:1d:0a:b0:8c:df:62:34: c8:26:da:49:14:6e:82:6b:c6:9b:b6:b0:73:0e:03: e7:7f:82:f1:c8 ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Client Authentication X509v3 Basic Constraints: critical CA:FALSE X509v3 Subject Key Identifier: 2B:95:C5:C5:BE:74:80:65:3E:54:5F:68:F7:59:46:28:C2:8E:58:00 X509v3 Authority Key Identifier: keyid:7A:41:0B:58:44:44:E4:C8:E1:57:55:24:EA:26:3A:9C:01:52:55:EB Signature Algorithm: sha256WithRSAEncryption 8b:8d:c9:36:ce:ad:bc:d5:85:13:1e:20:93:7f:d6:22:80:6c: a2:6f:67:72:e7:b1:5e:15:f1:4f:bb:a6:ef:92:09:8e:fb:35: 25:13:55:5e:1f:b4:28:26:23:cc:c5:4b:34:f4:53:71:29:f8: 8e:ab:31:20:65:94:6e:44:83:df:41:c8:dc:71:18:be:81:0e: 92:67:63:2a:c7:f8:1f:54:f4:1a:b1:f0:e6:b3:98:fd:bd:51: de:d1:06:82:f1:cb:f1:59:e5:8b:7a:91:3d:94:5f:6a:aa:e5: cf:19:5e:e3:15:a8:37:ae:8d:cd:f4:d4:97:ea:58:9a:c8:25: 00:fb:3a:ad:9d:ac:06:1c:0e:1e:84:a0:c8:f2:63:67:7b:e4: 72:fe:2a:d0:3f:eb:ed:b9:95:56:df:82:04:24:25:f8:a7:89: 1a:20:7b:7a:5e:1b:67:ff:ac:cd:a7:d5:a1:44:a3:a1:c6:dd: b9:d4:28:44:35:53:9c:35:15:3c:14:1a:54:7e:98:89:d9:79: d5:f1:e4:42:88:1a:25:9a:c2:8e:7c:4d:dc:d5:f7:16:3a:da: 0a:6b:67:85:28:de:77:90:13:d0:a9:33:a9:b8:ee:2a:9a:cb: fa:f3:67:ce:52:9c:af:7b:2f:c2:46:1e:a4:1c:29:e4:38:ab: b6:95:49:45 |
附录-Kubelet的Webhook授权：
配置kubelet用来授权外部请求的资源权限role
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
cat <<EOF | kubectl apply -f - apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubelet rules: - apiGroups: - "" resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics verbs: - "*" EOF |
绑定kubernetes这个用户使用上述权限，这个kubernetes用户名是API向kubelnet发起通信时候出示的客户端证书里的CN名字
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
cat <<EOF | kubectl apply -f - apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: system:kube-apiserver namespace: "" roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubernetes EOF |
实际上上述权限的验证授权过程是
-API server向kubelet发起HTTPS请求，SSL通信里API出示了一张证书（证书的CN是kubenetes）
-kubelet通过自己配置的client-ca来验证上述证书，验证通过
-因为kubelet配置webhook的授权模式，所以kubelet向API发起一个查询，这个查询是查询API的SubjectAccessReview 类型API，
--API返回相关授权结果（上述角色的绑定，给与了kubernetes用户权限node资源的权限），授权通过，通信完成。
原创文章，转载请注明来源
参考：
https://v1-10.docs.kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#token-authentication-file
https://v1-10.docs.kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
文章评论