Warning: Array to string conversion in /home/wwwroot/myf5/wp-content/plugins/wpjam-basic/public/wpjam-utils.php on line 496
准备工作
为了学习以及理解k8s各个模块组件间关系，没有使用yum来安装，也没有使用kubeadm来做建立集群。采用全手工安装和配置方式,为了简化安装过程，集群内未使用证书来实现安全的接口通信。
环境
三台主机
172.16.199.17/24 Master
172.16.199.27/24 node
172.16.199.37/24 node
所有机器上都已安装好 docker，三机构成etcd集群，全部安装了flannel网络
系统版本Centos7
Master安装：
文件下载
本次测试安装时，k8s维护版本是1.6.7
wget https://github.com/kubernetes/kubernetes/releases/download/v1.6.7/kubernetes.tar.gz
下载完毕后，当前k8s下载的包里不再含有k8s的相关二进制文件，需要执行里面的一个脚本下载：
[root@docker1 cluster]# pwd //root/kubernetes/cluster [root@docker1 cluster]# ./get-kube-binaries.sh
下载完毕后，进入server目录，可以看到多出一个kubernetes-server-linux-amd64.tar.gz 压缩包
//root/kubernetes/server [root@docker1 server]# ls bin kubernetes-manifests.tar.gz kubernetes-salt.tar.gz kubernetes-server-linux-amd64.tar.gz README
解压这个压缩包文件到/root/kubernetes/server/bin 目录下，随后将bin目录下的所有文件拷贝到/usr/bin下。 同时将这些文件一并拷贝到27和37两台机器的/usr/bin下
Master配置
apiserver
拷贝以下脚本，并执行
[root@docker1 scripts]# pwd
/root/kubernetes/cluster/centos/master/scripts
[root@docker1 scripts]# cat apiserver_jing.sh
#!/bin/bash
# Copyright 2014 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
MASTER_ADDRESS=${1:-"172.16.199.17"}
ETCD_SERVERS=${2:-"http://172.16.199.17:2379"}
SERVICE_CLUSTER_IP_RANGE=${3:-"169.169.0.0/16"}
ADMISSION_CONTROL=${4:-"AlwaysAdmit"}
mkdir -p /etc/kubernetes
cat <<EOF >/etc/kubernetes/kube-apiserver
# --alsologtostderr=true: log to standard error as well as log file
KUBE_ALSO_LOGTOSTDERR="--alsologtostderr=true"
# --log-dir
KUBE_LOG_DIR="--log-dir=/var/log/kubernetes"
# --v=0: log level for V logs
KUBE_LOG_LEVEL="--v=2"
#stderrthreshold: only show log that over the threshold to stderr
STDERR_THRESHOLD="--stderrthreshold=2"
# --etcd-servers=[]: List of etcd servers to watch (http://ip:port),
# comma separated. Mutually exclusive with -etcd-config
KUBE_ETCD_SERVERS="--etcd-servers=${ETCD_SERVERS}"
# --etcd-cafile="": SSL Certificate Authority file used to secure etcd communication.
#KUBE_ETCD_CAFILE="--etcd-cafile=/srv/kubernetes/etcd/ca.pem"
# --etcd-certfile="": SSL certification file used to secure etcd communication.
#KUBE_ETCD_CERTFILE="--etcd-certfile=/srv/kubernetes/etcd/client.pem"
# --etcd-keyfile="": key file used to secure etcd communication.
#KUBE_ETCD_KEYFILE="--etcd-keyfile=/srv/kubernetes/etcd/client-key.pem"
# --insecure-bind-address=127.0.0.1: The IP address on which to serve the --insecure-port.
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
# --insecure-port=8080: The port on which to serve unsecured, unauthenticated access.
KUBE_API_PORT="--insecure-port=8080"
# --kubelet-port=10250: Kubelet port
NODE_PORT="--kubelet-port=10250"
# --advertise-address=<nil>: The IP address on which to advertise
# the apiserver to members of the cluster.
KUBE_ADVERTISE_ADDR="--advertise-address=${MASTER_ADDRESS}"
# --allow-privileged=false: If true, allow privileged containers.
KUBE_ALLOW_PRIV="--allow-privileged=false"
# --service-cluster-ip-range=<nil>: A CIDR notation IP range from which to assign service cluster IPs.
# This must not overlap with any IP ranges assigned to nodes for pods.
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=${SERVICE_CLUSTER_IP_RANGE}"
# --admission-control="AlwaysAdmit": Ordered list of plug-ins
# to do admission control of resources into cluster.
# Comma-delimited list of:
# LimitRanger, AlwaysDeny, SecurityContextDeny, NamespaceExists,
# NamespaceLifecycle, NamespaceAutoProvision, AlwaysAdmit,
# ServiceAccount, DefaultStorageClass, DefaultTolerationSeconds, ResourceQuota
KUBE_ADMISSION_CONTROL="--admission-control=${ADMISSION_CONTROL}"
# --client-ca-file="": If set, any request presenting a client certificate signed
# by one of the authorities in the client-ca-file is authenticated with an identity
# corresponding to the CommonName of the client certificate.
#KUBE_API_CLIENT_CA_FILE="--client-ca-file=/srv/kubernetes/ca.crt"
# --tls-cert-file="": File containing x509 Certificate for HTTPS. (CA cert, if any,
# concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file
# and --tls-private-key-file are not provided, a self-signed certificate and key are
# generated for the public address and saved to /var/run/kubernetes.
#KUBE_API_TLS_CERT_FILE="--tls-cert-file=/srv/kubernetes/server.cert"
# --tls-private-key-file="": File containing x509 private key matching --tls-cert-file.
#KUBE_API_TLS_PRIVATE_KEY_FILE="--tls-private-key-file=/srv/kubernetes/server.key"
##enable RBAC add-by-myf5.net
KUBE_RBAC="--authorization-mode=RBAC"
EOF
KUBE_APISERVER_OPTS=" \${KUBE_ALSO_LOGTOSTDERR} \\
\${STDERR_THRESHOLD} \\
\${KUBE_LOG_DIR} \\
\${KUBE_LOG_LEVEL} \\
\${KUBE_ETCD_SERVERS} \\
\${KUBE_API_ADDRESS} \\
\${KUBE_API_PORT} \\
\${NODE_PORT} \\
\${KUBE_ADVERTISE_ADDR} \\
\${KUBE_ALLOW_PRIV} \\
\${KUBE_SERVICE_ADDRESSES} \\
\${KUBE_ADMISSION_CONTROL} \\
\${KUBE_RBAC}"
cat <<EOF >/usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/kube-apiserver
ExecStart=/usr/bin/kube-apiserver ${KUBE_APISERVER_OPTS}
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver
controller-manager
拷贝以下脚本并执行
[root@docker1 scripts]# cat controller-manager_jing.sh
#!/bin/bash
# Copyright 2014 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
MASTER_ADDRESS=${1:-"172.16.199.17"}
cat <<EOF >/etc/kubernetes/kube-controller-manager
KUBE_LOGTOSTDERR="--logtostderr=false"
KUBE_LOG_DIR="--log-dir=/var/log/kubernetes"
KUBE_LOG_LEVEL="--v=4"
KUBE_MASTER="--master=${MASTER_ADDRESS}:8080"
KUBE_CLUSTER_CIDR="--cluster-cidr=10.1.0.0/16"
# --root-ca-file="": If set, this root certificate authority will be included in
# service account's token secret. This must be a valid PEM-encoded CA bundle.
#KUBE_CONTROLLER_MANAGER_ROOT_CA_FILE="--root-ca-file=/srv/kubernetes/ca.crt"
# --service-account-private-key-file="": Filename containing a PEM-encoded private
# RSA key used to sign service account tokens.
#KUBE_CONTROLLER_MANAGER_SERVICE_ACCOUNT_PRIVATE_KEY_FILE="--service-account-private-key-file=/srv/kubernetes/server.key"
# --leader-elect
KUBE_LEADER_ELECT="--leader-elect=false"
EOF
KUBE_CONTROLLER_MANAGER_OPTS=" \${KUBE_LOGTOSTDERR} \\
\${KUBE_LOG_DIR} \\
\${KUBE_LOG_LEVEL} \\
\${KUBE_MASTER} \\
\${KUBE_CLUSTER_CIDR}\\
\${KUBE_LEADER_ELECT}"
cat <<EOF >/usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=kube-apiserver.service
After=network.target
Requires=kube-apiserver.service
[Service]
EnvironmentFile=-/etc/kubernetes/kube-controller-manager
ExecStart=/usr/bin/kube-controller-manager ${KUBE_CONTROLLER_MANAGER_OPTS}
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager
scheduler
拷贝以下脚本并执行
#!/bin/bash
# Copyright 2014 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
MASTER_ADDRESS=${1:-"172.16.199.17"}
cat <<EOF >/etc/kubernetes/kube-scheduler
###
# kubernetes scheduler config
# --logtostderr=true: log to standard error instead of files
KUBE_LOGTOSTDERR="--logtostderr=false"
# Log to file
KUBE_LOG_DIR="--log-dir=/var/log/kubernetes"
# --v=0: log level for V logs
KUBE_LOG_LEVEL="--v=4"
KUBE_MASTER="--master=${MASTER_ADDRESS}:8080"
# --leader-elect
KUBE_LEADER_ELECT="--leader-elect=false"
# Add your own!
KUBE_SCHEDULER_ARGS=""
EOF
KUBE_SCHEDULER_OPTS=" \${KUBE_LOGTOSTDERR} \\
\${KUBE_LOG_DIR} \\
\${KUBE_LOG_LEVEL} \\
\${KUBE_MASTER} \\
\${KUBE_LEADER_ELECT} \\
\${KUBE_SCHEDULER_ARGS}"
cat <<EOF >/usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=kube-apiserver.service
Requires=kube-apiserver.service
[Service]
EnvironmentFile=-/etc/kubernetes/kube-scheduler
ExecStart=/usr/bin/kube-scheduler ${KUBE_SCHEDULER_OPTS}
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-scheduler
systemctl restart kube-scheduler
Node安装：
kubelet
拷贝以下脚本并执行，注意在每个node上修改对应node的IP. DNS地址为预留地址，dns服务还未安装。
#!/bin/bash
# Copyright 2014 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
MASTER_ADDRESS=${1:-"172.16.199.17"}
NODE_ADDRESS=${2:-"172.16.199.27"}
DNS_SERVER_IP=${3:-"169.169.0.53"}
DNS_DOMAIN=${4:-"cluster.local"}
mkdir -p /etc/kubernetes
cat <<EOF >/etc/kubernetes/kubelet
# --logtostderr=true: log to standard error instead of files
KUBE_LOGTOSTDERR="--logtostderr=false"
#LOG TO FILE
KUBE_LOG_DIR="--log-dir=/var/log/kubernetes"
# --v=0: log level for V logs
KUBE_LOG_LEVEL="--v=4"
# --address=0.0.0.0: The IP address for the Kubelet to serve on (set to 0.0.0.0 for all interfaces)
NODE_ADDRESS="--address=${NODE_ADDRESS}"
# --port=10250: The port for the Kubelet to serve on. Note that "kubectl logs" will not work if you set this flag.
NODE_PORT=""
# --hostname-override="": If non-empty, will use this string as identification instead of the actual hostname.
NODE_HOSTNAME="--hostname-override=${NODE_ADDRESS}"
# --api-servers=[]: List of Kubernetes API servers for publishing events,
# and reading pods and services. (ip:port), comma separated.
KUBELET_API_SERVER="--api-servers=${MASTER_ADDRESS}:8080"
# --allow-privileged=false: If true, allow containers to request privileged mode. [default=false]
KUBE_ALLOW_PRIV="--allow-privileged=false"
# DNS info
KUBELET__DNS_IP="--cluster-dns=${DNS_SERVER_IP}"
KUBELET_DNS_DOMAIN="--cluster-domain=${DNS_DOMAIN}"
# Add your own!
KUBELET_ARGS=""
EOF
KUBE_PROXY_OPTS=" \${KUBE_LOGTOSTDERR} \\
\${KUBE_LOG_DIR} \\
\${KUBE_LOG_LEVEL} \\
\${NODE_ADDRESS} \\
\${NODE_PORT} \\
\${NODE_HOSTNAME} \\
\${KUBELET_API_SERVER} \\
\${KUBE_ALLOW_PRIV} \\
\${KUBELET__DNS_IP} \\
\${KUBELET_DNS_DOMAIN} \\
\$KUBELET_ARGS"
cat <<EOF >/usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service
[Service]
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet ${KUBE_PROXY_OPTS}
Restart=on-failure
KillMode=process
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet
kube-proxy
拷贝以下脚本并执行，注意在每个node上修改对应node的IP
#!/bin/bash
# Copyright 2014 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
MASTER_ADDRESS=${1:-"172.16.199.17"}
NODE_ADDRESS=${2:-"172.16.199.27"}
cat <<EOF >/etc/kubernetes/kube-proxy
# --logtostderr=true: log to standard error instead of files
KUBE_LOGTOSTDERR="--logtostderr=false"
KUBE_LOG_DIR="--log-dir=/var/log/kubernetes"
# --v=0: log level for V logs
KUBE_LOG_LEVEL="--v=2"
# --hostname-override="": If non-empty, will use this string as identification instead of the actual hostname.
NODE_HOSTNAME="--hostname-override=${NODE_ADDRESS}"
# --master="": The address of the Kubernetes API server (overrides any value in kubeconfig)
KUBE_MASTER="--master=http://${MASTER_ADDRESS}:8080"
EOF
KUBE_PROXY_OPTS=" \${KUBE_LOGTOSTDERR} \\
\${KUBE_LOG_DIR} \\
\${KUBE_LOG_LEVEL} \\
\${NODE_HOSTNAME} \\
\${KUBE_MASTER}"
cat <<EOF >/usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=-/etc/kubernetes/kube-proxy
ExecStart=/usr/bin/kube-proxy ${KUBE_PROXY_OPTS}
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy
初步验证
在master上执行
[root@docker1 scripts]# kubectl get svc -o wide NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubernetes 169.169.0.1 <none> 443/TCP 17h <none>
[root@docker1 scripts]# kubectl get node -o wide NAME STATUS AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION 172.16.199.27 Ready 40m v1.6.7 <none> CentOS Linux 7 (Core) 3.10.0-514.21.2.el7.x86_64 172.16.199.37 Ready 28m v1.6.7 <none> CentOS Linux 7 (Core) 3.10.0-514.21.2.el7.x86_64
Array
文章评论