VitePressAppearance
三 集群部署
为简单上手体验功能，可以先利用kubeadm安装测试，生产环境建议二进制或者一些成熟的集群高可用安装方式，Kubeadm 是 K8S 官方提供的快速部署工具，它提供了 kubeadm init 以及 kubeadm join 这两个命令作为快速创建 kubernetes 集群的最佳实践，本章节说明了使用 kubeadm 来部署 K8S 集群的过程。
- 集群组织结构
| 项目 | 说明 |
|---|---|
| 集群规模 | Master、node1、node2 |
| 系统 | CentOS 7.3 |
| 网络规划 | POD：10.244.0.0/16、Service：10.96.0.0/12 |
3.1 一般部署
本小节的所有的操作，在所有的节点上进行
3.1.1 关闭 firewalld 和 selinux
setenforce 0
sed -i '/^SELINUX=/cSELINUX=disabled' /etc/selinux/config
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i '/^SELINUX=/cSELINUX=disabled' /etc/selinux/config
systemctl stop firewalld
systemctl disable firewalld
3.1.2 加载 ipvs 内核模块
- 安装 IPVS 模块
yum -y install ipvsadm ipset sysstat conntrack libseccomp
yum -y install ipvsadm ipset sysstat conntrack libseccomp
- 设置开机加载配置文件
cat >>/etc/modules-load.d/ipvs.conf<<EOF
ip_vs_dh
ip_vs_ftp
ip_vs
ip_vs_lblc
ip_vs_lblcr
ip_vs_lc
ip_vs_nq
ip_vs_pe_sip
ip_vs_rr
ip_vs_sed
ip_vs_sh
ip_vs_wlc
ip_vs_wrr
nf_conntrack_ipv4
EOF
cat >>/etc/modules-load.d/ipvs.conf<<EOF
ip_vs_dh
ip_vs_ftp
ip_vs
ip_vs_lblc
ip_vs_lblcr
ip_vs_lc
ip_vs_nq
ip_vs_pe_sip
ip_vs_rr
ip_vs_sed
ip_vs_sh
ip_vs_wlc
ip_vs_wrr
nf_conntrack_ipv4
EOF
- 设置开机加载 IPVS 模块
systemctl enable systemd-modules-load.service # 设置开机加载内核模块
lsmod | grep -e ip_vs -e nf_conntrack_ipv4 # 重启后检查 ipvs 模块是否加载
systemctl enable systemd-modules-load.service # 设置开机加载内核模块
lsmod | grep -e ip_vs -e nf_conntrack_ipv4 # 重启后检查 ipvs 模块是否加载
- 如果集群已经部署在了 iptables 模式下，可以通过下面命令修改，修改 mode 为 ipvs 重启集群即可。
kubectl edit -n kube-system configmap kube-proxy
kubectl edit -n kube-system configmap kube-proxy
3.1.3 下载 Docker 和 K8S
- 设置 docker 源
curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
- 设置 k8s 源
cat >>/etc/yum.repos.d/kuberetes.repo<<EOF
[kuberneres]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
enabled=1
EOF
cat >>/etc/yum.repos.d/kuberetes.repo<<EOF
[kuberneres]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
enabled=1
EOF
- 安装 docker-ce 和 kubernetes
yum install docker-ce kubelet kubectl kubeadm -y
yum install docker-ce kubelet kubectl kubeadm -y
systemctl start docker
systemctl enable docker
systemctl enable kubelet
systemctl start docker
systemctl enable docker
systemctl enable kubelet
3.1.4 设置内核及 K8S 参数
- 设置内核参数
cat >>/etc/sysctl.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
cat >>/etc/sysctl.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
- 设置 kubelet 忽略 swap，使用 ipvs
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
KUBE_PROXY_MODE=ipvs
EOF
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
KUBE_PROXY_MODE=ipvs
EOF
3.2 部署 Master
本小节的所有的操作，只在 Master 节点上进行
3.2.1 提前拉取镜像
宿主机最好能访问国外资源，在kubeadm init 在初始化的时候会到谷歌的 docker hub 拉取镜像，如果宿主机测试无法访问 k8s.gcr.io 可以在服务器所以我们要提前部署好代理软件，本例中监听个本机 9666 进行部署。
如果条件不允许可以参考: https://blog.csdn.net/jinguangliu/article/details/82792617 来解决镜像问题。
- 配置 Docker 拉取镜像时候的代理地址，vim /usr/lib/systemd/system/docker.service。
[Service]
Environment="HTTPS_PROXY=127.0.0.1:9666"
Environment="NO_PROXY=127.0.0.0/8,172.16.0.0/16"
[Service]
Environment="HTTPS_PROXY=127.0.0.1:9666"
Environment="NO_PROXY=127.0.0.0/8,172.16.0.0/16"
- 提前拉取初始化需要的镜像
kubeadm config images pull
kubeadm config images pull
- 使用其他源镜像
docker pull mirrorgooglecontainers/kube-apiserver:v1.14.2
docker pull mirrorgooglecontainers/kube-controller-manager:v1.14.2
docker pull mirrorgooglecontainers/kube-scheduler:v1.14.2
docker pull mirrorgooglecontainers/kube-proxy:v1.14.2
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd:3.3.10
docker pull coredns/coredns:1.3.1
利用`kubeadm config images list` 查看需要的docker image name
k8s.gcr.io/kube-apiserver:v1.14.2
k8s.gcr.io/kube-controller-manager:v1.14.2
k8s.gcr.io/kube-scheduler:v1.14.2
k8s.gcr.io/kube-proxy:v1.14.2
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1
# 修改tag
docker tag docker.io/mirrorgooglecontainers/kube-apiserver:v1.14.2 k8s.gcr.io/kube-apiserver:v1.14.2
docker tag docker.io/mirrorgooglecontainers/kube-scheduler:v1.14.2 k8s.gcr.io/kube-scheduler:v1.14.2
docker tag docker.io/mirrorgooglecontainers/kube-proxy:v1.14.2 k8s.gcr.io/kube-proxy:v1.14.2
docker tag docker.io/mirrorgooglecontainers/kube-controller-manager:v1.14.2 k8s.gcr.io/kube-controller-manager:v1.14.2
docker tag docker.io/mirrorgooglecontainers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag docker.io/mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag docker.io/coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
docker rmi `docker images |grep docker.io/ |awk '{print $1":"$2}'`
docker pull mirrorgooglecontainers/kube-apiserver:v1.14.2
docker pull mirrorgooglecontainers/kube-controller-manager:v1.14.2
docker pull mirrorgooglecontainers/kube-scheduler:v1.14.2
docker pull mirrorgooglecontainers/kube-proxy:v1.14.2
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd:3.3.10
docker pull coredns/coredns:1.3.1
利用`kubeadm config images list` 查看需要的docker image name
k8s.gcr.io/kube-apiserver:v1.14.2
k8s.gcr.io/kube-controller-manager:v1.14.2
k8s.gcr.io/kube-scheduler:v1.14.2
k8s.gcr.io/kube-proxy:v1.14.2
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1
# 修改tag
docker tag docker.io/mirrorgooglecontainers/kube-apiserver:v1.14.2 k8s.gcr.io/kube-apiserver:v1.14.2
docker tag docker.io/mirrorgooglecontainers/kube-scheduler:v1.14.2 k8s.gcr.io/kube-scheduler:v1.14.2
docker tag docker.io/mirrorgooglecontainers/kube-proxy:v1.14.2 k8s.gcr.io/kube-proxy:v1.14.2
docker tag docker.io/mirrorgooglecontainers/kube-controller-manager:v1.14.2 k8s.gcr.io/kube-controller-manager:v1.14.2
docker tag docker.io/mirrorgooglecontainers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag docker.io/mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag docker.io/coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
docker rmi `docker images |grep docker.io/ |awk '{print $1":"$2}'`
3.2.2 初始化Master
- 使用 kubeadm 初始化 k8s 集群
kubeadm init --kubernetes-version=v1.14.0 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
kubeadm init --kubernetes-version=v1.14.0 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
- 如果有报错使用下面命令查看
journalctl -xeu kubelet
journalctl -xeu kubelet
- 如果初始化过程被中断可以使用下面命令来恢复
kubeadm reset
kubeadm reset
- 下面是最后执行成功显示的结果，需要保存这个执行结果，以让 node 节点加入集群
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.100.9:6443 --token 2dyd69.hrfsjkkxs4stim7n \
--discovery-token-ca-cert-hash sha256:4e30c1f41aefb177b708a404ccb7e818e31647c7dbdd2d42f6c5c9894b6f41e7
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.16.100.9:6443 --token 2dyd69.hrfsjkkxs4stim7n \
--discovery-token-ca-cert-hash sha256:4e30c1f41aefb177b708a404ccb7e818e31647c7dbdd2d42f6c5c9894b6f41e7
- 最好以普通用户的身份运行下面的命令
# 在当前用户家目录下创建.kube目录并配置访问集群的config 文件
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 在当前用户家目录下创建.kube目录并配置访问集群的config 文件
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 部署 flannel 网络插件
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
- 查看 kube-system 命名空间中运行的 pods
kubectl get pods -n kube-system
kubectl get pods -n kube-system
- 查看 k8s 集群组件的状态
kubectl get ComponentStatus
kubectl get ComponentStatus
- 配置命令补全
yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
3.3 部署 Node
本小节的所有的操作，只在 Node 节点上进行。
3.3.1 加入集群
- 加入集群，注意在命令尾部加上 --ignore-preflight-errors=Swap ，以忽略 k8s 对主机 swap 的检查（k8s为了性能所以要求进制 swap ）
kubeadm join 172.16.100.9:6443 --token 2dyd69.hrfsjkkxs4stim7n \
--discovery-token-ca-cert-hash sha256:4e30c1f41aefb177b708a404ccb7e818e31647c7dbdd2d42f6c5c9894b6f41e7 --ignore-preflight-errors=Swap
kubeadm join 172.16.100.9:6443 --token 2dyd69.hrfsjkkxs4stim7n \
--discovery-token-ca-cert-hash sha256:4e30c1f41aefb177b708a404ccb7e818e31647c7dbdd2d42f6c5c9894b6f41e7 --ignore-preflight-errors=Swap
- 返回结果，表示加入集群成功
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
3.3.2 查看进度
当 node 节点加入 K8S 集群中后，Master 会调度到 Node 节点上一些组件，用于处理集群事务，这些组件没有下载完成之前 Node 节点在集群中还是未就绪状态
- 在 node 执行下面命令，可以查看镜像的下载进度，下面是最终结果显示
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.14.0 5cd54e388aba 6 weeks ago 82.1MB
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 3 months ago 52.6MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 16 months ago 742kB
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.14.0 5cd54e388aba 6 weeks ago 82.1MB
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 3 months ago 52.6MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 16 months ago 742kB
- 可以在 Master 上使用下面命令来查看新加入的节点状态
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 3d21h v1.14.1
node1 Ready <none> 3d21h v1.14.1
node2 Ready <none> 3d21h v1.14.1
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 3d21h v1.14.1
node1 Ready <none> 3d21h v1.14.1
node2 Ready <none> 3d21h v1.14.1
- 查看集群状态
[root@master ~]\# kubectl cluster-info
Kubernetes master is running at https://10.234.2.204:6443
KubeDNS is running at https://10.234.2.204:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://10.234.2.204:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@master ~]\# kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
[root@master ~]\# kubectl cluster-info
Kubernetes master is running at https://10.234.2.204:6443
KubeDNS is running at https://10.234.2.204:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://10.234.2.204:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@master ~]\# kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
如果嫌网络pull镜像慢可以在一台上面将镜像打包发送至其他node节点
拷贝到node节点
for i in /tmp/*.tar; do scp -i $i root@172.16.0.15:/root/;done
node节点还原
for i in *.tar ;do docker load -i $i;done
拷贝到node节点
for i in /tmp/*.tar; do scp -i $i root@172.16.0.15:/root/;done
node节点还原
for i in *.tar ;do docker load -i $i;done
- 查看 kube-system 这个 k8s 命名空间中有哪些组件，分别运行在哪个节点，-o wide 是以详细方式显示。
$ kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-fb8b8dccf-cp24r 1/1 Running 0 26m 10.244.0.2 i-xeahpl98 <none> <none>
coredns-fb8b8dccf-ljswp 1/1 Running 0 26m 10.244.0.3 i-xeahpl98 <none> <none>
etcd-i-xeahpl98 1/1 Running 0 25m 172.16.100.9 i-xeahpl98 <none> <none>
kube-apiserver-i-xeahpl98 1/1 Running 0 25m 172.16.100.9 i-xeahpl98 <none> <none>
kube-controller-manager-i-xeahpl98 1/1 Running 0 25m 172.16.100.9 i-xeahpl98 <none> <none>
kube-flannel-ds-amd64-crft8 1/1 Running 3 16m 172.16.100.6 i-me87b6gw <none> <none>
kube-flannel-ds-amd64-nckw4 1/1 Running 0 6m41s 172.16.100.10 i-qhcc2owe <none> <none>
kube-flannel-ds-amd64-zb7sg 1/1 Running 0 23m 172.16.100.9 i-xeahpl98 <none> <none>
kube-proxy-7kjkf 1/1 Running 0 6m41s 172.16.100.10 i-qhcc2owe <none> <none>
kube-proxy-c5xs2 1/1 Running 2 16m 172.16.100.6 i-me87b6gw <none> <none>
kube-proxy-rdzq2 1/1 Running 0 26m 172.16.100.9 i-xeahpl98 <none> <none>
kube-scheduler-i-xeahpl98 1/1 Running 0 25m 172.16.100.9 i-xeahpl98 <none> <none>
$ kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-fb8b8dccf-cp24r 1/1 Running 0 26m 10.244.0.2 i-xeahpl98 <none> <none>
coredns-fb8b8dccf-ljswp 1/1 Running 0 26m 10.244.0.3 i-xeahpl98 <none> <none>
etcd-i-xeahpl98 1/1 Running 0 25m 172.16.100.9 i-xeahpl98 <none> <none>
kube-apiserver-i-xeahpl98 1/1 Running 0 25m 172.16.100.9 i-xeahpl98 <none> <none>
kube-controller-manager-i-xeahpl98 1/1 Running 0 25m 172.16.100.9 i-xeahpl98 <none> <none>
kube-flannel-ds-amd64-crft8 1/1 Running 3 16m 172.16.100.6 i-me87b6gw <none> <none>
kube-flannel-ds-amd64-nckw4 1/1 Running 0 6m41s 172.16.100.10 i-qhcc2owe <none> <none>
kube-flannel-ds-amd64-zb7sg 1/1 Running 0 23m 172.16.100.9 i-xeahpl98 <none> <none>
kube-proxy-7kjkf 1/1 Running 0 6m41s 172.16.100.10 i-qhcc2owe <none> <none>
kube-proxy-c5xs2 1/1 Running 2 16m 172.16.100.6 i-me87b6gw <none> <none>
kube-proxy-rdzq2 1/1 Running 0 26m 172.16.100.9 i-xeahpl98 <none> <none>
kube-scheduler-i-xeahpl98 1/1 Running 0 25m 172.16.100.9 i-xeahpl98 <none> <none>
3.3.3 镜像下载太慢
node 节点需要翻墙下载镜像太慢，建议使用 docker 镜像的导入导出功能 先将master的三个镜像打包发送到node节点，load后再jion
- 导出
docker image save -o /tmp/kube-proxy.tar k8s.gcr.io/kube-proxy
docker image save -o /tmp/flannel.tar quay.io/coreos/flannel
docker image save -o /tmp/pause.tar k8s.gcr.io/pause
docker image save -o /tmp/kube-proxy.tar k8s.gcr.io/kube-proxy
docker image save -o /tmp/flannel.tar quay.io/coreos/flannel
docker image save -o /tmp/pause.tar k8s.gcr.io/pause
- 导入
docker image load -i /tmp/kube-proxy.tar
docker image load -i /tmp/pause.tar
docker image load -i /tmp/flannel.tar
docker image load -i /tmp/kube-proxy.tar
docker image load -i /tmp/pause.tar
docker image load -i /tmp/flannel.tar
3.4 jio本部署
3.4.1 预先准备
环境：
| 角色 | IP |
|---|---|
| master | 172.16.1.129 |
| node01 | 172.16.1.133 |
修改当前的主机名，确保不是 localhost：
hostnamectl set-hostname master
hostnamectl set-hostname master
为各节点添加主机名解析， 编辑/etc/hosts：
...
172.16.1 master master.agou-ops.com
172.16.1 node01 node01.aogu-ops.com
...
172.16.1 master master.agou-ops.com
172.16.1 node01 node01.aogu-ops.com
确保任意节点上
Kubelet使用的 IP 地址可互通（无需 NAT 映射即可相互访问），且没有防火墙、安全组隔离
关闭防火墙和SElinux：
sed -i 's/SELINUX=*/SELINUX=disabled/' /etc/selinux/config
# getenforce 确定当前SElinux状态
systemctl disable firewalld && systemctl stop firewalld
sed -i 's/SELINUX=*/SELINUX=disabled/' /etc/selinux/config
# getenforce 确定当前SElinux状态
systemctl disable firewalld && systemctl stop firewalld
安装 docker 以及 kubelet ：
# --------- 在 master 和 node 节点执行 -----------
# 使用阿里云 docker 镜像加速，可以自行去控制台申请，在这里我使用公共加速镜像
export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
# 编辑脚本快速安装，内容如下：
#!/bin/bash
# 在 master 节点和 worker 节点都要执行
# 安装 docker
# 参考文档如下
# https://docs.docker.com/install/linux/docker-ce/centos/
# https://docs.docker.com/install/linux/linux-postinstall/
# 卸载旧版本
yum remove -y docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
# 设置 yum repository
yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安装并启动 docker
yum install -y docker-ce-19.03.8 docker-ce-cli-19.03.8 containerd.io
systemctl enable docker
systemctl start docker
# 安装 nfs-utils
# 必须先安装 nfs-utils 才能挂载 nfs 网络存储
yum install -y nfs-utils
yum install -y wget
# 关闭 swap ××重要××
swapoff -a && echo "vm.swappiness=0" >> /etc/sysctl.conf && sysctl -p && free –h
# 修改 /etc/sysctl.conf
# 如果有配置，则修改
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g" /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g" /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.disable_ipv6.*#net.ipv6.conf.all.disable_ipv6=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.default.disable_ipv6.*#net.ipv6.conf.default.disable_ipv6=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.lo.disable_ipv6.*#net.ipv6.conf.lo.disable_ipv6=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.forwarding.*#net.ipv6.conf.all.forwarding=1#g" /etc/sysctl.conf
# 可能没有，追加
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding = 1" >> /etc/sysctl.conf
# 执行命令以应用
sysctl -p
# 配置K8S的yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 卸载旧版本
yum remove -y kubelet kubeadm kubectl
# 安装kubelet、kubeadm、kubectl
# 将 ${1} 替换为 kubernetes 版本号，例如 1.17.2
yum install -y kubelet-${1} kubeadm-${1} kubectl-${1}
# 修改docker Cgroup Driver为systemd
# # 将/usr/lib/systemd/system/docker.service文件中的这一行 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
# # 修改为 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
# 如果不修改，在添加 worker 节点时可能会碰到如下错误
# [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd".
# Please follow the guide at https://kubernetes.io/docs/setup/cri/
sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service
# 设置 docker 镜像，提高 docker 镜像下载速度和稳定性
# 如果您访问 https://hub.docker.io 速度非常稳定，亦可以跳过这个步骤
curl -sSL https://kuboard.cn/install-script/set_mirror.sh | sh -s ${REGISTRY_MIRROR}
# 重启 docker，并启动 kubelet
systemctl daemon-reload
systemctl restart docker
systemctl enable kubelet && systemctl start kubelet
docker version
# --------- 在 master 和 node 节点执行 -----------
# 使用阿里云 docker 镜像加速，可以自行去控制台申请，在这里我使用公共加速镜像
export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
# 编辑脚本快速安装，内容如下：
#!/bin/bash
# 在 master 节点和 worker 节点都要执行
# 安装 docker
# 参考文档如下
# https://docs.docker.com/install/linux/docker-ce/centos/
# https://docs.docker.com/install/linux/linux-postinstall/
# 卸载旧版本
yum remove -y docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
# 设置 yum repository
yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安装并启动 docker
yum install -y docker-ce-19.03.8 docker-ce-cli-19.03.8 containerd.io
systemctl enable docker
systemctl start docker
# 安装 nfs-utils
# 必须先安装 nfs-utils 才能挂载 nfs 网络存储
yum install -y nfs-utils
yum install -y wget
# 关闭 swap ××重要××
swapoff -a && echo "vm.swappiness=0" >> /etc/sysctl.conf && sysctl -p && free –h
# 修改 /etc/sysctl.conf
# 如果有配置，则修改
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g" /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g" /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.disable_ipv6.*#net.ipv6.conf.all.disable_ipv6=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.default.disable_ipv6.*#net.ipv6.conf.default.disable_ipv6=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.lo.disable_ipv6.*#net.ipv6.conf.lo.disable_ipv6=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.forwarding.*#net.ipv6.conf.all.forwarding=1#g" /etc/sysctl.conf
# 可能没有，追加
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding = 1" >> /etc/sysctl.conf
# 执行命令以应用
sysctl -p
# 配置K8S的yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 卸载旧版本
yum remove -y kubelet kubeadm kubectl
# 安装kubelet、kubeadm、kubectl
# 将 ${1} 替换为 kubernetes 版本号，例如 1.17.2
yum install -y kubelet-${1} kubeadm-${1} kubectl-${1}
# 修改docker Cgroup Driver为systemd
# # 将/usr/lib/systemd/system/docker.service文件中的这一行 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
# # 修改为 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
# 如果不修改，在添加 worker 节点时可能会碰到如下错误
# [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd".
# Please follow the guide at https://kubernetes.io/docs/setup/cri/
sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service
# 设置 docker 镜像，提高 docker 镜像下载速度和稳定性
# 如果您访问 https://hub.docker.io 速度非常稳定，亦可以跳过这个步骤
curl -sSL https://kuboard.cn/install-script/set_mirror.sh | sh -s ${REGISTRY_MIRROR}
# 重启 docker，并启动 kubelet
systemctl daemon-reload
systemctl restart docker
systemctl enable kubelet && systemctl start kubelet
docker version
⚠️ WARNING
如果此时执行
service status kubelet命令，将得到 kubelet 启动失败的错误提示，请忽略此错误，因为必须完成后续步骤中kubeadm init的操作，kubelet 才能正常启动
赋予权限及运行：
chmod +x preinstall.sh
./preinstall.sh 1.18.2
chmod +x preinstall.sh
./preinstall.sh 1.18.2
3.4.2 初始化 master 节点
关于初始化时用到的环境变量
APISERVER_NAME不能是 master 的 hostnameAPISERVER_NAME必须全为小写字母、数字、小数点，不能包含减号POD_SUBNET所使用的网段不能与 master节点/worker节点 所在的网段重叠。该字段的取值为一个 CIDR 值，如果您对 CIDR 这个概念还不熟悉，请仍然执行export POD_SUBNET=10.100.0.1/16命令，不做修改
在 master 节点上运行：
# 替换 x.x.x.x 为 master 节点的内网IP
# export 命令只在当前 shell 会话中有效，开启新的 shell 窗口后，如果要继续安装过程，请重新执行此处的 export 命令
export MASTER_IP=172.16.1.129
# 替换 apiserver.demo 为 您想要的 dnsName
export APISERVER_NAME=apiserver.agou.com
# Kubernetes 容器组所在的网段，该网段安装完成后，由 kubernetes 创建，事先并不存在于您的物理网络中
export POD_SUBNET=10.100.0.1/16
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
# 替换 x.x.x.x 为 master 节点的内网IP
# export 命令只在当前 shell 会话中有效，开启新的 shell 窗口后，如果要继续安装过程，请重新执行此处的 export 命令
export MASTER_IP=172.16.1.129
# 替换 apiserver.demo 为 您想要的 dnsName
export APISERVER_NAME=apiserver.agou.com
# Kubernetes 容器组所在的网段，该网段安装完成后，由 kubernetes 创建，事先并不存在于您的物理网络中
export POD_SUBNET=10.100.0.1/16
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
在master节点上添加以下脚本：
#!/bin/bash
# 只在 master 节点执行
# 脚本出错时终止执行
set -e
if [ ${#POD_SUBNET} -eq 0 ] || [ ${#APISERVER_NAME} -eq 0 ]; then
echo -e "\033[31;1m请确保您已经设置了环境变量 POD_SUBNET 和 APISERVER_NAME \033[0m"
echo 当前POD_SUBNET=$POD_SUBNET
echo 当前APISERVER_NAME=$APISERVER_NAME
exit 1
fi
# 查看完整配置选项 https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2
rm -f ./kubeadm-config.yaml
cat <<EOF > ./kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v${1}
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
controlPlaneEndpoint: "${APISERVER_NAME}:6443"
networking:
serviceSubnet: "10.96.0.0/16"
podSubnet: "${POD_SUBNET}"
dnsDomain: "cluster.local"
EOF
# kubeadm init
# 根据您服务器网速的情况，您需要等候 3 - 10 分钟
kubeadm init --config=kubeadm-config.yaml --upload-certs
# 配置 kubectl
rm -rf /root/.kube/
mkdir /root/.kube/
cp -i /etc/kubernetes/admin.conf /root/.kube/config
# 安装 calico 网络插件
# 参考文档 https://docs.projectcalico.org/v3.13/getting-started/kubernetes/self-managed-onprem/onpremises
echo "安装calico-3.13.1"
rm -f calico-3.13.1.yaml
wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml
kubectl apply -f calico-3.13.1.yaml
#!/bin/bash
# 只在 master 节点执行
# 脚本出错时终止执行
set -e
if [ ${#POD_SUBNET} -eq 0 ] || [ ${#APISERVER_NAME} -eq 0 ]; then
echo -e "\033[31;1m请确保您已经设置了环境变量 POD_SUBNET 和 APISERVER_NAME \033[0m"
echo 当前POD_SUBNET=$POD_SUBNET
echo 当前APISERVER_NAME=$APISERVER_NAME
exit 1
fi
# 查看完整配置选项 https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2
rm -f ./kubeadm-config.yaml
cat <<EOF > ./kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v${1}
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
controlPlaneEndpoint: "${APISERVER_NAME}:6443"
networking:
serviceSubnet: "10.96.0.0/16"
podSubnet: "${POD_SUBNET}"
dnsDomain: "cluster.local"
EOF
# kubeadm init
# 根据您服务器网速的情况，您需要等候 3 - 10 分钟
kubeadm init --config=kubeadm-config.yaml --upload-certs
# 配置 kubectl
rm -rf /root/.kube/
mkdir /root/.kube/
cp -i /etc/kubernetes/admin.conf /root/.kube/config
# 安装 calico 网络插件
# 参考文档 https://docs.projectcalico.org/v3.13/getting-started/kubernetes/self-managed-onprem/onpremises
echo "安装calico-3.13.1"
rm -f calico-3.13.1.yaml
wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml
kubectl apply -f calico-3.13.1.yaml
执行该脚本：
bash master_init.sh 1.18.2
bash master_init.sh 1.18.2
检查master初始化结果：
# 执行如下命令，等待 3-10 分钟，直到所有的容器组处于 Running 状态
watch kubectl get pod -n kube-system -o wide
# 查看 master 节点初始化结果
kubectl get nodes -o wide
# 执行如下命令，等待 3-10 分钟，直到所有的容器组处于 Running 状态
watch kubectl get pod -n kube-system -o wide
# 查看 master 节点初始化结果
kubectl get nodes -o wide
3.4.3 初始化 node 节点
3.4.3.1 获得 join命令参数
在master 节点上获取join命令行：
kubeadm token create --print-join-command
kubeadm token create --print-join-command
ℹ️ 有效时间
该
token的有效时间为 2 个小时，2小时内，您可以使用此 token 初始化任意数量的 worker 节点。
3.4.3.2 初始化 node 节点
编辑本地主机解析文件/etc/hosts：
# 添加以下内容
172.16.1.129 apiserver.agou.com
# 添加以下内容
172.16.1.129 apiserver.agou.com
执行从master节点上获取的join命令：
kubeadm join apiserver.agou.com:6443 --token 7t8xh1.2y6pzjpgl9eai0kd --discovery-token-ca-cert-hash sha256:a57479ae585f0c0a617d890c62c15c71023d85d4d55f4dd8fffeb56d9a467ef7
kubeadm join apiserver.agou.com:6443 --token 7t8xh1.2y6pzjpgl9eai0kd --discovery-token-ca-cert-hash sha256:a57479ae585f0c0a617d890c62c15c71023d85d4d55f4dd8fffeb56d9a467ef7
3.4.4 检查初始化结果
在master节点上执行：
kubectl get nodes -o wide
kubectl get nodes -o wide
3.5 使用 Kubespray 部署
3.5.1 Kubespray 简介
Kubespray 是 Kubernetes incubator 中的项目，目标是提供 Production Ready Kubernetes 部署方案，该项目基础是通过 Ansible Playbook 来定义系统与 Kubernetes 集群部署的任务，具有以下几个特点：
- 可以部署在 AWS, GCE, Azure, OpenStack 以及裸机上.
- 部署 High Available Kubernetes 集群.
- 可组合性 (Composable)，可自行选择 Network Plugin (flannel, calico, canal, weave) 来部署.
- 支持多种 Linux distributions(CoreOS, Debian Jessie, Ubuntu 16.04, CentOS/RHEL7).
Kubespray 由一系列的 Ansible playbook、生成 inventory 的命令行工具以及生成 OS/Kubernetes 集群配置管理任务的专业知识构成。
3.5.2 初始化环境
主机环境：
| 角色 | IP |
|---|---|
| master | 172.16.1.128 |
| node01 | 172.16.1.129 |
编辑/etc/hosts文件，使各主机之间可以通过主机名互相通信：
# 添加以下内容
172.16.1.128 master master.agou-ops.com
172.16.1.129 node01 node01.agou-ops.com
# 添加以下内容
172.16.1.128 master master.agou-ops.com
172.16.1.129 node01 node01.agou-ops.com
关闭 SELinux 和防火墙：
sed -i 's/SELINUX=*/SELINUX=disabled/' /etc/selinux/config
systemctl disable firewalld && systemctl stop firewalld
sed -i 's/SELINUX=*/SELINUX=disabled/' /etc/selinux/config
systemctl disable firewalld && systemctl stop firewalld
Kubernetes 1.8 开始要求关闭系统的 Swap 交换分区，方法如下：
swapoff -a && echo "vm.swappiness=0" >> /etc/sysctl.conf && sysctl -p && free –h
swapoff -a && echo "vm.swappiness=0" >> /etc/sysctl.conf && sysctl -p && free –h
Docker 从 1.13 版本开始调整了默认的防火墙规则，禁用了 iptables filter 表中FOWARD链，这样会引起 Kubernetes 集群中跨 Node 的 Pod 无法通信，在各个 Docker 节点执行下面的命令：
iptables -P FORWARD ACCEPT
iptables -P FORWARD ACCEPT
配置 SSH Key 认证。确保本机也可以 SSH 连接，否则下面部署失败。
ssh-keygen -t rsa -N ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh-keygen -t rsa -N ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
更新系统内核为 4.4.x , CentOS 默认为 3.10.x 。
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install -y kernel-lt kernel-lt-devel
grub2-set-default 0
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install -y kernel-lt kernel-lt-devel
grub2-set-default 0
重启系统：reboot
增加内核配置，编辑/etc/sysctl.conf文件：
# 增加以下内容
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
# 增加以下内容
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
使其内核配置生效：
sysctl -p
sysctl -p
3.5.3 安装 Kubespray
安装 Centos 的 EPEL 源：
yum -y install epel-release
yum -y install epel-release
更新缓存：
yum clean all && yum makecache
yum clean all && yum makecache
安装相关软件（Ansible 版本必须 >= 2.7）：
yum install -y python-pip python3 python-netaddr python3-pip ansible git
yum install -y python-pip python3 python-netaddr python3-pip ansible git
下载源码，当前 Kubespray 项目的 Master 分支默认安装 K8s 1.13.1 版本：
git clone https://github.com/kubernetes-sigs/kubespray.git
git clone https://github.com/kubernetes-sigs/kubespray.git
安装 Kubespray 依赖，若无特殊说明，后续操作均在 ~/kubespray `目录下执行：
cd kubespray
pip3 install -r requirements.txt
cd kubespray
pip3 install -r requirements.txt
配置 Kubespray：
cp -rfp inventory/sample inventory/mycluster
# Update Ansible inventory file with inventory builder
declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
cp -rfp inventory/sample inventory/mycluster
# Update Ansible inventory file with inventory builder
declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
修改配置文件 inventory/mycluster/hosts.ini ：
[all]
master ansible_host=master ip=172.16.1.128
[kube-master]
master
[etcd]
master
[kube-node]
node01
[k8s-cluster:children]
kube-master
kube-node
[calico-rr]
[all]
master ansible_host=master ip=172.16.1.128
[kube-master]
master
[etcd]
master
[kube-node]
node01
[k8s-cluster:children]
kube-master
kube-node
[calico-rr]
ℹ️此处也可以用：kubespray prepare --masters master1 --etcds master1 --nodes node1 node2 node3来自动生成inventory文件
修改配置文件 inventory/mycluster/group_vars/all/all.yml：
# 修改如下配置:
loadbalancer_apiserver_localhost: true
# 加载内核模块，否则 ceph, gfs 等无法挂载客户端
kubelet_load_modules: true
# 修改如下配置:
loadbalancer_apiserver_localhost: true
# 加载内核模块，否则 ceph, gfs 等无法挂载客户端
kubelet_load_modules: true
修改镜像默认的 Repo 地址，使用 Calico 三层网络，同时可以指定安装的 K8s版本，参数为 kube_version。编辑文件inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml(在这里我能够科学上网就不做修改了)：
kube_image_repo: "gcr.io/google-containers"
kube_image_repo: "gcr.io/google-containers"
如需设置代理，在 cluster.yml中编辑 default 值即可：
...
proxy_env:
http_proxy: "【【 http_proxy | default ('192.168.43.37:8888') 】】"
HTTP_PROXY: "【【 http_proxy | default ('192.168.43.37:8888') 】】"
https_proxy: "【【 https_proxy | default ('192.168.43.37:8888') 】】"
HTTPS_PROXY: "【【 https_proxy | default ('192.168.43.37:8888') 】】"
no_proxy: "【【 no_proxy | default ('') 】】"
NO_PROXY: "【【 no_proxy | default ('') 】】"
no_log: true
...
...
proxy_env:
http_proxy: "【【 http_proxy | default ('192.168.43.37:8888') 】】"
HTTP_PROXY: "【【 http_proxy | default ('192.168.43.37:8888') 】】"
https_proxy: "【【 https_proxy | default ('192.168.43.37:8888') 】】"
HTTPS_PROXY: "【【 https_proxy | default ('192.168.43.37:8888') 】】"
no_proxy: "【【 no_proxy | default ('') 】】"
NO_PROXY: "【【 no_proxy | default ('') 】】"
no_log: true
...
如需设置docker pull 代理，新建/etc/systemd/system/docker.service.d/http-proxy.conf文件，添加以下内容：
[Service]
Environment="http_proxy=192.168.43.37:8888"
Environment="https_proxy=192.168.43.37:8888"
Environment="NO_PROXY= hostname.example.com,172.10.10.10"
# 最后重启docker
systemctl daemon-reload
systemctl restart docker
[Service]
Environment="http_proxy=192.168.43.37:8888"
Environment="https_proxy=192.168.43.37:8888"
Environment="NO_PROXY= hostname.example.com,172.10.10.10"
# 最后重启docker
systemctl daemon-reload
systemctl restart docker
修改./roles/kubernetes-apps/ansible/templates/dashboard.yml.j2文件，使用 NodePort 方式访问 Dashboard：
# ------------------- Dashboard Service ------------------- #…………
targetPort: 8443
type: NodePort //添加这一行
selector:
k8s-app: kubernetes-dashboard
# ------------------- Dashboard Service ------------------- #…………
targetPort: 8443
type: NodePort //添加这一行
selector:
k8s-app: kubernetes-dashboard
⚠️ 注意：如果是单节点部署 K8s，Kubespray 默认会创建 2 个 coredns Pod，但 Deployment 中又用到了 podAntiAffinity，因此会导致其中一个 coredns pod pending，所以需要修改./roles/kubernetes-apps/ansible/templates/coredns-deployment.yml.j2代码如下：
# 注释掉以下几行代码
affinity:
#podAntiAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# - topologyKey: "kubernetes.io/hostname"
# labelSelector:
# matchLabels:
# k8s-app: coredns【【 coredns_ordinal_suffix | default('') 】】
# 或者在spec一行添加代码：
spec:
replicas: 1 //指定pod为1个副本
# 注释掉以下几行代码
affinity:
#podAntiAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# - topologyKey: "kubernetes.io/hostname"
# labelSelector:
# matchLabels:
# k8s-app: coredns【【 coredns_ordinal_suffix | default('') 】】
# 或者在spec一行添加代码：
spec:
replicas: 1 //指定pod为1个副本
最后执行部署命令：
ansible-playbook -i inventory/mycluster/hosts.ini --become --become-user=root cluster.yml -b -v
ansible-playbook -i inventory/mycluster/hosts.ini --become --become-user=root cluster.yml -b -v
3.5.4 登录 Dashboard
登陆 Dashboard 支持 kubeconfig 和 token 两种认证方式，kubeconfig 也依赖 token 字段，所以生成 token 这一步是必不可少的。此处，我们获取集群管理员（拥有所有命名空间的 admin 权限）的 token。
查看 kubernetes-dashboard 暴露的端口，如下所示，这里是31777端口。
[root@master kubespray]\# kubectl get svc --all-namespaces | grep kubernetes-dashboard
kube-system kubernetes-dashboard NodePort 10.233.41.202 <none> 443:30548/TCP 8m16s
[root@master kubespray]\# kubectl get svc --all-namespaces | grep kubernetes-dashboard
kube-system kubernetes-dashboard NodePort 10.233.41.202 <none> 443:30548/TCP 8m16s
获取 admin 的 token
[root@master kubespray]\# kubectl -n kube-system describe $(kubectl -n kube-system get secret -n kube-system -o name | grep namespace) | grep token
Name: namespace-controller-token-ksdvp
Type: kubernetes.io/service-account-token
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkV0N0pLMXVqMzNxS2xCNXRlTkxpRTlOYnNMVzRiajNrLU9kdi1qRW5jTDQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJuYW1lc3BhY2UtY29udHJvbGxlci10b2tlbi1rc2R2cCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJuYW1lc3BhY2UtY29udHJvbGxlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImZiM2NkMDBhLTE4ZjAtNGI4OC1hZGNiLWYwNGVjZWNlZTUzNiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpuYW1lc3BhY2UtY29udHJvbGxlciJ9.bYWcoopcYFC6EYNMNPkoiZeUQqfidC9NwlrMzzkZD3T9e-PbDd37pmTeWAcU_E4DCDzeVc9CXfWPhWCfr3syZKWiIXPNtDrNrIgnGs34Id2evsh7evVTgOjQtWkRoqX9UFdjWZdQPxvJChLZacRqbUp718umCzhR9evuE0zq8JeruBCTrcilQDDYobavfYs72HrwZ5xlIj2GMb66FeS7mYZacP-2-M3oVsziIWLs_kfBIaN_OkpImUPpvJxF-8xMmVP2BCKWyHWaLPIUdVsF8FkiLWH7bIS8f0cm8D4wEcMZ4IYkVe2FMcmMaiFJx5HEXrwA4YT7bMVy4PJhR71Thg
[root@master kubespray]\# kubectl -n kube-system describe $(kubectl -n kube-system get secret -n kube-system -o name | grep namespace) | grep token
Name: namespace-controller-token-ksdvp
Type: kubernetes.io/service-account-token
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkV0N0pLMXVqMzNxS2xCNXRlTkxpRTlOYnNMVzRiajNrLU9kdi1qRW5jTDQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJuYW1lc3BhY2UtY29udHJvbGxlci10b2tlbi1rc2R2cCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJuYW1lc3BhY2UtY29udHJvbGxlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImZiM2NkMDBhLTE4ZjAtNGI4OC1hZGNiLWYwNGVjZWNlZTUzNiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpuYW1lc3BhY2UtY29udHJvbGxlciJ9.bYWcoopcYFC6EYNMNPkoiZeUQqfidC9NwlrMzzkZD3T9e-PbDd37pmTeWAcU_E4DCDzeVc9CXfWPhWCfr3syZKWiIXPNtDrNrIgnGs34Id2evsh7evVTgOjQtWkRoqX9UFdjWZdQPxvJChLZacRqbUp718umCzhR9evuE0zq8JeruBCTrcilQDDYobavfYs72HrwZ5xlIj2GMb66FeS7mYZacP-2-M3oVsziIWLs_kfBIaN_OkpImUPpvJxF-8xMmVP2BCKWyHWaLPIUdVsF8FkiLWH7bIS8f0cm8D4wEcMZ4IYkVe2FMcmMaiFJx5HEXrwA4YT7bMVy4PJhR71Thg
在 dashboard 登录页面上使用上面输出中的那个非常长的字符串作为 token 登录，即可以拥有管理员权限操作整个 kubernetes 集群中的对象。当然您也可以将这串 token 加到 admin 用户的 kubeconfig 文件中，继续使用 kubeconfig 登录，两种认证方式任您选择。
登录 dashboard：https://172.16.1.128:30548
注意：由于这里使用的 HTTPS，并未使用证书，因此使用 Google 等浏览器会终止访问。
3.5.5 验证 K8s 集群
- 查看集群版本
[root@master ~]\# kubelet --version
Kubernetes v1.18.2
[root@master ~]\# kubelet --version
Kubernetes v1.18.2
- 查看集群状态
kubectl get nodes
kubectl get nodes
- 查看集群 Pod
kubectl get pods --all-namespaces
kubectl get pods --all-namespaces
- 查看 IPVS
ipvsadm -L -n
ipvsadm -L -n
3.5.6 其他
增加 node 节点（提前在hosts.ini文件中增加主机节点）：
ansible-playbook -i inventory/mycluster/hosts.ini scale.yml -b -v -k
ansible-playbook -i inventory/mycluster/hosts.ini scale.yml -b -v -k
将 hosts.ini 文件中的 master 和 etcd 的机器增加到多台，执行部署命令：
ansible-playbook -i inventory/mycluster/hosts.ini cluster.yml -b -vvv
ansible-playbook -i inventory/mycluster/hosts.ini cluster.yml -b -vvv
刪除节点，如果不指定节点就是刪除整个集群：
ansible-playbook -i inventory/mycluster/hosts.ini remove-node.yml -b -v
ansible-playbook -i inventory/mycluster/hosts.ini remove-node.yml -b -v
如果需要卸载，可以执行以下命令：
ansible-playbook -i inventory/mycluster/hosts.ini reset.yml -b –vvv
ansible-playbook -i inventory/mycluster/hosts.ini reset.yml -b –vvv
升级 K8s 集群，选择对应的 K8s 版本信息，执行升级命令。涉及文件为 upgrade-cluster.yml：
ansible-playbook upgrade-cluster.yml -b -i inventory/mycluster/hosts.ini -e kube_version=vX.XX.XX -vvv
ansible-playbook upgrade-cluster.yml -b -i inventory/mycluster/hosts.ini -e kube_version=vX.XX.XX -vvv
参考链接
kubespray GetStarted：https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting-started.md
使用 Kubespray 在基础设施或云平台上安装 Kubernetes：https://k8smeetup.github.io/docs/getting-started-guides/kubespray/