使用kubeadm部署一套高可用k8s集群

有疑问的地方可以看官方文档

我的机器如下, 系统为ubuntu20.04, kubernetes版本1.21.0

hostname

IP

硬件配置(最低)

ha

10.0.0.10

1核1G

k8s-master0

10.0.0.20

4核4G

k8s-master1

10.0.0.21

4核4G

k8s-master2

10.0.0.22

4核4G

k8s-work0

10.0.0.30

4核4G

基础环境是不管master还是work都需要的环境

1. 禁用swap
2. 确保每个节点上 MAC 地址和 product_uuid 的唯一性

product_uuid可以这样查看

sudo cat /sys/class/dmi/id/product_uuid

1. 修改hostname

2. 允许 iptables 检查桥接流量

   cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
   br_netfilter
   EOF

   cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
   net.bridge.bridge-nf-call-ip6tables = 1
   net.bridge.bridge-nf-call-iptables = 1
   EOF
   sudo sysctl --system

3. 安装runtime

   这里安装docker

   curl -fsSL get.docker.com | bash

修改docker配置

cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://go38p9zi.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  }
}
EOF

1. 安装kubeadm、kubelet 和 kubectl

kubectl 并不是每台机器都需要安装

更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包：

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

下载 Google Cloud 公开签名秘钥：

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

添加 Kubernetes apt 仓库：

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

更新 apt 包索引，安装 kubelet、kubeadm 和 kubectl，并锁定其版本：

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

在ha机器上执行

安装haproxy

apt install -y haproxy

配置haproxy

--- 前面保持默认配置 ---

frontend k8s_api_fe
    bind :6443
    default_backend k8s_api_be
    mode tcp
    option tcplog
backend k8s_api_be
    balance source
    mode tcp
    server      k8s-master0 10.0.0.20:6443 check
    server      k8s-master1 10.0.0.21:6443 check
    server      k8s-master2 10.0.0.22:6443 check

frontend http_ingress_traffic_fe
    bind :80
    default_backend http_ingress_traffic_be
    mode tcp
    option tcplog
backend http_ingress_traffic_be
    balance source
    mode tcp
    server      k8s-work0 10.0.0.30:80 check

frontend https_ingress_traffic_fe
    bind *:443
    default_backend https_ingress_traffic_be
    mode tcp
    option tcplog
backend https_ingress_traffic_be
    balance source
    mode tcp
    server      k8s-work0 10.0.0.30:443 check

kubeadm init

在k8s-master0上执行

kubeadm init \
--apiserver-advertise-address=10.0.0.10 \
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16

其他参数:

--image-repository registry.aliyuncs.com/k8sxio

--kubernetes-version v1.18.0

安装flannel插件

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

获取join命令, 增加新的节点

node

kubeadm init 后会输出在终端上, 有效期2小时, 超时后可以重新生成

生成添加命令:

kubeadm token create --print-join-command

master

生成证书, 记录输出结果

kubeadm init phase upload-certs --upload-certs

获取加入命令

kubeadm token create --print-join-command

上面两步可以简化成

echo "`kubeadm token create --print-join-command` --control-plane --certificate-key `kubeadm init phase upload-certs --upload-certs | tail -1`"

移除节点

kubectl drain k8s-node1 --ignore-daemonsets
kubectl delete node k8s-node1

如果是master节点还需要移除etcd member

kubectl exec -it -n kube-system etcd-k8s-master0 /bin/sh

etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
可以看到有多少个节点

etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 12637f5ec2bd02b8
通过ID来删除