在之前的实验中,kubernetes集群都是一台master和两台node组成的小集群,在实际的生产环境中需要考虑到集群的高可用。
在node节点实际已经实现了高可用,pod分布在不同的节点上,当一个节点宕机的时候,其上的pod会漂移到正常的节点上。所以,重点的高可用重心就要放在master上。
图中可以看出,用户通过kubectl发送命令经过LB进行负载均衡到后端的master上的apiserver,再由具体的某一个master进行向集群内部的节点的转发。
同理,节点也是通过LB进行负载均衡连接到master上的apiserver,去获取到apiserver中配置的信息。
图中可以看到,每一台node上都部署了 nginx做负载均衡到master的apiserver,而kube-scheduler和controller-manager不需要做高可用,因为它们默认会通过选举产生,可以通过下面的命令查看:
将master节点扩展至2个,新增加的master的ip为:10.10.99.240
在原先的master上的server-csr.json
中增加新增额度master的ip:
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"10.10.10.1",
"10.10.99.225",
"10.10.99.233",
"10.10.99.228",
"10.10.99.240",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Shenzhen",
"ST": "Guangzhou",
"O": "k8s",
"OU": "System"
}
]
}
重新生成server证书:
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
cp -p server-key.pem server.pem /opt/kubernetes/ssl/
重启apiserver:
systemctl restart kube-apiserver
将原先master上的/opt/kubernetes发送到新的master节点上:
scp -r /opt/kubernetes/ root@10.10.99.240:/opt
从原先的master上拷贝服务的配置文件到新的master上:
scp /usr/lib/systemd/system/{kube-apiserver,kube-scheduler,kube-controller-manager}.service root@10.10.99.240:/usr/lib/systemd/system/
修改新master节点上的kube-apiserver
配置文件:
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://10.10.99.225:2379,https://10.10.99.228:2379,https://10.10.99.233:2379 \
--insecure-bind-address=127.0.0.1 \
--bind-address=10.10.99.240 \
--insecure-port=8080 \
--secure-port=6443 \
--advertise-address=10.10.99.240 \
--allow-privileged=true \
--service-cluster-ip-range=10.10.10.0/24 \
--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node \
--kubelet-https=true \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/kubernetes/ssl/ca.pem \
--etcd-certfile=/opt/kubernetes/ssl/server.pem \
--etcd-keyfile=/opt/kubernetes/ssl/server-key.pem"
将advertise-address和bind-address改为本机的ip
启动新master上的组件:
systemctl daemon-reload
systemctl start kube-apiserver.service
systemctl enable kube-apiserver.service
systemctl start kube-scheduler.service
systemctl enable kube-scheduler.service
systemctl start kube-controller-manager.service
systemctl enable kube-controller-manager.service
在新的master上使用kubectl查看集群中的节点:
echo PATH=$PATH:/opt/kubernetes/bin >> /etc/profile
source /etc/profile
kubectl get node
在新的master上安装iptables并添加默认规则:
yum remove firewalld
yum install -y iptables iptables-services
systemctl start iptables
systemctl enable iptables
vim /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -s 10.1.0.0/20 -j ACCEPT
-A RH-Firewall-1-INPUT -s 10.10.0.0/12 -j ACCEPT
-A RH-Firewall-1-INPUT -s 172.16.1.0/24 -m udp -p udp --dport 161 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -s 192.168.0.0/16 -j ACCEPT
-A RH-Firewall-1-INPUT -s 172.30.0.0/16 -j ACCEPT
-A RH-Firewall-1-INPUT -s 10.254.0.0/16 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
测试,之前在node节点上也装了一个kubectl,在那个节点上修改 /root/.kube/config
中的server
字段指向新的master,然后在这个节点测试命令kubectl get node
也是可以查看到node信息的。
首先在两台node节点上安装nginx做4层负载均衡:
yum install -y nginx
将来node节点的kubelet和proxy将连接本地的nginx,然后nginx做负载均衡转发到master上的apiserver
配置nginx:
vim /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 10.10.99.225:6443;
server 10.10.99.240:6443;
}
server {
listen 127.0.0.1:6443;
proxy_pass k8s-apiserver;
}
}
在node节点上修改bootstrap.kubeconfig kubelet.kubeconfig kube-proxy.kubeconfig
中的apiserver地址为本地:
server: https://127.0.0.1:6443
在node节点上重启服务:
systemctl restart kubelet.service
systemctl restart kube-proxy.service
在node节点上启动nginx:
systemctl start nginx
systemctl enable nginx
查看一下nginx日志有没有代理记录:
tail /var/log/nginx/k8s-access.log
手机扫一扫
移动阅读更方便
你可能感兴趣的文章