使用kubeadm安装安全高可用kubernetes集群

系统架构图

          kubectl dashboard
                 |
                 V 
     +------------------------+ join
     | LB  10.1.245.94        | <--- Nodes
     +------------------------+
     |                                                   
     |--master1 manager1 schedule1   10.1.245.93                                                
     |--master2 manager2 schedule2   10.1.245.95    =============>  etcd cluster  http://10.1.245.93:2379,http://10.1.245.94:2379,http://10.1.245.95:2379
     |--master3 manager3 schedule3   10.1.245.94                                              

起动etcd集群

cat etcd.yaml

version: '2'
services:
    etcd:
        container_name: etcd_infra0
        image: quay.io/coreos/etcd:v3.1.10
        command: |
                etcd --name infra0
                --initial-advertise-peer-urls http://10.1.245.94:2380
                --listen-peer-urls http://10.1.245.94:2380
                --listen-client-urls http://10.1.245.94:2379,http://127.0.0.1:2379
                --advertise-client-urls http://10.1.245.94:2379
                --data-dir /etcd-data.etcd
                --initial-cluster-token etcd-cluster-1
                -initial-cluster infra0=http://10.1.245.93:2380,infra1=http://10.1.245.94:2379,infra2=http://10.1.245.95:2379
                --initial-cluster-state new
        volumes:
           - /data/etcd-data.etcd:/etcd-data.etcd
        network_mode: "host"

其它两个节点照抄,修改ip即可

使用docker-compose启动,如果没装:

$ pip install docker-compose

三个节点分别启动:

$ docker-compose -f etcd.yaml up -d

检查是不是安装成功:

$ docker exec etcd_infra0 etcdctl menber list

kubeadm配置

config.yaml

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
apiServerCertSANs:
- 10.1.245.93
- 10.1.245.94
- 10.1.245.95
- 47.52.227.242
etcd:
  endpoints:
  - http://10.1.245.94:2379
networking:
  podSubnet: 192.168.0.0/16
kubernetesVersion: v1.8.2

注意版本号 apiServerCertSANs与证书配置有关,把你所有master的ip和lb的ip都写进去,或者你允许的域名等

$ kubeadm init --config config.yaml

启动多个master

别的master节点初始化好之后,把第一个master的/etc/kubernetes目录拷贝到别的master节点上

$ scp -r root@10.1.245.93:/etc/kubernetes /etc

修改该目录下各conf的ip,改成本机ip, 如下命令搜出来的都要改

grep 245.93 . -rn

启动kubelet

systemctl start kubelet

启动loadbalance

我比较推荐使用四层代理 HAproxy配置: cat /root/haproxy/haproxy.cfg

global
  daemon
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  maxconn 4096

defaults
  log               global
  retries           3
  maxconn           2000
  timeout connect   5s
  timeout client    50s
  timeout server    50s

frontend k8s
  bind *:6444
  mode tcp
  default_backend k8s-backend

backend k8s-backend
  balance roundrobin
  mode tcp
  server k8s-1 10.1.245.93:6443 check
  server k8s-1 10.1.245.94:6443 check
  server k8s-2 10.1.245.95:6443 check
docker run --net=host -v /root/haproxy:/usr/local/etc/haproxy --name ha -d haproxy:1.7

join node节点

还是在node节点执行第一个master输出的命令,不过IP换成LB的ip地址,就是上面haproxy的地址 如

$ kubeadm join --token <token> 10.1.245.94:6444 --discovery-token-ca-cert-hash sha256:<hash>

kubectl配置

修改~/.kube/config文件,ip改成LB的ip 10.1.245.94:6444

或者通过命令修改:

$ kubectl config set-cluster kubernetes --server=https://47.52.227.242:6443 --kubeconfig=$HOME/.kube/config