内网ubuntu环境下离线部署K8s

背景

最近工作在内网环境，需要从头搭建一个K8s集群并进行微服务迁移。

前期调研的K8S离线部署的方案，可以参考其他博主的博文——万字长文详解 PaaS toB 场景下 K8s 离线部署方案

Item	Language	Star	Fork	离线部署支持情况
kops	Golang	13.2k	4.1k	不支持
kubespray	Ansible	11.1k	4.7k	支持，需自行构建安装包
kubeasz	Ansible	7.2k	2.7k	支持，需自行构建安装包
sealos	Golang	4.1k	790	支持，需付费充值会员
RKE	Golang	2.5k	480	不支持，需自行安装 docker
sealer	Golang	503	112	支持，源自 sealos
kubekey	Golang	471	155	部分支持，仅镜像可离线

以上的方案我尝试了sealos、kubekey，但是由于最初给到我的机器本身网络环境不干净，导致cni一直无法正常拉起，而这些部署工具在简化部署的同时，也屏蔽了大量细节，导致定位问题比较麻烦，后续还是使用kubeadm自行构建。

部署物料准备

服务器：

• 系统环境：Ubuntu18.04

• 机器：3台

docker：

• 版本：19.03
• 安装包下载地址：https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/

wget -P /home/deploy/deb/docker/ https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce_19.03.13~3-0~ubuntu-bionic_amd64.deb
wget -P /home/deploy/deb/docker/ https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/containerd.io_1.3.7-1_amd64.deb
wget -P /home/deploy/deb/docker/ https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce-cli_19.03.13~3-0~ubuntu-bionic_amd64.deb

K8S：

• 版本：1.19.16

apt-get update && apt-get install -y apt-transport-https

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

cat > /etc/apt/sources.list.d/kubernetes.list << ERIC
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
ERIC

apt-get update

apt-cache madison kubeadm

VERSION=1.19.16-00
###### 将包下载到本地
apt-get install -y --download-only -o dir::cache::archives=/home/deploy/deb/k8s kubelet=$VERSION kubeadm=$VERSION kubectl=$VERSION

镜像准备

国内镜像准备

registry.cn-hangzhou.aliyuncs.com/google_containers

在联网环境下准备以下镜像：

➜  ~ kubeadm config images  --kubernetes-version=v1.19.16 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0

拉取

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0

保存

docker save -o k8s.tar registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0

Flannel

下载：https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

实施

关闭swap分区

sudo swapoff -a

关闭防火墙

systemctl disable ufw && systemctl stop ufw

安装docker

dpkg -i /home/deploy/deb/docker/*.deb

安装后默认cgroups驱动使用cgroupfs ，需要调整为systemd，因此，编辑docker配置文件，执行：

sudo vi /etc/docker/daemon.json

添加如下内容：

{
"exec-opts": ["native.cgroupdriver=systemd"]
}

重启docker，执行：

sudo systemctl daemon-reload && sudo systemctl restart docker

安装kubeadm、kubelet 和 kubectl

###### 离线安装 k8s
dpkg -i /home/deploy/deb/k8s/*.deb

导入镜像

docker load < k8s.tar

启动

sudo kubeadm init --pod-network-cidr 10.244.0.0/16 \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers

执行init成功后，记录下以“kubeadm join”开头的最后两行：

kubeadm join 192.168.20.104:6443 --token 0mj488.h6v5r010bfhlq9b1 \
    --discovery-token-ca-cert-hash sha256:3ea2cc19ceb0f109834f82bde13f5d29c534aba115cd41f8d3719db6b8ec074b
root@master01:/home/deploy/deb/yaml

最后依次执行：

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

启用Flannel网络

执行

kubectl apply -f ./kube-flannel.yml 

执行成功后，稍等3、5分钟，再次执行kubectl get nodes 和 kubectl get pods –all-namespaces，会看到状态正常了

yance@yance-ub:~$ kubectl get pod -n kube-system
NAME                               READY   STATUS    RESTARTS   AGE
coredns-6c76c8bb89-vjghr           1/1     Running   0          46m
coredns-6c76c8bb89-zswv9           1/1     Running   0          46m
etcd-yance-ub                      1/1     Running   0          46m
kube-apiserver-yance-ub            1/1     Running   0          46m
kube-controller-manager-yance-ub   1/1     Running   0          46m
kube-flannel-ds-dlxgv              1/1     Running   0          23m
kube-proxy-nhdwj                   1/1     Running   0          46m
kube-scheduler-yance-ub            1/1     Running   0          46m

加入worker节点

在每个worker节点上，执行

kubeadm join 192.168.20.104:6443 --token 0mj488.h6v5r010bfhlq9b1 \
    --discovery-token-ca-cert-hash sha256:3ea2cc19ceb0f109834f82bde13f5d29c534aba115cd41f8d3719db6b8ec074b
root@master01:/home/deploy/deb/yaml

在master上执行：

kubectl label node node_name node-role.kubernetes.io/worker=worker 

安装 Kuboard v3.x

sudo docker run -d \
  --restart=unless-stopped \
  --name=kuboard \
  -p 80:80/tcp \
  -p 10081:10081/tcp \
  -e KUBOARD_ENDPOINT="http://内网IP:80" \
  -e KUBOARD_AGENT_SERVER_TCP_PORT="10081" \
  -v /root/kuboard-data:/data \
  eipwork/kuboard:v3

在浏览器输入 http://your-host-ip:80 即可访问 Kuboard v3.x 的界面，登录方式：

• 用户名： admin
• 密 码： Kuboard123