内网ubuntu环境下离线部署K8s

本文最后更新于:2022年8月1日 凌晨

背景

最近工作在内网环境,需要从头搭建一个K8s集群并进行微服务迁移。

前期调研的K8S离线部署的方案,可以参考其他博主的博文——万字长文详解 PaaS toB 场景下 K8s 离线部署方案

Item Language Star Fork 离线部署支持情况
kops Golang 13.2k 4.1k 不支持
kubespray Ansible 11.1k 4.7k 支持,需自行构建安装包
kubeasz Ansible 7.2k 2.7k 支持,需自行构建安装包
sealos Golang 4.1k 790 支持,需付费充值会员
RKE Golang 2.5k 480 不支持,需自行安装 docker
sealer Golang 503 112 支持,源自 sealos
kubekey Golang 471 155 部分支持,仅镜像可离线

以上的方案我尝试了sealoskubekey,但是由于最初给到我的机器本身网络环境不干净,导致cni一直无法正常拉起,而这些部署工具在简化部署的同时,也屏蔽了大量细节,导致定位问题比较麻烦,后续还是使用kubeadm自行构建。

部署物料准备

服务器:

  • 系统环境:Ubuntu18.04

  • 机器:3台

docker:

1
2
3
wget -P /home/deploy/deb/docker/ https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce_19.03.13~3-0~ubuntu-bionic_amd64.deb
wget -P /home/deploy/deb/docker/ https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/containerd.io_1.3.7-1_amd64.deb
wget -P /home/deploy/deb/docker/ https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce-cli_19.03.13~3-0~ubuntu-bionic_amd64.deb

K8S:

  • 版本:1.19.16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apt-get update && apt-get install -y apt-transport-https

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

cat > /etc/apt/sources.list.d/kubernetes.list << ERIC
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
ERIC

apt-get update

apt-cache madison kubeadm

VERSION=1.19.16-00
###### 将包下载到本地
apt-get install -y --download-only -o dir::cache::archives=/home/deploy/deb/k8s kubelet=$VERSION kubeadm=$VERSION kubectl=$VERSION

镜像准备

国内镜像准备

1
registry.cn-hangzhou.aliyuncs.com/google_containers

在联网环境下准备以下镜像:

1
2
3
4
5
6
7
8
➜  ~ kubeadm config images  --kubernetes-version=v1.19.16 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0

拉取

1
2
3
4
5
6
7
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.16
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0

保存

1
2
3
4
5
6
7
docker save -o k8s.tar registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.16
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0

Flannel

下载:https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

实施

关闭swap分区

1
sudo swapoff -a

关闭防火墙

1
systemctl disable ufw && systemctl stop ufw

安装docker

1
dpkg -i /home/deploy/deb/docker/*.deb

安装后默认cgroups驱动使用cgroupfs ,需要调整为systemd,因此,编辑docker配置文件,执行:

1
sudo vi /etc/docker/daemon.json

添加如下内容:

1
2
3
{
"exec-opts": ["native.cgroupdriver=systemd"]
}

重启docker,执行:

1
sudo systemctl daemon-reload && sudo systemctl restart docker

安装kubeadm、kubelet 和 kubectl

1
2
###### 离线安装 k8s
dpkg -i /home/deploy/deb/k8s/*.deb

导入镜像

1
docker load < k8s.tar

启动

1
2
sudo kubeadm init --pod-network-cidr 10.244.0.0/16 \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers

执行init成功后,记录下以“kubeadm join”开头的最后两行:

1
2
3
kubeadm join 192.168.20.104:6443 --token 0mj488.h6v5r010bfhlq9b1 \
--discovery-token-ca-cert-hash sha256:3ea2cc19ceb0f109834f82bde13f5d29c534aba115cd41f8d3719db6b8ec074b
root@master01:/home/deploy/deb/yaml

最后依次执行:

1
2
3
4
5
mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

启用Flannel网络

执行

1
kubectl apply -f ./kube-flannel.yml 

执行成功后,稍等3、5分钟,再次执行kubectl get nodes 和 kubectl get pods –all-namespaces,会看到状态正常了

1
2
3
4
5
6
7
8
9
10
yance@yance-ub:~$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6c76c8bb89-vjghr 1/1 Running 0 46m
coredns-6c76c8bb89-zswv9 1/1 Running 0 46m
etcd-yance-ub 1/1 Running 0 46m
kube-apiserver-yance-ub 1/1 Running 0 46m
kube-controller-manager-yance-ub 1/1 Running 0 46m
kube-flannel-ds-dlxgv 1/1 Running 0 23m
kube-proxy-nhdwj 1/1 Running 0 46m
kube-scheduler-yance-ub 1/1 Running 0 46m

加入worker节点

在每个worker节点上,执行

1
2
3
kubeadm join 192.168.20.104:6443 --token 0mj488.h6v5r010bfhlq9b1 \
--discovery-token-ca-cert-hash sha256:3ea2cc19ceb0f109834f82bde13f5d29c534aba115cd41f8d3719db6b8ec074b
root@master01:/home/deploy/deb/yaml

在master上执行:

1
kubectl label node node_name node-role.kubernetes.io/worker=worker 

安装 Kuboard v3.x

1
2
3
4
5
6
7
8
9
sudo docker run -d \
--restart=unless-stopped \
--name=kuboard \
-p 80:80/tcp \
-p 10081:10081/tcp \
-e KUBOARD_ENDPOINT="http://内网IP:80" \
-e KUBOARD_AGENT_SERVER_TCP_PORT="10081" \
-v /root/kuboard-data:/data \
eipwork/kuboard:v3

在浏览器输入 http://your-host-ip:80 即可访问 Kuboard v3.x 的界面,登录方式:

  • 用户名: admin
  • 密 码: Kuboard123


内网ubuntu环境下离线部署K8s
https://yance.wiki/k8s_delopy/
作者
Yance Huang
发布于
2022年1月1日
许可协议