理论是灰色的&＃xff0c;实践之树长青&＃x1f332; ——恩格斯
近日在做毕设项目&＃xff0c;涉及到在K8s和swarm基础上部署JupyterHub&＃xff0c;经过两天时间的学习和部署&＃xff0c;N次的失败尝试&＃xff0c;最终在服务器上成功部署了JupyterHub&＃xff01;
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
因为我们需要在k8s上安装Jupyterhub&＃xff0c;所以需要安装k8s&＃xff0c;k8s有多种安装方式&＃xff0c;在这里我们使用kubeadm进行安装。
由于众所周知强的问题&＃xff0c;所以这里需要更新一下下载的镜像&＃xff0c;使用如下命令&＃xff1a;
apt-get update && apt-get install -y apt-transport-https
curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update
root&＃64;host1:~# apt-get install -y kubelet kubeadm kubectl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:conntrack cri-tools kubernetes-cni socat
The following NEW packages will be installed:conntrack cri-tools kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 7 newly installed, 0 to remove and 88 not upgraded.
Need to get 68.7 MB of archives.
After this operation, 293 MB of additional disk space will be used.
Get:1 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 cri-tools amd64 1.13.0-01 [8,775 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 conntrack amd64 1:1.4.4&＃43;snapshot20161117-6ubuntu2 [30.6 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB]
Get:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.7-00 [25.0 MB]
Get:5 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubelet amd64 1.20.5-00 [18.9 MB]
Get:6 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubectl amd64 1.20.5-00 [7,945 kB]
Get:7 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubeadm amd64 1.20.5-00 [7,709 kB]
Fetched 68.7 MB in 51s (1,360 kB/s)
Selecting previously unselected package conntrack.
(Reading database ... 111595 files and directories currently installed.)
Preparing to unpack .../0-conntrack_1%3a1.4.4&＃43;snapshot20161117-6ubuntu2_amd64.deb ...
Unpacking conntrack (1:1.4.4&＃43;snapshot20161117-6ubuntu2) ...
Selecting previously unselected package cri-tools.
Preparing to unpack .../1-cri-tools_1.13.0-01_amd64.deb ...
Unpacking cri-tools (1.13.0-01) ...
Selecting previously unselected package kubernetes-cni.
Preparing to unpack .../2-kubernetes-cni_0.8.7-00_amd64.deb ...
Unpacking kubernetes-cni (0.8.7-00) ...
Selecting previously unselected package socat.
Preparing to unpack .../3-socat_1.7.3.2-2ubuntu2_amd64.deb ...
Unpacking socat (1.7.3.2-2ubuntu2) ...
Selecting previously unselected package kubelet.
Preparing to unpack .../4-kubelet_1.20.5-00_amd64.deb ...
Unpacking kubelet (1.20.5-00) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../5-kubectl_1.20.5-00_amd64.deb ...
Unpacking kubectl (1.20.5-00) ...
Selecting previously unselected package kubeadm.
Preparing to unpack .../6-kubeadm_1.20.5-00_amd64.deb ...
Unpacking kubeadm (1.20.5-00) ...
Setting up conntrack (1:1.4.4&＃43;snapshot20161117-6ubuntu2) ...
Setting up kubernetes-cni (0.8.7-00) ...
Setting up cri-tools (1.13.0-01) ...
Setting up socat (1.7.3.2-2ubuntu2) ...
Setting up kubelet (1.20.5-00) ...
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service.
Setting up kubectl (1.20.5-00) ...
Setting up kubeadm (1.20.5-00) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
# 确保关闭交换空间
$ sudo swapoff -a# 获取最新 Kubernetes 版本号
$ KUBERNETES_RELEASE_VERSION&＃61;"$(curl -sSL https://dl.k8s.io/release/stable.txt)"# 可以用下面的命令列出 kubeadm 需要的 images
$ kubeadm config images list --kubernetes-version&＃61;${KUBERNETES_RELEASE_VERSION}
# 所需版本如下&＃xff08;每个人情况可能不同&＃xff0c;以自己的结果为主&＃xff09;
root&＃64;host1:~# kubeadm config images list --kubernetes-version&＃61;${KUBERNETES_RELEASE_VERSION}
k8s.gcr.io/kube-apiserver:v1.20.5
k8s.gcr.io/kube-controller-manager:v1.20.5
k8s.gcr.io/kube-scheduler:v1.20.5
k8s.gcr.io/kube-proxy:v1.20.5
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/coredns:1.7.0
由于墙的问题&＃xff0c;我们不能将这些image拉下来&＃xff0c;所以在这里我们采用脚本的方式将其下载&＃xff1b;
编写脚本k8sInstall.py下载所需镜像,注意&＃xff0c;脚本中的版本号要和kubeadm config images list命令显示的版本号一致&＃xff0c;脚本文件如下&＃xff1a;
// k8sInstall.py
#! /usr/bin/python3import osimages&＃61;["kube-apiserver:v1.20.5","kube-controller-manager:v1.20.5","kube-scheduler:v1.20.5","kube-proxy:v1.20.5","pause:3.2","etcd:3.4.13-0","coredns:1.7.0",
]for i in images:pullCMD &＃61; "docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/{}".format(i)print("run cmd &＃39;{}&＃39;, please wait ...".format(pullCMD))os.system(pullCMD)tagCMD &＃61; "docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/{} k8s.gcr.io/{}".format(i, i)print("run cmd &＃39;{}&＃39;, please wait ...".format(tagCMD ))os.system(tagCMD)rmiCMD &＃61; "docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/{}".format(i)print("run cmd &＃39;{}&＃39;, please wait ...".format(rmiCMD ))os.system(rmiCMD)# 执行脚本文件&＃xff0c;下载镜像
root&＃64;host1:~# vim k8sInstall.py
root&＃64;host1:~# chmod 775 k8sInstall.py
root&＃64;host1:~# sudo ./k8sInstall.py
run cmd &＃39;docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.5&＃39;, please wait ...
v1.20.5: Pulling from google_containers/kube-apiserver
fefd475334af: Pull complete
742efefc8a44: Pull complete
98d681774b17: Pull complete
Digest: sha256:e2ca052cbde8532d4c2e613a7fbb3145a5ecbea1bd47ac243877815ed80abb5a
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.5
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.5
run cmd &＃39;docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.5 k8s.gcr.io/kube-apiserver:v1.20.5&＃39;, please wait ...
run cmd &＃39;docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.5&＃39;, please wait ...
Untagged: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.20.5
Untagged: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver&＃64;sha256:e2ca052cbde8532d4c2e613a7fbb3145a5ecbea1bd47ac243877815ed80abb5a
run cmd &＃39;docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.20.5&＃39;, please wait ...
v1.20.5: Pulling from google_containers/kube-controller-manager
fefd475334af: Already exists
742efefc8a44: Already exists
454a7944c47b: Pull complete
Digest: sha256:ddba11d3205b4fde9b8621b79bbb942f1d0797ffcc9dfb72d197b0c84943e369
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.20.5
......
服务器必须是2核及以上&＃xff0c;否则在这一步会出现问题&＃xff01;&＃xff01;&＃xff01;
root&＃64;host1:~# kubeadm init --ignore-preflight-errors&＃61;SystemVerification
[init] Using Kubernetes version: v1.20.5
[preflight] Running pre-flight checks[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.5. Latest validated version: 19.03
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using &＃39;kubeadm config images pull&＃39;
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [izwz9hwh629hd38je8xvscz kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.21.24.197]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [izwz9hwh629hd38je8xvscz localhost] and IPs [172.21.24.197 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [izwz9hwh629hd38je8xvscz localhost] and IPs [172.21.24.197 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 85.005388 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node izwz9hwh629hd38je8xvscz as control-plane by adding the labels "node-role.kubernetes.io/master&＃61;&＃39;&＃39;" and "node-role.kubernetes.io/control-plane&＃61;&＃39;&＃39; (deprecated)"
[mark-control-plane] Marking the node izwz9hwh629hd38je8xvscz as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 33wev0.vwvmk139exv3ohx3
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxyYour Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG&＃61;/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 172.21.24.197:6443 --token 33wev0.vwvmk139exv3ohx3 \--discovery-token-ca-cert-hash sha256:f25ef958e127c21168c661355c7df7280c71dc39ce277de9dabd3c1fb9cd9579
这里需要使用到kube-flannel.yml文件&＃xff0c;但由于墙的问题https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml 该网址下载不了&＃xff1b;
可以使用 https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml下载或复制到服务器&＃xff1b;
root&＃64;host1:~# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
由于博主这里是单节点集群&＃xff0c;所以需要将master节点也设置为调度节点
$ kubectl taint nodes --all node-role.kubernetes.io/master-
$ curl -s https://get.helm.sh/helm-v3.5.2-linux-amd64.tar.gz | tar xzv
$ sudo cp linux-amd64/helm /usr/local/bin
$ rm -rf linux-amd64
添加使用 azure.cn 提供的 charts 镜像
$ helm repo add stable https://mirror.azure.cn/kubernetes/charts/
$ helm repo add incubator https://mirror.azure.cn/kubernetes/charts-incubator/# 更新本地 charts repo
$ helm repo update# 测试安装 redis chart
$ helm install my-redis stable/redis# 删除 redis
$ helm uninstall my-redis
到这里我们安装Jupyter的前置工作已经完成了&＃xff0c;接下来就需要我们去安装Jupyter&＃xff01;
现在我们已经有了Kubernetes集群&＃xff0c;我们可以使用Helm通过 Helm chart安装JupyterHub和相关的Kubernetes资源。
# 1、生成随机字符串作安全标识&＃xff1a;
$ openssl rand -hex 32# 2、创建一个名为config.yaml的文件touch config.yaml# 3、使用命令vi config.yaml添加以下内容&＃xff1a;
proxy:secretToken: "
helm添加软件仓库并更新仓库&＃xff1a;
$ helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
$ helm repo update
# 安装完成
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "sqlfiddle" chart repository
...Successfully got an update from the "jupyterhub" chart repository
...Successfully got an update from the "emberstack" chart repository
Update Complete. ⎈ Happy Helming!⎈
在config.yaml所在目录下执行下列命令安装chart&＃xff1a;
RELEASE&＃61;jhub
NAMESPACE&＃61;jhubhelm upgrade --cleanup-on-fail \--install $RELEASE jupyterhub/jupyterhub \--namespace $NAMESPACE \--create-namespace \--version&＃61;0.10.6 \--values config.yaml
⚠️在helm3中我们需要指定release&＃xff0c;如果没有就使用 --generat-name&＃61;
在执行步骤2的过程中&＃xff0c;您可以通过输入其他终端来查看正在创建的Pod&＃xff1a;
root&＃64;host1:~# kubectl get pod --namespace jhub
NAME READY STATUS RESTARTS AGE
continuous-image-puller-mp7mq 1/1 Running 0 25s
hub-55dcd585c-6dpwr 0/1 ContainerCreating 0 25s
proxy-7645dd5d94-fvgsm 1/1 Running 0 25s
user-scheduler-7db7bfbdc6-k76xj 0/1 ErrImagePull 0 25s
user-scheduler-7db7bfbdc6-qg64l 0/1 ContainerCreating 0 25s
等待hub和proxy都进入running状态
root&＃64;host1:~# kubectl get pod --namespace jhub
NAME READY STATUS RESTARTS AGE
continuous-image-puller-mp7mq 1/1 Running 0 2m22s
hub-55dcd585c-6dpwr 1/1 Running 0 2m22s
proxy-7645dd5d94-fvgsm 1/1 Running 0 2m22s
使用 kubectl get service --namespace jhub命令找到proxy-public对应的端口号
root&＃64;host1:~# kubectl get service --namespace jhub
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hub ClusterIP 10.105.101.125 <none> 8081/TCP 14h
proxy-api ClusterIP 10.100.241.193 <none> 8001/TCP 14h
proxy-public NodePort 10.101.104.188 <none> 80:31420/TCP 14h
打开浏览器&＃xff0c;地址栏输入主机ip:端口号访问并使用jupyterhub。
⚠️这里如果有朋友使用的是服务器&＃xff0c;一定要将服务器的安全组修改&＃xff0c;入方向的端口和IP修改一下&＃xff01;&＃xff01;&＃xff01;切记
1、proxy-public如果处于pending状态&＃xff0c;是因为svc暴露给外网的方式为LoadBalancer&＃xff0c;又没有配转发&＃xff0c;所以获取不到ip&＃xff1b;修改svc中的type为NodePort&＃xff0c;通过使用本机ip&＃43;端口的方式暴露给外网。使用如下命令&＃xff1a;
kubectl -n jhub get svc
kubectl -n jhub edit svc proxy-public
# 找到type字段将LoadBalancer修改为NodePort
2、hub的pod处于pending状态&＃xff0c;kubectl describe pod &＃43; pod名&＃xff0c;如果报错“pod has unbound immediate PersistentVolumeClaims”&＃xff0c;则修改helm安装jupyterhub时的config.yaml文件&＃xff0c;添加以下内容&＃xff1a;&＃xff08;这个问题困扰了我很久&＃xff0c;感谢博友的解答&＃xff09;
hub:db:type: sqlite-memory
singleuser:storage:type: none
然后重新执行以下命令&＃xff1a;
RELEASE&＃61;jhub
NAMESPACE&＃61;jhubhelm upgrade --cleanup-on-fail \--install $RELEASE jupyterhub/jupyterhub \--namespace $NAMESPACE \--create-namespace \--version&＃61;0.10.6 \--values config.yaml
在整个安装过程中遇到了各类bug&＃xff0c;好在博友们都很强大&＃xff0c;在github和博客上有很多解决方案&＃xff0c;大家可以去参考&＃xff01;
在这里记录一下对我有重要帮助的博友链接&＃xff1a;
https://github.com/maguowei/gotok8s
https://zero-to-jupyterhub.readthedocs.io/en/stable/jupyterhub/installation.html
https://blog.csdn.net/u012174752/article/details/89173916
https://www.jianshu.com/p/491a562127fa
欢迎大家一起关注交流学习哈&＃xff01;
个人GitHub&＃xff1a;https://github.com/SpecialAll
京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有 
