Table of Contents
前言
Charmed Kuberenetes 是 Canonical 提供的 Kubernetes 部署方式，可以透過 juju 將 Kubernetes 部署至各種不同環境。
本篇將介紹如何部署 Charmed Kubernetes 至 OpenStack 上，並且利用 OpenStack Integrator 使用 OpenStack 提供 Persistent Volume 和 Load Balancer 給 Kubernetes 使用。
設定及部署 juju OpenStack Cloud Controller
juju add-cloud --client openstack輸入以下資訊
- cloud type: openstack
- endpoint:
- cert path: none
- auth type: userpass
- region: RegionOne (應該是預設值)
- API endpoint url for the region: 跳過，會直接使用 endpoint
-
Enter another region? (y/N): N
加入 OpenStack Credential
juju autoload-credentials上傳 image
juju deploy glance-simplestreams-sync --to 0 --channel 2023.2/stable --config use_swift=false
juju integrate glance-simplestreams-sync:identity-service keystone:identity-service
juju integrate glance-simplestreams-sync:certificates vault:certificatesjuju run glance-simplestreams-sync/leader sync-images可以透過以下指令獲得 Image ID
openstack image list設定 image metadata
mkdir simplestreams
export IMAGE=<IMAGE_ID>
juju metadata generate-image -d ~/simplestreams -i $IMAGE -s jammy -r RegionOne -u <OPENSTACK_API_ENDPOINT>設定私有網路
openstack network create --internal user1_net
openstack subnet create --network user1_net --dns-nameserver 8.8.8.8 \
--subnet-range 192.168.0/24 \
--allocation-pool start=192.168.0.10,end=192.168.0.99 \
user1_subnet
openstack router create user1_router
openstack router add subnet user1_router user1_subnet
openstack router set user1_router --external-gateway ext_net在 OpenStack 上建立 juju controller
juju bootstrap --debug --config network=user1_net --config external-network=<external_network_id> --bootstrap-constraints allocate-public-ip=true --bootstrap-series jammy --bootstrap-constraints instance-type=m1.small --metadata-source $HOME/simplestreams/ openstack openstack同時需於另外一個 terminal 在 bootstrap 的 instance 上加入 floating IP，使 client node 可以連上 juju controller
FLOATING_IP=$(openstack floating ip create -f value -c floating_ip_address ext_net)
openstack server add floating ip <server_id> $FLOATING_IP部署 Charmed Kubernetes
加入新的 juju model
juju add-model --config default-series=jammy k8s openstack
juju switch openstack:k8s建立 openstack-overlay.yaml
description: Charmed Kubernetes overlay to add native OpenStack support.
applications:
kubeapi-load-balancer: null
openstack-integrator:
annotations:
gui-x: "600"
gui-y: "300"
charm: openstack-integrator
num_units: 1
constraints: "cores=1 mem=1G root-disk=15G"
trust: true
relations:
- ['openstack-integrator', 'kubernetes-control-plane:openstack']
- ['openstack-integrator', 'kubernetes-worker:openstack']
- ['openstack-integrator', 'kubernetes-control-plane:loadbalancer']建立 cilium-overlay.yaml
description: Charmed Kubernetes overlay to add Cilium CNI.
applications:
calico: null
cilium:
charm: cilium
kubernetes-control-plane:
options:
allow-privileged: "true"
sysctl: &sysctl "{net.ipv4.conf.all.forwarding: 1, net.ipv4.conf.all.rp_filter: 0, net.ipv4.neigh.default.gc_thresh1: 128, net.ipv4.neigh.default.gc_thresh2: 28672, net.ipv4.neigh.default.gc_thresh3: 32768, net.ipv6.neigh.default.gc_thresh1: 128, net.ipv6.neigh.default.gc_thresh2: 28672, net.ipv6.neigh.default.gc_thresh3: 32768, fs.inotify.max_user_instances: 8192, fs.inotify.max_user_watches: 1048576, kernel.panic: 10, kernel.panic_on_oops: 1, vm.overcommit_memory: 1}"
kubernetes-worker:
options:
sysctl: *sysctl
relations:
- [cilium:cni, kubernetes-control-plane:cni]
- [cilium:cni, kubernetes-worker:cni]部署 Kubernetes
juju deploy charmed-kubernetes --channel=1.28/stable --overlay openstack-overlay.yaml --trust --overlay cilium-overlay.yaml如果資源不足，可以使用 kubernetes-core bundle 做測試
juju deploy kubernetes-core --channel=1.28/stable --overlay openstack-overlay.yaml --trust --overlay cilium-overlay.yaml注意 Charmed Kubernetes 有些預設的 Instance constraints，OpenStack 上需要有符合的 flavor
可以額外透過 overlay 進行 override
例：
application:
"kubernetes-worker":
num_units: 1
constraints: cores=2 mem=4G root-disk=20G
"kubernetes-control-plane":
num_units: 1
constraints: cores=2 mem=4G root-disk=20G
"etcd":
num_units: 1
constraints: "cores=1 mem=2G root-disk=20G"
"easyrsa":
num_units: 1
constraints: "cores=1 mem=1G root-disk=15G"部署完成 juju status 輸出將會像這樣（kubernetes-core 為例）：
Model Controller Cloud/Region Version SLA Timestamp
k8s openstack openstack/RegionOne 3.1.6 unsupported 00:46:56Z
App Version Status Scale Charm Channel Rev Exposed Message
cilium 1.12.5,1.12.5 active 2 cilium stable 24 no Ready
containerd 1.6.8 active 2 containerd 1.28/stable 73 no Container runtime available
easyrsa 3.0.1 active 1 easyrsa 1.28/stable 48 no Certificate Authority connected.
etcd 3.4.22 active 1 etcd 1.28/stable 748 no Healthy with 1 known peer
kubernetes-control-plane 1.28.4 active 1 kubernetes-control-plane 1.28/stable 321 yes Kubernetes control-plane running.
kubernetes-worker 1.28.4 active 1 kubernetes-worker 1.28/stable 134 yes Kubernetes worker running.
openstack-integrator yoga active 1 openstack-integrator stable 69 no Ready
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0/lxd/0 252.82.3.157 Certificate Authority connected.
etcd/0* active idle 0 192.168.0.82 2379/tcp Healthy with 1 known peer
kubernetes-control-plane/0* active idle 0 192.168.0.82 6443/tcp Kubernetes control-plane running.
cilium/1* active idle 192.168.0.82 Ready
containerd/1* active idle 192.168.0.82 Container runtime available
kubernetes-worker/0* active idle 1 192.168.0.68 80,443/tcp Kubernetes worker running.
cilium/0 active idle 192.168.0.68 Ready
containerd/0 active idle 192.168.0.68 Container runtime available
openstack-integrator/1* active idle 3 192.168.0.52 Ready
Machine State Address Inst id Base AZ Message
0 started 192.168.0.82 91545e2c-0bbc-475d-9528-fd4742efa0b3 ubuntu@22.04 nova ACTIVE
0/lxd/0 started 252.82.3.157 juju-572a8e-0-lxd-0 ubuntu@22.04 nova Container started
1 started 192.168.0.68 4c3aaf88-05fc-4de2-95fb-d7abaf75535d ubuntu@22.04 nova ACTIVE
3 started 192.168.0.52 386403bf-ed3d-4efd-8206-4d77693a7e29 ubuntu@22.04 nova ACTIVE取得 kubeconfig
juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config此時 kubectl get pods -A 的輸出應該會有這些 pods：
ubuntu@juju-572a8e-k8s-0:~$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx-kubernetes-worker default-http-backend-kubernetes-worker-5c79cc75ff-cvqw7 1/1 Running 0 14m
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-bc7zc 1/1 Running 0 12m
kube-system cilium-7ndz7 1/1 Running 0 14m
kube-system cilium-operator-577bfbbd5b-5fmvj 1/1 Running 0 14m
kube-system cilium-operator-577bfbbd5b-8d4m4 1/1 Running 0 14m
kube-system cilium-zb7dp 1/1 Running 0 14m
kube-system coredns-59cfb5bf46-6tpcg 1/1 Running 0 16m
kube-system csi-cinder-controllerplugin-684cfb8c48-6qcxp 6/6 Running 0 16m
kube-system csi-cinder-nodeplugin-7pxjl 3/3 Running 0 14m
kube-system csi-cinder-nodeplugin-wsp9z 3/3 Running 0 15m
kube-system hubble-generate-certs-394f790584-t7j48 0/1 Completed 0 16m
kube-system kube-state-metrics-78c475f58b-8cjvv 1/1 Running 0 16m
kube-system metrics-server-v0.6.3-69d7fbfdf8-xc2xv 2/2 Running 0 16m
kube-system openstack-cloud-controller-manager-gdgng 1/1 Running 0 2m24s
kubernetes-dashboard dashboard-metrics-scraper-5dd7cb5fc-bjq29 1/1 Running 0 16m
kubernetes-dashboard kubernetes-dashboard-7b899cb9d9-kxmmt 1/1 Running 0 16m測試 OpenStack Integrator
最後測試 OpenStack Integrator 正常運作
Storage Integration
建立 PVC
kubectl create -f - <<EOY
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: testclaim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
storageClassName: cdk-cinder
EOYkubectl get pv 應該可以見到 pv 被建立
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-d302df77-7cbc-4a7b-af7f-5373f91abbd3 1Gi RWO Delete Bound default/testclaim cdk-cinder 15sopenstack volume list 可以看到 cinder 建立了 volume
+--------------------------------------+------------------------------------------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+------------------------------------------+-----------+------+-------------+
| 37734a31-5786-48c2-9757-f4782e6cdfd6 | pvc-d302df77-7cbc-4a7b-af7f-5373f91abbd3 | available | 1 | |
+--------------------------------------+------------------------------------------+-----------+------+-------------+Load Balancer Integration
建立測試的 pods 並且透過 LB expose
kubectl create deployment hello-world --image=gcr.io/google-samples/node-hello:1.0
kubectl scale deployment hello-world --replicas=5
kubectl expose deployment hello-world --type=LoadBalancer --name=hello --port=8080此時 Load Balancer 將被建立，可以透過 kubectl get svc hello -o wide 看 external IP
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
hello LoadBalancer 10.152.183.41 192.168.99.144 8080:30777/TCP 6m16s app=hello-world透過 external IP 可以連到服務
curl 192.168.99.144:8080
Hello Kubernetes!openstack loadbalancer list 也可以看到 load balancer 被建立
openstack loadbalancer list
+--------------------------------------+------------------------------------------------------------------------+----------------------------------+--------------+---------------------+------------------+----------+
| id | name | project_id | vip_address | provisioning_status | operating_status | provider |
+--------------------------------------+------------------------------------------------------------------------+----------------------------------+--------------+---------------------+------------------+----------+
| 4cb1c8da-3c71-4fcf-9b13-23f6f21e0336 | openstack-integrator-5a087e572a8e-kubernetes-control-plane | 4badc745662a485b8957de81ae403ee2 | 192.168.0.78 | ACTIVE | ONLINE | ovn |
| 5cc3a0ce-b798-4b38-a1aa-33f637327560 | kube_service_kubernetes-df70v6ftc5r56zmdyd68zps0cwdmizal_default_hello | 4badc745662a485b8957de81ae403ee2 | 192.168.0.46 | ACTIVE | ONLINE | ovn |
+--------------------------------------+------------------------------------------------------------------------+----------------------------------+--------------+---------------------+------------------+----------+小結
Charmed Kubernetes 在部署上並不會太困難，預設上也幫你安裝了一些實用的 add-ons，如 ingress-nginx 等，不過在設定上的彈性筆者覺得沒有比 Kops 來得好。
想暸解如何利用 Kops 部署 Kubernetes 可以參考這篇文章。
Reference
- https://charmhub.io/openstack-integrator?channel=1.28/stable
- https://ubuntu.com/kubernetes/docs/install-manual
- https://ubuntu.com/kubernetes/charmed-k8s
Copyright Notice: All articles in this blog are licensed under CC BY-NC-SA 4.0 unless stating additionally.