OPSaid – Kubernetes

Docs: etcd

Thu, 03 Aug 2023 09:42:05 +0800

简要概述

记录 etcd 相关的场景问题。

无法启动类

“received signal; shutting down”

关键日志

{"level":"info","ts":"2023-07-27T13:31:34.277Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"edacac26a3fb24b3","remote-peer-id":"2c8b55a3459bb7ff"}
{"level":"info","ts":"2023-07-27T13:33:03.898Z","caller":"osutil/interrupt_unix.go:64","msg":"received signal; shutting down","signal":"terminated"}
{"level":"info","ts":"2023-07-27T13:33:03.898Z","caller":"embed/etcd.go:367","msg":"closing etcd server","name":"node3","data-dir":"/data/etcd/","advertise-peer-urls":["https://10.49.2.36:2380"],"advertise-client-urls":["https://10.49.2.36:2379"]}

问题原因

在文件 /etc/containerd/config.toml 中的 SystemdCgroup 配置为 false 了，与 kubelet 中不同。

解决步骤

更改配置，使用同 “kubelet” 一样的 cgroup 为 systemd

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
 SystemdCgroup = true

备份恢复类

控制节点 etcd 数据恢复

登录故障节点，移除 etcd 静态 POD

cd /etc/kubernetes/manifests/
mv etcd.yaml /usr/local/src/

登录另外正常的 etcd 集群节点

查看成员列表

ctr --namespace k8s.io run \
 --rm \
 --net-host \
 --mount type=bind,src=/etc/kubernetes,dst=/etc/kubernetes,options=rbind:ro \
 registry.cn-hangzhou.aliyuncs.com/kube-image-repo/etcd:v3.5.10 etcd-health \
 etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt \
 --key /etc/kubernetes/pki/etcd/peer.key \
 --cacert /etc/kubernetes/pki/etcd/ca.crt \
 --endpoints https://127.0.0.1:2379 \
 member list

移除异常的成员

ctr --namespace k8s.io run \
 --rm \
 --net-host \
 --mount type=bind,src=/etc/kubernetes,dst=/etc/kubernetes,options=rbind:ro \
 registry.cn-hangzhou.aliyuncs.com/kube-image-repo/etcd:v3.5.10 etcd-health \
 etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt \
 --key /etc/kubernetes/pki/etcd/peer.key \
 --cacert /etc/kubernetes/pki/etcd/ca.crt \
 --endpoints https://127.0.0.1:2379 \
 member remove 63e2da0114a7b9b7

更改故障节点 etcd 作为新成员加入

ctr --namespace k8s.io run \
 --rm \
 --net-host \
 --mount type=bind,src=/etc/kubernetes,dst=/etc/kubernetes,options=rbind:ro \
 registry.cn-hangzhou.aliyuncs.com/kube-image-repo/etcd:v3.5.10 etcd-health \
 etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt \
 --key /etc/kubernetes/pki/etcd/peer.key \
 --cacert /etc/kubernetes/pki/etcd/ca.crt \
 --endpoints https://127.0.0.1:2379 \
 member add node2 --peer-urls="https://192.168.201.122:2380"

ETCD_NAME="node2"
ETCD_INITIAL_CLUSTER="node3=https://192.168.201.123:2380,node2=https://192.168.201.122:2380,node1=https://192.168.201.121:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.201.122:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

启动故障节点的 etcd 服务

--initial-cluster-state=new

更改为

--initial-cluster-state=existing

Docs: kube-apiserver

Mon, 06 May 2024 14:05:48 +0800

简要概述

记录 kube-apiserver 相关问题。

无法启动类

“dial tcp 127.0.0.1:8080: connect: connection refused”

关键日志

root@k8s/etc/kubernetes/manifests# kubectl get nodes
E0803 10:01:41.668372 300010 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0803 10:01:41.669287 300010 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0803 10:01:41.671326 300010 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0803 10:01:41.672078 300010 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0803 10:01:41.674043 300010 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?
root@k8s/etc/kubernetes/manifests#

问题原因

新版不在开放本地的 “127.0.0.1:8080” 端口，需配置下 kubeconfig 路径，如：$HOME/.kube/config

“failed to verify certificate: x509”

关键日志

root@k8s/etc/kubernetes/manifests# kubectl get nodes
E0803 10:27:46.079494 300470 memcache.go:265] couldn't get current server API group list: Get "https://10.49.2.108:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
E0803 10:27:46.094227 300470 memcache.go:265] couldn't get current server API group list: Get "https://10.49.2.108:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
E0803 10:27:46.105654 300470 memcache.go:265] couldn't get current server API group list: Get "https://10.49.2.108:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
E0803 10:27:46.117784 300470 memcache.go:265] couldn't get current server API group list: Get "https://10.49.2.108:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
E0803 10:27:46.127952 300470 memcache.go:265] couldn't get current server API group list: Get "https://10.49.2.108:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority
root@k8s/etc/kubernetes/manifests#

问题原因

在 “$HOME/.kube/config” 中内嵌的 ca 证书不对，检查方式：

获取 “$HOME/.kube/config” 中 “certificate-authority-data” 内容，使用 base64 反解：

echo -n '${certificate-authority-data}' | base64 -d

资源无法释放

命名空间无法删除

NAME STATUS AGE
argo Active 3d21h
argo-events Terminating 6d
argocd Active 3d22h

命名空间 “argo-events” 无法删除，持续处于 “Terminating” 状态，这种情况一般是因为相关的 crd 资源没有删除干净，此时通过查询 etcd 中键值。

ETCDCTL_API=3 etcdctl \
 --endpoints https://127.0.0.1:2379 \
 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
 --cert=/etc/kubernetes/pki/etcd/server.crt \
 --key=/etc/kubernetes/pki/etcd/server.key \
 get / --prefix --keys-only | grep argo-events

/kubernetes/fake/registry/argoproj.io/eventbus/argo-events/default
/kubernetes/fake/registry/argoproj.io/eventsources/argo-events/oneops-syncmi-v1
/kubernetes/fake/registry/argoproj.io/sensors/argo-events/log
/kubernetes/fake/registry/clusterrolebindings/argo-events-binding
/kubernetes/fake/registry/clusterrolebindings/argo-events-webhook-binding
/kubernetes/fake/registry/clusterroles/argo-events-aggregate-to-admin
/kubernetes/fake/registry/clusterroles/argo-events-aggregate-to-edit
/kubernetes/fake/registry/clusterroles/argo-events-aggregate-to-view
/kubernetes/fake/registry/clusterroles/argo-events-role
/kubernetes/fake/registry/clusterroles/argo-events-webhook
/kubernetes/fake/registry/namespaces/argo-events

通过键值可知还有哪些资源存在，此时只要把以下自定义资源移除，命名空间也就自动删除了。

# /kubernetes/fake/registry/argoproj.io/sensors/argo-events/log
# /kubernetes/fake/registry/argoproj.io/eventsources/argo-events/oneops-syncmi-v1
# /kubernetes/fake/registry/argoproj.io/eventbus/argo-events/default

ETCDCTL_API=3 etcdctl \
 --endpoints https://127.0.0.1:2379 \
 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
 --cert=/etc/kubernetes/pki/etcd/server.crt \
 --key=/etc/kubernetes/pki/etcd/server.key \
 del /kubernetes/fake/registry/argoproj.io/eventbus/argo-events/default

基础配置更改

更换 apiserver.crt 证书

因需更换 vip 等原因，需要对证书重新签发更换，可通过以下步骤。

进入部署仓库

cd ./ansible

变更 vip 为新地址

vim group_vars/all.yaml

......
# 集群南北流量高可用的ip地址
# 一般为vip，没有则设置为控制平面任意一个实际的ip地址
#apiserver_haip: 192.168.17.139
apiserver_haip: 10.155.17.139
......

备份旧证书

mv roles/pki/files/apiserver/apiserver.crt roles/pki/files/apiserver/apiserver.crt.bak
mv roles/pki/files/apiserver/apiserver.key roles/pki/files/apiserver/apiserver.key.bak

重新生成证书

ansible-playbook -i hosts.ini deploy.yaml -t pki

检查证书是否符合期望

openssl x509 -in roles/pki/files/apiserver/apiserver.crt -noout -text | less

循环切换节点使用新证书

ansible-playbook -i hosts.ini deploy.yaml -t control,worker

是否 /etc/hosts 绑定了域名

确认各节点上 /etc/hosts 是否静态配置了 vip ，如果有手工更改为最新地址下。

重启 kube-apiserver 与各 kubelet 服务

for i in $(kubectl get pods -n kube-system | grep kube-apiserver | awk '{ print $1 }')
do
 kubectl delete pod $i -n kube-system
done

ansible kube_control_plane,kube_node -i hosts.ini -m shell -a "systemctl restart kubelet"

验证证书情况

kubeadm certs check-expiration