OPSaid – 集群部署

Docs: 阿里云

Tue, 16 Apr 2024 10:20:22 +0800

简要概述

本文主要介绍如何使用云服务器ECS 自建 K8S 集群，相比官方 ACK 产品有以下优势：

控制更加灵活。
没有机型限制，如 ENI 等。

如非特殊情况，线上业务均建议直接购买官方 ACK 产品。

注意：请确保安全组允许以下端口访问，如果其他请自行开通。

协议	端口	用途
TCP	6443	kube-apiserver
TCP	10250	metric-server

云控制器

用于将你的集群连接到云提供商的 API 之上，可以同步路由、Service、Ingress 等创建对应云平台上资源。

通过分离 Kubernetes 和底层云基础设置之间的互操作性逻辑，cloud-controller-manager 组件使云提供商能够以不同于 Kubernetes 主项目的步调发布新特征：

集群网络模式

方案一：在 VPC 添加路由

假设以下配置：

节点名	归属 VPC	归属交换机	宿主 IP	cluster_cidr	pod cidr
node1	10.0.0.0/8	10.100.197.0/24	10.100.197.203	192.168.201.0/24	192.168.201.0/28
node2	10.0.0.0/8	10.100.197.0/24	10.100.197.204	192.168.201.0/24	192.168.201.16/28
node3	10.0.0.0/8	10.100.197.0/24	10.100.197.205	192.168.201.0/24	192.168.201.32/28
node4	10.0.0.0/8	10.100.197.0/24	10.100.197.206	192.168.201.0/24	192.168.201.48/28

这样需要在 VPC 上添加路由：

192.168.201.0/28 via node1
192.168.201.16/28 via node2
192.168.201.32/28 via node3
192.168.201.48/28 via node4

在 VPC 上添加路由有以下两种方式：

规划后集群大小后，在阿里云后台 “专有网络/路由表” 中手动添加；
配合云控制器阿里云 CCM 组件自动添加发布主机路由至 VPC。

方案二：独立 VPC 子网

既在阿里云上创建单独的 VPC 字网用于 POD 网段。

该方式依赖主机规格以及可支持的弹性网卡 ENI 数量限制，如需使用建议直接购买 ACK 产品。

方案三：使用封装形式

这种性能低，暂不考虑。

注意事项

组件 CCM 正常但污点未移除

根据 Kubernetes 运行云管理控制器约定，当 kubelet 配置了 “–cloud-provider=external” 则在节点启动时会自动添加 “node.cloudprovider.kubernetes.io/uninitialized” 污点，当且仅当 CCM 组件运行成功后，会自动移除且设置节点可调度。

如果出现 CCM 组件正常，但各节点污点均未被移除，在排除权限等问题后，可检查配置 “–controllers” 是否有开启 “node” 控制器功能，否则即使 ccm 组件正常运行，节点的污点也是无法被组件移除的。

apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: cloud-controller-manager
 namespace: kube-system
spec:
 template:
 spec:
......
 containers:
 - command:
 - /cloud-controller-manager
 - --kubeconfig=/etc/kubernetes/cloud-controller-manager.conf
 - --cloud-config=/etc/kubernetes/config/cloud-config.conf
 # 支持 node,route,service,ingress
 - --controllers=node,route
......

同时以下几个公知标签也不能被自动添加：

beta.kubernetes.io/instance-type
node.kubernetes.io/instance-type

failure-domain.beta.kubernetes.io/region
failure-domain.beta.kubernetes.io/zone

topology.kubernetes.io/zone
topology.kubernetes.io/region

Docs: Init

Fri, 02 Dec 2022 15:04:36 +0800

简要概述

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration

代码路径：

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1beta3/types.go

配置示例

TODO;

数据结构

InitConfiguration

// InitConfiguration contains a list of elements that is specific "kubeadm init"-only runtime
// information.
type InitConfiguration struct {
 metav1.TypeMeta `json:",inline"`

 // `kubeadm init`-only information. These fields are solely used the first time `kubeadm init` runs.
 // After that, the information in the fields IS NOT uploaded to the `kubeadm-config` ConfigMap
 // that is used by `kubeadm upgrade` for instance. These fields must be omitempty.

 // BootstrapTokens is respected at `kubeadm init` time and describes a set of Bootstrap Tokens to create.
 // This information IS NOT uploaded to the kubeadm cluster configmap, partly because of its sensitive nature
 // +optional
 BootstrapTokens []bootstraptokenv1.BootstrapToken `json:"bootstrapTokens,omitempty"`

 // NodeRegistration holds fields that relate to registering the new control-plane node to the cluster
 // +optional
 NodeRegistration NodeRegistrationOptions `json:"nodeRegistration,omitempty"`

 // LocalAPIEndpoint represents the endpoint of the API server instance that's deployed on this control plane node
 // In HA setups, this differs from ClusterConfiguration.ControlPlaneEndpoint in the sense that ControlPlaneEndpoint
 // is the global endpoint for the cluster, which then loadbalances the requests to each individual API server. This
 // configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible
 // on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process
 // fails you may set the desired value here.
 // +optional
 LocalAPIEndpoint APIEndpoint `json:"localAPIEndpoint,omitempty"`

 // CertificateKey sets the key with which certificates and keys are encrypted prior to being uploaded in
 // a secret in the cluster during the uploadcerts init phase.
 // +optional
 CertificateKey string `json:"certificateKey,omitempty"`

 // SkipPhases is a list of phases to skip during command execution.
 // The list of phases can be obtained with the "kubeadm init --help" command.
 // The flag "--skip-phases" takes precedence over this field.
 // +optional
 SkipPhases []string `json:"skipPhases,omitempty"`

 // Patches contains options related to applying patches to components deployed by kubeadm during
 // "kubeadm init".
 // +optional
 Patches *Patches `json:"patches,omitempty"`
}

bootstraptokenv1.BootstrapToken

bootstraptokenv1 "k8s.io/kubernetes/cmd/kubeadm/app/apis/bootstraptoken/v1"

// BootstrapToken describes one bootstrap token, stored as a Secret in the cluster
// +k8s:deepcopy-gen=true
type BootstrapToken struct {
 // Token is used for establishing bidirectional trust between nodes and control-planes.
 // Used for joining nodes in the cluster.
 Token *BootstrapTokenString `json:"token" datapolicy:"token"`
 // Description sets a human-friendly message why this token exists and what it's used
 // for, so other administrators can know its purpose.
 // +optional
 Description string `json:"description,omitempty"`
 // TTL defines the time to live for this token. Defaults to 24h.
 // Expires and TTL are mutually exclusive.
 // +optional
 TTL *metav1.Duration `json:"ttl,omitempty"`
 // Expires specifies the timestamp when this token expires. Defaults to being set
 // dynamically at runtime based on the TTL. Expires and TTL are mutually exclusive.
 // +optional
 Expires *metav1.Time `json:"expires,omitempty"`
 // Usages describes the ways in which this token can be used. Can by default be used
 // for establishing bidirectional trust, but that can be changed here.
 // +optional
 Usages []string `json:"usages,omitempty"`
 // Groups specifies the extra groups that this token will authenticate as when/if
 // used for authentication
 // +optional
 Groups []string `json:"groups,omitempty"`
}

// BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used
// for both validation of the practically of the API server from a joining node's point
// of view and as an authentication method for the node in the bootstrap phase of
// "kubeadm join". This token is and should be short-lived
type BootstrapTokenString struct {
 ID string `json:"-"`
 Secret string `json:"-" datapolicy:"token"`
}

NodeRegistrationOptions

// NodeRegistrationOptions holds fields that relate to registering a new control-plane or node to the cluster, either via "kubeadm init" or "kubeadm join"
type NodeRegistrationOptions struct {

 // Name is the `.Metadata.Name` field of the Node API object that will be created in this `kubeadm init` or `kubeadm join` operation.
 // This field is also used in the CommonName field of the kubelet's client certificate to the API server.
 // Defaults to the hostname of the node if not provided.
 // +optional
 Name string `json:"name,omitempty"`

 // CRISocket is used to retrieve container runtime info. This information will be annotated to the Node API object, for later re-use
 // +optional
 CRISocket string `json:"criSocket,omitempty"`

 // Taints specifies the taints the Node API object should be registered with. If this field is unset, i.e. nil, in the `kubeadm init` process
 // it will be defaulted to []v1.Taint{'node-role.kubernetes.io/master=""'}. If you don't want to taint your control-plane node, set this field to an
 // empty slice, i.e. `taints: []` in the YAML file. This field is solely used for Node registration.
 Taints []corev1.Taint `json:"taints"`

 // KubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file
 // kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the kubelet-config-1.X ConfigMap
 // Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on.
 // A key in this map is the flag name as it appears on the
 // command line except without leading dash(es).
 // +optional
 KubeletExtraArgs map[string]string `json:"kubeletExtraArgs,omitempty"`

 // IgnorePreflightErrors provides a slice of pre-flight errors to be ignored when the current node is registered.
 // +optional
 IgnorePreflightErrors []string `json:"ignorePreflightErrors,omitempty"`

 // ImagePullPolicy specifies the policy for image pulling during kubeadm "init" and "join" operations.
 // The value of this field must be one of "Always", "IfNotPresent" or "Never".
 // If this field is unset kubeadm will default it to "IfNotPresent", or pull the required images if not present on the host.
 // +optional
 ImagePullPolicy corev1.PullPolicy `json:"imagePullPolicy,omitempty"`
}

APIEndpoint

// APIEndpoint struct contains elements of API server instance deployed on a node.
type APIEndpoint struct {
 // AdvertiseAddress sets the IP address for the API server to advertise.
 // +optional
 AdvertiseAddress string `json:"advertiseAddress,omitempty"`

 // BindPort sets the secure port for the API Server to bind to.
 // Defaults to 6443.
 // +optional
 BindPort int32 `json:"bindPort,omitempty"`
}

Docs: Etcd

Thu, 06 Oct 2022 10:54:31 +0800

简要概述

通过 kubeadm 工具创建高可用 etcd 集群。

执行步骤

启动 containerd

参考：

启动 kubelet

参考：

cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
[Service]
ExecStart=
# Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
# Replace the value of "--container-runtime-endpoint" for a different container runtime if needed.
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock
Restart=always
EOF
systemctl daemon-reload
systemctl restart kubelet

生成 etcd ca

kubeadm init phase certs etcd-ca

目录：/etc/kubernetes/pki/etcd/

实际等效于：

[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
CN = etcd-ca

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = etcd-ca

[ v3_ext ]
keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign
basicConstraints=critical,CA:true,pathlen:1
subjectKeyIdentifier=hash
subjectAltName=@alt_names

这里添加了pathlen:1限制证实链

生成 etcd 配置

#!/bin/bash

# Update HOST0, HOST1 and HOST2 with the IPs of your hosts
export HOST0=192.168.30.201
export HOST1=192.168.30.202
export HOST2=192.168.30.203

# Update NAME0, NAME1 and NAME2 with the hostnames of your hosts
export NAME0="infra0"
export NAME1="infra1"
export NAME2="infra2"

# Create temp directories to store files that will end up on other hosts.
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

HOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=(${NAME0} ${NAME1} ${NAME2})

for i in "${!HOSTS[@]}"; do
HOST=${HOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
---
apiVersion: "kubeadm.k8s.io/v1beta3"
kind: InitConfiguration
nodeRegistration:
 name: ${NAME}
localAPIEndpoint:
 advertiseAddress: ${HOST}
---
apiVersion: "kubeadm.k8s.io/v1beta3"
kind: ClusterConfiguration
etcd:
 local:
 serverCertSANs:
 - "${HOST}"
 peerCertSANs:
 - "${HOST}"
 extraArgs:
 initial-cluster: ${NAMES[0]}=https://${HOSTS[0]}:2380,${NAMES[1]}=https://${HOSTS[1]}:2380,${NAMES[2]}=https://${HOSTS[2]}:2380
 initial-cluster-state: new
 name: ${NAME}
 listen-peer-urls: https://${HOST}:2380
 listen-client-urls: https://${HOST}:2379
 advertise-client-urls: https://${HOST}:2379
 initial-advertise-peer-urls: https://${HOST}:2380
EOF
done

为每个etcd成员生成证书

kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST2}/
# cleanup non-reusable certificates
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
cp -R /etc/kubernetes/pki /tmp/${HOST1}/
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
# No need to move the certs because they are for HOST0

# clean up certs that should not be copied off this host
find /tmp/${HOST2} -name ca.key -type f -delete
find /tmp/${HOST1} -name ca.key -type f -delete

传输各个etcd证书至各节点

scp -r /tmp/${HOST}/* ${USER}@${HOST}:/etc/kubernetes/

生成 etcd pod

各个节点执行

kubeadm init phase etcd local --config /etc/kubernetes/kubeadmcfg.yaml

查看etcd集群状态

docker

docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes registry.k8s.io/etcd:${ETCD_TAG} etcdctl \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://${HOST0}:2379 endpoint health --cluster

containerd

ctr --namespace k8s.io run \
 --rm \
 --net-host \
 --mount type=bind,src=/etc/kubernetes,dst=/etc/kubernetes,options=rbind:ro \
 registry.cn-hangzhou.aliyuncs.com/kube-image-repo/etcd:3.5.0-0 etcd-health \
 etcdctl --cert /etc/kubernetes/pki/etcd/peer.crt \
 --key /etc/kubernetes/pki/etcd/peer.key \
 --cacert /etc/kubernetes/pki/etcd/ca.crt \
 --endpoints https://192.168.30.201:2379 \
 endpoint health --cluster

Docs: 快速开始

Wed, 22 Dec 2021 16:46:49 +0800

简要概述

k8s生态组件过多，对新人入门并不友好，本项目在于结合当前生产集群维护经验，总结一套标准的部署模版，接入内部gitlab ci实现集群基础设施的自动化管理，同时也会不断迭代优化，以实现快速搭建一套生产可用的环境。

前置说明

操作系统

操作系统支持：ubuntu 20.04 或 debian11

ubuntu 20.04

选择最小化，同时安装sshd服务

主机拓扑

角色	建议数量	CPU架构	示例IP
ansible中控	1	amd64	192.168.31.200
负载均衡器	至少1	amd64	192.168.31.200
k8s控制节点	1、3、5	amd64	192.168.31.201/202/203
k8s工作节点	至少1	amd64	192.168.31.204

中控必须配置好免密ssh登录其他节点，如果仅用于部署测试负载均衡器可与中控共用，生产环境负载均衡器必须至少2节点且支持VIP。

环境部署

ansible中控

在中控执行以下步骤：

安装基础软件

ubuntu20.04

apt install -y git ansible

获取部署代码

git clone https://github.com/opsaid/deploy-k8s.git

配置ssh免密登录

ssh-copy-id -p 22 root@192.168.31.20X

负载均衡器

TODO

部署控制平面

步骤1：首次初始化前配置

必要更改的文件

ansible/hosts.ini
ansible/group_vars/all.yaml

可选更改的文件

ansible/group_vars/default.yaml
ansible/group_vars/component-v1.22.yaml
ansible/group_vars/component-v1.26.yaml

选择 component-v1.22 或 compoent-v1.26 是由部署的 k8s 版本所决定。

云提供商个性参数

ansible/group_vars/cloud-provider/private.yaml
ansible/group_vars/cloud-provider/alibaba-cloud.yaml

根据目标主机提供商上，按需更改相应参数。

步骤2：下载依赖的镜像包

方式1：预先下载安装包（推荐）

二进制文件包与镜像包

https://github.com/opsaid/deploy-k8s/releases

方式2：在线下载构建

# 由于网络原因，需要代理 HTTPS 配置用于加速通道传输（127.0.0.1:1087仅为示例）
#export HTTPS_PROXY=http://127.0.0.1:1087

yum install -y docker
systemctl start docker

# 编译制作 pause 镜像需要
yum install -y gcc glibc-static

ansible-playbook -i hosts.ini deploy.yaml -t binary

步骤3：生成集群依赖证书

ansible-playbook -i hosts.ini deploy.yaml -t pki

步骤4：系统公共组件安装

ansible-playbook -i hosts.ini deploy.yaml -t common

确保主机节点已在 host.ini 配置项 [kube_control_plane] 中

步骤5：部署控制平面节点

ansible-playbook -i hosts.ini deploy.yaml -t control

确保主机节点已在 host.ini 配置项 [kube_control_plane] 中

步骤6：验证控制平面组件

登录任一控制节点，查看关键组件的运行状态（可能需等待一段时间）

通过 crictl 查看本节点容器

crictl ps

其中"kube-apiserver"、“kube-controller-manager”、“etcd”、“kube-scheduler"组件，必须是"Running"状态。

通过 kubectl 查看所有节点容器

kubectl get pods -n kube-system

其中"kube-apiserver”、“kube-controller-manager”、“etcd”、“kube-scheduler"组件，在各节点必须是"Running"状态。

通过 kubectl 为节点添加标签

当控制平面 Pod 均正常运行时，为控制平面节点添加特殊标签：

for i in $(kubectl get node -o=template --template='{{ range .items }} {{ .metadata.name }} {{end }}')
do
 kubectl label node $i node-role.kubernetes.io/master=
 kubectl label node $i node-role.kubernetes.io/control-plane=
done

通过 kubectl 查看节点状态

kubectl get nodes

确认节点列表 “ROLES” 中均包含 “control-plane,master” 内容。

步骤7：部署集群基础组件

在 k8s 1.26 及之后使用 “cilium” 作为 CNI 并替换 “calico”、“kube-proxy”、“ingress-nginx” 组件功能。

登录任意一个控制平面节点，部署 addons

部署 CNI

kubectl apply -k /etc/kubernetes/addons/cilium/

部署 kube-system 权限等

kubectl apply -k /etc/kubernetes/addons/kube-system/

部署 CoreDNS 等

kubectl apply -k /etc/kubernetes/addons/coredns/
kubectl apply -k /etc/kubernetes/addons/metrics-server/

也可使用 kustomize 替代 kubectl apply -k，如：kustomize build /etc/kubernetes/addons/coredns | kubectl apply -f -

步骤8：本地配置存储集群

存储 kubeadm 配置至集群 ConfigMap

其中配置取 “ClusterConfiguration” 类型同步至 “kube-system” 下 ConfigMap “kubeadm-config”

kubeadm init phase upload-config kubeadm --config /etc/kubernetes/kubeadm/config.yaml

存储 kubelet 配置至集群 ConfigMap

其中 kubelet 配置取 KubeletConfiguration 同步至 “kube-system” 下 ConfigMap “kubelet-config”

kubeadm init phase upload-config kubelet --config /etc/kubernetes/kubeadm/config.yaml

存储证书至空间 “kube-system” 下 Secret 资源

kubeadm init phase upload-certs --upload-certs --config /etc/kubernetes/kubeadm/config.yaml

其中 “kubeadm-certs” 包含以下几个证书文件：

etcd-ca.crt
etcd-ca.key
front-proxy-ca.crt
front-proxy-ca.key
sa.key
sa.pub
ca.crt
ca.key

步骤9：控制平面相关维护

见常见问题。

新增工作节点

步骤1：在ansible的`host.ini`中添加节点

[all]
......
node4 ansible_host=192.168.31.204 ansible_port=22 ip=192.168.31.204

[kube_node]
......
node4

步骤2：ansible中控节点可ssh节点

$HOME/.ssh/authorized_keys

步骤3：系统公共组件安装

ansible-playbook -i hosts.ini deploy.yaml -t common

步骤4：添加集群工作节点

ansible-playbook -i hosts.ini deploy.yaml -t worker

步骤5：验证允许节点加入

查看节点加入请求

kubectl get csr

其中"CONDITION”=“Pending"状态；

授权制定节点的加入

kubectl certificate approve ${node-csr-XXXX}

等待节点加入并状态为可用

kubectl get nodes

在任一控制平面节点执行

完成验收与后续

TODO

使用 calico 作为 CNI

kustomize build /etc/kubernetes/addons/calico/ | kubectl apply -f -

其他可选

kustomize build /etc/kubernetes/addons/ingress-nginx/ | kubectl apply -f -
kustomize build /etc/kubernetes/addons/kube-proxy/ | kubectl apply -f -

Docs: Cluster

Fri, 02 Dec 2022 13:52:44 +0800

简要概述

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration

代码路径：

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1beta3/types.go

数据结构

ClusterConfiguration

// ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster
type ClusterConfiguration struct {
 metav1.TypeMeta `json:",inline"`

 // Etcd holds configuration for etcd.
 // +optional
 Etcd Etcd `json:"etcd,omitempty"`

 // Networking holds configuration for the networking topology of the cluster.
 // +optional
 Networking Networking `json:"networking,omitempty"`

 // KubernetesVersion is the target version of the control plane.
 // +optional
 KubernetesVersion string `json:"kubernetesVersion,omitempty"`

 // ControlPlaneEndpoint sets a stable IP address or DNS name for the control plane; it
 // can be a valid IP address or a RFC-1123 DNS subdomain, both with optional TCP port.
 // In case the ControlPlaneEndpoint is not specified, the AdvertiseAddress + BindPort
 // are used; in case the ControlPlaneEndpoint is specified but without a TCP port,
 // the BindPort is used.
 // Possible usages are:
 // e.g. In a cluster with more than one control plane instances, this field should be
 // assigned the address of the external load balancer in front of the
 // control plane instances.
 // e.g. in environments with enforced node recycling, the ControlPlaneEndpoint
 // could be used for assigning a stable DNS to the control plane.
 // +optional
 ControlPlaneEndpoint string `json:"controlPlaneEndpoint,omitempty"`

 // APIServer contains extra settings for the API server control plane component
 // +optional
 APIServer APIServer `json:"apiServer,omitempty"`

 // ControllerManager contains extra settings for the controller manager control plane component
 // +optional
 ControllerManager ControlPlaneComponent `json:"controllerManager,omitempty"`

 // Scheduler contains extra settings for the scheduler control plane component
 // +optional
 Scheduler ControlPlaneComponent `json:"scheduler,omitempty"`

 // DNS defines the options for the DNS add-on installed in the cluster.
 // +optional
 DNS DNS `json:"dns,omitempty"`

 // CertificatesDir specifies where to store or look for all required certificates.
 // +optional
 CertificatesDir string `json:"certificatesDir,omitempty"`

 // ImageRepository sets the container registry to pull images from.
 // If empty, `k8s.gcr.io` will be used by default; in case of kubernetes version is a CI build (kubernetes version starts with `ci/` or `ci-cross/`)
 // `gcr.io/k8s-staging-ci-images` will be used as a default for control plane components and for kube-proxy, while `k8s.gcr.io`
 // will be used for all the other images.
 // +optional
 ImageRepository string `json:"imageRepository,omitempty"`

 // FeatureGates enabled by the user.
 // +optional
 FeatureGates map[string]bool `json:"featureGates,omitempty"`

 // The cluster name
 // +optional
 ClusterName string `json:"clusterName,omitempty"`
}

Etcd

// Etcd contains elements describing Etcd configuration.
type Etcd struct {

 // Local provides configuration knobs for configuring the local etcd instance
 // Local and External are mutually exclusive
 // +optional
 Local *LocalEtcd `json:"local,omitempty"`

 // External describes how to connect to an external etcd cluster
 // Local and External are mutually exclusive
 // +optional
 External *ExternalEtcd `json:"external,omitempty"`
}

// LocalEtcd describes that kubeadm should run an etcd cluster locally
type LocalEtcd struct {
 // ImageMeta allows to customize the container used for etcd
 ImageMeta `json:",inline"`

 // DataDir is the directory etcd will place its data.
 // Defaults to "/var/lib/etcd".
 DataDir string `json:"dataDir"`

 // ExtraArgs are extra arguments provided to the etcd binary
 // when run inside a static pod.
 // A key in this map is the flag name as it appears on the
 // command line except without leading dash(es).
 // +optional
 ExtraArgs map[string]string `json:"extraArgs,omitempty"`

 // ServerCertSANs sets extra Subject Alternative Names for the etcd server signing cert.
 // +optional
 ServerCertSANs []string `json:"serverCertSANs,omitempty"`
 // PeerCertSANs sets extra Subject Alternative Names for the etcd peer signing cert.
 // +optional
 PeerCertSANs []string `json:"peerCertSANs,omitempty"`
}

// ExternalEtcd describes an external etcd cluster.
// Kubeadm has no knowledge of where certificate files live and they must be supplied.
type ExternalEtcd struct {
 // Endpoints of etcd members. Required for ExternalEtcd.
 Endpoints []string `json:"endpoints"`

 // CAFile is an SSL Certificate Authority file used to secure etcd communication.
 // Required if using a TLS connection.
 CAFile string `json:"caFile"`

 // CertFile is an SSL certification file used to secure etcd communication.
 // Required if using a TLS connection.
 CertFile string `json:"certFile"`

 // KeyFile is an SSL key file used to secure etcd communication.
 // Required if using a TLS connection.
 KeyFile string `json:"keyFile"`
}

Networking

// Networking contains elements describing cluster's networking configuration
type Networking struct {
 // ServiceSubnet is the subnet used by k8s services. Defaults to "10.96.0.0/12".
 // +optional
 ServiceSubnet string `json:"serviceSubnet,omitempty"`
 // PodSubnet is the subnet used by pods.
 // +optional
 PodSubnet string `json:"podSubnet,omitempty"`
 // DNSDomain is the dns domain used by k8s services. Defaults to "cluster.local".
 // +optional
 DNSDomain string `json:"dnsDomain,omitempty"`
}

APIServer

// APIServer holds settings necessary for API server deployments in the cluster
type APIServer struct {
 ControlPlaneComponent `json:",inline"`

 // CertSANs sets extra Subject Alternative Names for the API Server signing cert.
 // +optional
 CertSANs []string `json:"certSANs,omitempty"`

 // TimeoutForControlPlane controls the timeout that we use for API server to appear
 // +optional
 TimeoutForControlPlane *metav1.Duration `json:"timeoutForControlPlane,omitempty"`
}

ControlPlaneComponent

// ControlPlaneComponent holds settings common to control plane component of the cluster
type ControlPlaneComponent struct {
 // ExtraArgs is an extra set of flags to pass to the control plane component.
 // A key in this map is the flag name as it appears on the
 // command line except without leading dash(es).
 // TODO: This is temporary and ideally we would like to switch all components to
 // use ComponentConfig + ConfigMaps.
 // +optional
 ExtraArgs map[string]string `json:"extraArgs,omitempty"`

 // ExtraVolumes is an extra set of host volumes, mounted to the control plane component.
 // +optional
 ExtraVolumes []HostPathMount `json:"extraVolumes,omitempty"`
}

DNS

// DNS defines the DNS addon that should be used in the cluster
type DNS struct {
 // ImageMeta allows to customize the image used for the DNS component
 ImageMeta `json:",inline"`
}

HostPathMount

// HostPathMount contains elements describing volumes that are mounted from the
// host.
type HostPathMount struct {
 // Name of the volume inside the pod template.
 Name string `json:"name"`
 // HostPath is the path in the host that will be mounted inside
 // the pod.
 HostPath string `json:"hostPath"`
 // MountPath is the path inside the pod where hostPath will be mounted.
 MountPath string `json:"mountPath"`
 // ReadOnly controls write access to the volume
 // +optional
 ReadOnly bool `json:"readOnly,omitempty"`
 // PathType is the type of the HostPath.
 // +optional
 PathType corev1.HostPathType `json:"pathType,omitempty"`
}

ImageMeta

// ImageMeta allows to customize the image used for components that are not
// originated from the Kubernetes/Kubernetes release process
type ImageMeta struct {
 // ImageRepository sets the container registry to pull images from.
 // if not set, the ImageRepository defined in ClusterConfiguration will be used instead.
 // +optional
 ImageRepository string `json:"imageRepository,omitempty"`

 // ImageTag allows to specify a tag for the image.
 // In case this value is set, kubeadm does not change automatically the version of the above components during upgrades.
 // +optional
 ImageTag string `json:"imageTag,omitempty"`

 //TODO: evaluate if we need also a ImageName based on user feedbacks
}

Docs: 官方镜像

Sat, 22 Jan 2022 12:08:39 +0800

简要概述

这里说明所用到 k8s 官方镜像与相关的组件地址。

当在部署时，可以从外部获取依赖文件至以下路径：

ansible/roles/binary/files

控制平面

kubeadm config images list --kubernetes-version stable-1.22
kubeadm config images list --kubernetes-version stable-1.26

https://cdn.dl.k8s.io/release/stable-1.26.txt

k8s 1.26 版本

registry.k8s.io/kube-apiserver:v1.26.11
registry.k8s.io/kube-controller-manager:v1.26.11
registry.k8s.io/kube-scheduler:v1.26.11
registry.k8s.io/kube-proxy:v1.26.11
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.9.3

k8s 1.22 版本

k8s.gcr.io/kube-apiserver:v1.22.6
k8s.gcr.io/kube-controller-manager:v1.22.6
k8s.gcr.io/kube-scheduler:v1.22.6
k8s.gcr.io/kube-proxy:v1.22.6
k8s.gcr.io/pause:3.5
k8s.gcr.io/etcd:3.5.0-0
k8s.gcr.io/coredns/coredns:v1.8.4

基本术语

变量	取值范围	说明
KUBE_VERSION	见版本历史	安装k8s的版本，默认获取根目录下"version"文件内容
KUBERNETES_SERVER_ARCH	[amd64, arm, arm64, ppc64le, s390x]	服务端cpu架构，默认为amd64
KUBERNETES_CLIENT_OS	[linux, darwin, windows]	客户端的操作系统类型，默认为当前系统类型
KUBERNETES_CLIENT_ARCH	[amd64, arm, arm64, ppc64le, s390x, windows]	客户的cpu架构，默认为当前系统类型
KUBERNETES_SKIP_CONFIRM	任意值	如果存在该变量，下载则无需确认
KUBERNETES_RELEASE_URL	[https://dl.k8s.io]	下载的地址，格式如下
KUBERNETES_DOWNLOAD_TESTS	任意值	如果存在该变量，则下载测试用例

KUBERNETES_RELEASE_URL地址格式：

${KUBERNETES_RELEASE_URL}/${KUBE_VERSION}/kubernetes-server-${KUBERNETES_CLIENT_OS}-${KUBERNETES_SERVER_ARCH}.tar.gz
${KUBERNETES_RELEASE_URL}/${KUBE_VERSION}/kubernetes-client-${KUBERNETES_CLIENT_OS}-${KUBERNETES_SERVER_ARCH}.tar.gz

Kubernetes镜像

下载Releasing包

https://github.com/kubernetes/kubernetes/releases

选择需要的版本，展开"Assets"：

名称	说明
Kubernetes Source Code	编译后的结果
Source code (zip)	源代码zip压缩打包
Source code (tar.gz)	源代码以tar.gz压缩打包

点击下载 “Kubernetes Source Code” 获得文件 “kubernetes.tar.gz"，通过SHA256验证后，则解压至目录 “ansible/roles/binary/files” 里。

SHA256计算方式：
shasum -a 256 kubernetes.tar.gz

https://github.com/kubernetes/kubernetes/tree/master/build#releasing

获取二进制文件

cd kubernetes

export KUBERNETES_SERVER_ARCH=amd64
export KUBERNETES_CLIENT_OS=linux
export KUBERNETES_CLIENT_ARCH=amd64
export KUBERNETES_SKIP_CONFIRM=true
export KUBERNETES_RELEASE_URL=https://dl.k8s.io
export KUBERNETES_DOWNLOAD_TESTS=true

./cluster/get-kube-binaries.sh

以上几个变量说明：

https://dl.k8s.io/v1.22.6/kubernetes-server-linux-amd64.tar.gz
https://dl.k8s.io/v1.22.6/kubernetes-client-linux-amd64.tar.gz

# 以下为设置了 KUBERNETES_DOWNLOAD_TESTS 变量才做下载
https://dl.k8s.io/v1.22.6/kubernetes-test-portable.tar.gz
https://dl.k8s.io/v1.22.6/kubernetes-test-linux-amd64.tar.gz

server/kubernetes-server-linux-amd64.tar.gz
client/kubernetes-client-linux-amd64.tar.gz

# 以下为设置了 KUBERNETES_DOWNLOAD_TESTS 变量才做下载
test/kubernetes-test-linux-amd64.tar.gz
test/kubernetes-test-portable.tar.gz

解压各压缩包

cd ..

tar xzvf kubernetes/server/kubernetes-server-linux-amd64.tar.gz
tar xzvf kubernetes/client/kubernetes-client-linux-amd64.tar.gz

# 以下为设置了 KUBERNETES_DOWNLOAD_TESTS 变量才做下载
tar xzvf kubernetes/test/kubernetes-test-linux-amd64.tar.gz
tar xzvf kubernetes/test/kubernetes-test-portable.tar.gz

导入容器镜像

export KUBE_VERSION=v1.22.6
export INTERNAL_IMAGES_NAMESPACE=registry.cn-hangzhou.aliyuncs.com/kube-image-repo

cd kubernetes/server/bin

for img in kube-apiserver kube-controller-manager kube-proxy kube-scheduler
do
 #docker import ${img}.tar ${INTERNAL_IMAGES_NAMESPACE}/${img}:${KUBE_VERSION}
 docker load -i ${img}.tar

 img_id=$(docker images | grep k8s.gcr.io/${img} | grep ${KUBE_VERSION} | awk '{ print $3 }')
 docker tag ${img_id} ${INTERNAL_IMAGES_NAMESPACE}/${img}:${KUBE_VERSION}

 docker push ${INTERNAL_IMAGES_NAMESPACE}/${img}:${KUBE_VERSION}
done

注意：这里必须使用docker load，而不是docker import

pause镜像

https://github.com/kubernetes/kubernetes/tree/master/build/pause

etcd镜像

# 下载代码
git clone https://github.com/etcd-io/etcd.git

# 切换指定版本
git checkout $tag

# 编译linux amd64版本
ARCH=amd64 GOOS=linux make build

# 打包容器镜像
cd bin
cp ../Dockerfile-release.amd64 Dockerfile
docker build -f Dockerfile ./ -t registry.cn-hangzhou.aliyuncs.com/kube-image-repo/etcd:3.5.1

coredns镜像

https://github.com/coredns/coredns

cni-plugins

https://github.com/containernetworking/plugins/releases

https://github.com/containernetworking/plugins/releases/download/v1.0.1/cni-plugins-linux-amd64-v1.0.1.tgz

cri-tools

VERSION="v1.23.0"
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz

VERSION="v1.23.0"
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/critest-$VERSION-linux-amd64.tar.gz
sudo tar zxvf critest-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f critest-$VERSION-linux-amd64.tar.gz

containerd

https://github.com/containerd/containerd/releases

calico

https://github.com/projectcalico/calico/releases

flanneld

https://github.com/flannel-io/flannel/releases

Docs: Join

Fri, 02 Dec 2022 18:16:38 +0800

简要概述

apiVersion: kubeadm.k8s.io/v1beta3
kind: JoinConfiguration

代码路径：

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1beta3/types.go

配置示例

TODO;

数据结构

JoinConfiguration

// JoinConfiguration contains elements describing a particular node.
type JoinConfiguration struct {
 metav1.TypeMeta `json:",inline"`

 // NodeRegistration holds fields that relate to registering the new control-plane node to the cluster
 // +optional
 NodeRegistration NodeRegistrationOptions `json:"nodeRegistration,omitempty"`

 // CACertPath is the path to the SSL certificate authority used to
 // secure comunications between node and control-plane.
 // Defaults to "/etc/kubernetes/pki/ca.crt".
 // +optional
 CACertPath string `json:"caCertPath,omitempty"`

 // Discovery specifies the options for the kubelet to use during the TLS Bootstrap process
 Discovery Discovery `json:"discovery"`

 // ControlPlane defines the additional control plane instance to be deployed on the joining node.
 // If nil, no additional control plane instance will be deployed.
 // +optional
 ControlPlane *JoinControlPlane `json:"controlPlane,omitempty"`

 // SkipPhases is a list of phases to skip during command execution.
 // The list of phases can be obtained with the "kubeadm join --help" command.
 // The flag "--skip-phases" takes precedence over this field.
 // +optional
 SkipPhases []string `json:"skipPhases,omitempty"`

 // Patches contains options related to applying patches to components deployed by kubeadm during
 // "kubeadm join".
 // +optional
 Patches *Patches `json:"patches,omitempty"`
}

JoinControlPlane

// JoinControlPlane contains elements describing an additional control plane instance to be deployed on the joining node.
type JoinControlPlane struct {
 // LocalAPIEndpoint represents the endpoint of the API server instance to be deployed on this node.
 // +optional
 LocalAPIEndpoint APIEndpoint `json:"localAPIEndpoint,omitempty"`

 // CertificateKey is the key that is used for decryption of certificates after they are downloaded from the secret
 // upon joining a new control plane node. The corresponding encryption key is in the InitConfiguration.
 // +optional
 CertificateKey string `json:"certificateKey,omitempty"`
}

Docs: Kubelet

Fri, 02 Dec 2022 15:20:10 +0800

简要概述

apiVersion: kubelet.config.k8s.io/v1beta
kind: KubeletConfiguration

代码路径：

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1beta3/types.go

配置示例

TODO;

数据结构

KubeletConfiguration

// KubeletConfiguration contains the configuration for the Kubelet
type KubeletConfiguration struct {
 metav1.TypeMeta `json:",inline"`

 // enableServer enables Kubelet's secured server.
 // Note: Kubelet's insecure port is controlled by the readOnlyPort option.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Default: true
 EnableServer *bool `json:"enableServer,omitempty"`
 // staticPodPath is the path to the directory containing local (static) pods to
 // run, or the path to a single static pod file.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // the set of static pods specified at the new path may be different than the
 // ones the Kubelet initially started with, and this may disrupt your node.
 // Default: ""
 // +optional
 StaticPodPath string `json:"staticPodPath,omitempty"`
 // syncFrequency is the max period between synchronizing running
 // containers and config.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // shortening this duration may have a negative performance impact, especially
 // as the number of Pods on the node increases. Alternatively, increasing this
 // duration will result in longer refresh times for ConfigMaps and Secrets.
 // Default: "1m"
 // +optional
 SyncFrequency metav1.Duration `json:"syncFrequency,omitempty"`
 // fileCheckFrequency is the duration between checking config files for
 // new data.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // shortening the duration will cause the Kubelet to reload local Static Pod
 // configurations more frequently, which may have a negative performance impact.
 // Default: "20s"
 // +optional
 FileCheckFrequency metav1.Duration `json:"fileCheckFrequency,omitempty"`
 // httpCheckFrequency is the duration between checking http for new data.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // shortening the duration will cause the Kubelet to poll staticPodURL more
 // frequently, which may have a negative performance impact.
 // Default: "20s"
 // +optional
 HTTPCheckFrequency metav1.Duration `json:"httpCheckFrequency,omitempty"`
 // staticPodURL is the URL for accessing static pods to run.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // the set of static pods specified at the new URL may be different than the
 // ones the Kubelet initially started with, and this may disrupt your node.
 // Default: ""
 // +optional
 StaticPodURL string `json:"staticPodURL,omitempty"`
 // staticPodURLHeader is a map of slices with HTTP headers to use when accessing the podURL.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt the ability to read the latest set of static pods from StaticPodURL.
 // Default: nil
 // +optional
 StaticPodURLHeader map[string][]string `json:"staticPodURLHeader,omitempty"`
 // address is the IP address for the Kubelet to serve on (set to 0.0.0.0
 // for all interfaces).
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Default: "0.0.0.0"
 // +optional
 Address string `json:"address,omitempty"`
 // port is the port for the Kubelet to serve on.
 // The port number must be between 1 and 65535, inclusive.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Default: 10250
 // +optional
 Port int32 `json:"port,omitempty"`
 // readOnlyPort is the read-only port for the Kubelet to serve on with
 // no authentication/authorization.
 // The port number must be between 1 and 65535, inclusive.
 // Setting this field to 0 disables the read-only service.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Default: 0 (disabled)
 // +optional
 ReadOnlyPort int32 `json:"readOnlyPort,omitempty"`
 // tlsCertFile is the file containing x509 Certificate for HTTPS. (CA cert,
 // if any, concatenated after server cert). If tlsCertFile and
 // tlsPrivateKeyFile are not provided, a self-signed certificate
 // and key are generated for the public address and saved to the directory
 // passed to the Kubelet's --cert-dir flag.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Default: ""
 // +optional
 TLSCertFile string `json:"tlsCertFile,omitempty"`
 // tlsPrivateKeyFile is the file containing x509 private key matching tlsCertFile.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Default: ""
 // +optional
 TLSPrivateKeyFile string `json:"tlsPrivateKeyFile,omitempty"`
 // tlsCipherSuites is the list of allowed cipher suites for the server.
 // Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Default: nil
 // +optional
 TLSCipherSuites []string `json:"tlsCipherSuites,omitempty"`
 // tlsMinVersion is the minimum TLS version supported.
 // Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Default: ""
 // +optional
 TLSMinVersion string `json:"tlsMinVersion,omitempty"`
 // rotateCertificates enables client certificate rotation. The Kubelet will request a
 // new certificate from the certificates.k8s.io API. This requires an approver to approve the
 // certificate signing requests.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // disabling it may disrupt the Kubelet's ability to authenticate with the API server
 // after the current certificate expires.
 // Default: false
 // +optional
 RotateCertificates bool `json:"rotateCertificates,omitempty"`
 // serverTLSBootstrap enables server certificate bootstrap. Instead of self
 // signing a serving certificate, the Kubelet will request a certificate from
 // the 'certificates.k8s.io' API. This requires an approver to approve the
 // certificate signing requests (CSR). The RotateKubeletServerCertificate feature
 // must be enabled when setting this field.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // disabling it will stop the renewal of Kubelet server certificates, which can
 // disrupt components that interact with the Kubelet server in the long term,
 // due to certificate expiration.
 // Default: false
 // +optional
 ServerTLSBootstrap bool `json:"serverTLSBootstrap,omitempty"`
 // authentication specifies how requests to the Kubelet's server are authenticated.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Defaults:
 // anonymous:
 // enabled: false
 // webhook:
 // enabled: true
 // cacheTTL: "2m"
 // +optional
 Authentication KubeletAuthentication `json:"authentication"`
 // authorization specifies how requests to the Kubelet's server are authorized.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Defaults:
 // mode: Webhook
 // webhook:
 // cacheAuthorizedTTL: "5m"
 // cacheUnauthorizedTTL: "30s"
 // +optional
 Authorization KubeletAuthorization `json:"authorization"`
 // registryPullQPS is the limit of registry pulls per second.
 // The value must not be a negative number.
 // Setting it to 0 means no limit.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact scalability by changing the amount of traffic produced
 // by image pulls.
 // Default: 5
 // +optional
 RegistryPullQPS *int32 `json:"registryPullQPS,omitempty"`
 // registryBurst is the maximum size of bursty pulls, temporarily allows
 // pulls to burst to this number, while still not exceeding registryPullQPS.
 // The value must not be a negative number.
 // Only used if registryPullQPS is greater than 0.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact scalability by changing the amount of traffic produced
 // by image pulls.
 // Default: 10
 // +optional
 RegistryBurst int32 `json:"registryBurst,omitempty"`
 // eventRecordQPS is the maximum event creations per second. If 0, there
 // is no limit enforced. The value cannot be a negative number.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact scalability by changing the amount of traffic produced by
 // event creations.
 // Default: 5
 // +optional
 EventRecordQPS *int32 `json:"eventRecordQPS,omitempty"`
 // eventBurst is the maximum size of a burst of event creations, temporarily
 // allows event creations to burst to this number, while still not exceeding
 // eventRecordQPS. This field canot be a negative number and it is only used
 // when eventRecordQPS > 0.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact scalability by changing the amount of traffic produced by
 // event creations.
 // Default: 10
 // +optional
 EventBurst int32 `json:"eventBurst,omitempty"`
 // enableDebuggingHandlers enables server endpoints for log access
 // and local running of containers and commands, including the exec,
 // attach, logs, and portforward features.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // disabling it may disrupt components that interact with the Kubelet server.
 // Default: true
 // +optional
 EnableDebuggingHandlers *bool `json:"enableDebuggingHandlers,omitempty"`
 // enableContentionProfiling enables lock contention profiling, if enableDebuggingHandlers is true.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // enabling it may carry a performance impact.
 // Default: false
 // +optional
 EnableContentionProfiling bool `json:"enableContentionProfiling,omitempty"`
 // healthzPort is the port of the localhost healthz endpoint (set to 0 to disable).
 // A valid number is between 1 and 65535.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that monitor Kubelet health.
 // Default: 10248
 // +optional
 HealthzPort *int32 `json:"healthzPort,omitempty"`
 // healthzBindAddress is the IP address for the healthz server to serve on.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that monitor Kubelet health.
 // Default: "127.0.0.1"
 // +optional
 HealthzBindAddress string `json:"healthzBindAddress,omitempty"`
 // oomScoreAdj is The oom-score-adj value for kubelet process. Values
 // must be within the range [-1000, 1000].
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact the stability of nodes under memory pressure.
 // Default: -999
 // +optional
 OOMScoreAdj *int32 `json:"oomScoreAdj,omitempty"`
 // clusterDomain is the DNS domain for this cluster. If set, kubelet will
 // configure all containers to search this domain in addition to the
 // host's search domains.
 // Dynamic Kubelet Config (deprecated): Dynamically updating this field is not recommended,
 // as it should be kept in sync with the rest of the cluster.
 // Default: ""
 // +optional
 ClusterDomain string `json:"clusterDomain,omitempty"`
 // clusterDNS is a list of IP addresses for the cluster DNS server. If set,
 // kubelet will configure all containers to use this for DNS resolution
 // instead of the host's DNS servers.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // changes will only take effect on Pods created after the update. Draining
 // the node is recommended before changing this field.
 // Default: nil
 // +optional
 ClusterDNS []string `json:"clusterDNS,omitempty"`
 // streamingConnectionIdleTimeout is the maximum time a streaming connection
 // can be idle before the connection is automatically closed.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact components that rely on infrequent updates over streaming
 // connections to the Kubelet server.
 // Default: "4h"
 // +optional
 StreamingConnectionIdleTimeout metav1.Duration `json:"streamingConnectionIdleTimeout,omitempty"`
 // nodeStatusUpdateFrequency is the frequency that kubelet computes node
 // status. If node lease feature is not enabled, it is also the frequency that
 // kubelet posts node status to master.
 // Note: When node lease feature is not enabled, be cautious when changing the
 // constant, it must work with nodeMonitorGracePeriod in nodecontroller.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact node scalability, and also that the node controller's
 // nodeMonitorGracePeriod must be set to N*NodeStatusUpdateFrequency,
 // where N is the number of retries before the node controller marks
 // the node unhealthy.
 // Default: "10s"
 // +optional
 NodeStatusUpdateFrequency metav1.Duration `json:"nodeStatusUpdateFrequency,omitempty"`
 // nodeStatusReportFrequency is the frequency that kubelet posts node
 // status to master if node status does not change. Kubelet will ignore this
 // frequency and post node status immediately if any change is detected. It is
 // only used when node lease feature is enabled. nodeStatusReportFrequency's
 // default value is 5m. But if nodeStatusUpdateFrequency is set explicitly,
 // nodeStatusReportFrequency's default value will be set to
 // nodeStatusUpdateFrequency for backward compatibility.
 // Default: "5m"
 // +optional
 NodeStatusReportFrequency metav1.Duration `json:"nodeStatusReportFrequency,omitempty"`
 // nodeLeaseDurationSeconds is the duration the Kubelet will set on its corresponding Lease,
 // when the NodeLease feature is enabled. This feature provides an indicator of node
 // health by having the Kubelet create and periodically renew a lease, named after the node,
 // in the kube-node-lease namespace. If the lease expires, the node can be considered unhealthy.
 // The lease is currently renewed every 10s, per KEP-0009. In the future, the lease renewal interval
 // may be set based on the lease duration.
 // The field value must be greater than 0.
 // Requires the NodeLease feature gate to be enabled.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // decreasing the duration may reduce tolerance for issues that temporarily prevent
 // the Kubelet from renewing the lease (e.g. a short-lived network issue).
 // Default: 40
 // +optional
 NodeLeaseDurationSeconds int32 `json:"nodeLeaseDurationSeconds,omitempty"`
 // imageMinimumGCAge is the minimum age for an unused image before it is
 // garbage collected.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may trigger or delay garbage collection, and may change the image overhead
 // on the node.
 // Default: "2m"
 // +optional
 ImageMinimumGCAge metav1.Duration `json:"imageMinimumGCAge,omitempty"`
 // imageGCHighThresholdPercent is the percent of disk usage after which
 // image garbage collection is always run. The percent is calculated by
 // dividing this field value by 100, so this field must be between 0 and
 // 100, inclusive. When specified, the value must be greater than
 // imageGCLowThresholdPercent.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may trigger or delay garbage collection, and may change the image overhead
 // on the node.
 // Default: 85
 // +optional
 ImageGCHighThresholdPercent *int32 `json:"imageGCHighThresholdPercent,omitempty"`
 // imageGCLowThresholdPercent is the percent of disk usage before which
 // image garbage collection is never run. Lowest disk usage to garbage
 // collect to. The percent is calculated by dividing this field value by 100,
 // so the field value must be between 0 and 100, inclusive. When specified, the
 // value must be less than imageGCHighThresholdPercent.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may trigger or delay garbage collection, and may change the image overhead
 // on the node.
 // Default: 80
 // +optional
 ImageGCLowThresholdPercent *int32 `json:"imageGCLowThresholdPercent,omitempty"`
 // volumeStatsAggPeriod is the frequency for calculating and caching volume
 // disk usage for all pods.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // shortening the period may carry a performance impact.
 // Default: "1m"
 // +optional
 VolumeStatsAggPeriod metav1.Duration `json:"volumeStatsAggPeriod,omitempty"`
 // kubeletCgroups is the absolute name of cgroups to isolate the kubelet in
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: ""
 // +optional
 KubeletCgroups string `json:"kubeletCgroups,omitempty"`
 // systemCgroups is absolute name of cgroups in which to place
 // all non-kernel processes that are not already in a container. Empty
 // for no container. Rolling back the flag requires a reboot.
 // The cgroupRoot must be specified if this field is not empty.
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: ""
 // +optional
 SystemCgroups string `json:"systemCgroups,omitempty"`
 // cgroupRoot is the root cgroup to use for pods. This is handled by the
 // container runtime on a best effort basis.
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: ""
 // +optional
 CgroupRoot string `json:"cgroupRoot,omitempty"`
 // cgroupsPerQOS enable QoS based CGroup hierarchy: top level CGroups for QoS classes
 // and all Burstable and BestEffort Pods are brought up under their specific top level
 // QoS CGroup.
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: true
 // +optional
 CgroupsPerQOS *bool `json:"cgroupsPerQOS,omitempty"`
 // cgroupDriver is the driver kubelet uses to manipulate CGroups on the host (cgroupfs
 // or systemd).
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: "cgroupfs"
 // +optional
 CgroupDriver string `json:"cgroupDriver,omitempty"`
 // cpuManagerPolicy is the name of the policy to use.
 // Requires the CPUManager feature gate to be enabled.
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: "None"
 // +optional
 CPUManagerPolicy string `json:"cpuManagerPolicy,omitempty"`
 // cpuManagerPolicyOptions is a set of key=value which allows to set extra options
 // to fine tune the behaviour of the cpu manager policies.
 // Requires both the "CPUManager" and "CPUManagerPolicyOptions" feature gates to be enabled.
 // Dynamic Kubelet Config (beta): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: nil
 // +optional
 CPUManagerPolicyOptions map[string]string `json:"cpuManagerPolicyOptions,omitempty"`
 // cpuManagerReconcilePeriod is the reconciliation period for the CPU Manager.
 // Requires the CPUManager feature gate to be enabled.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // shortening the period may carry a performance impact.
 // Default: "10s"
 // +optional
 CPUManagerReconcilePeriod metav1.Duration `json:"cpuManagerReconcilePeriod,omitempty"`
 // memoryManagerPolicy is the name of the policy to use by memory manager.
 // Requires the MemoryManager feature gate to be enabled.
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: "none"
 // +optional
 MemoryManagerPolicy string `json:"memoryManagerPolicy,omitempty"`
 // topologyManagerPolicy is the name of the topology manager policy to use.
 // Valid values include:
 //
 // - `restricted`: kubelet only allows pods with optimal NUMA node alignment for
 // requested resources;
 // - `best-effort`: kubelet will favor pods with NUMA alignment of CPU and device
 // resources;
 // - `none`: kublet has no knowledge of NUMA alignment of a pod's CPU and device resources.
 // - `single-numa-node`: kubelet only allows pods with a single NUMA alignment
 // of CPU and device resources.
 //
 // Policies other than "none" require the TopologyManager feature gate to be enabled.
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: "none"
 // +optional
 TopologyManagerPolicy string `json:"topologyManagerPolicy,omitempty"`
 // topologyManagerScope represents the scope of topology hint generation
 // that topology manager requests and hint providers generate. Valid values include:
 //
 // - `container`: topology policy is applied on a per-container basis.
 // - `pod`: topology policy is applied on a per-pod basis.
 //
 // "pod" scope requires the TopologyManager feature gate to be enabled.
 // Default: "container"
 // +optional
 TopologyManagerScope string `json:"topologyManagerScope,omitempty"`
 // qosReserved is a set of resource name to percentage pairs that specify
 // the minimum percentage of a resource reserved for exclusive use by the
 // guaranteed QoS tier.
 // Currently supported resources: "memory"
 // Requires the QOSReserved feature gate to be enabled.
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: nil
 // +optional
 QOSReserved map[string]string `json:"qosReserved,omitempty"`
 // runtimeRequestTimeout is the timeout for all runtime requests except long running
 // requests - pull, logs, exec and attach.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may disrupt components that interact with the Kubelet server.
 // Default: "2m"
 // +optional
 RuntimeRequestTimeout metav1.Duration `json:"runtimeRequestTimeout,omitempty"`
 // hairpinMode specifies how the Kubelet should configure the container
 // bridge for hairpin packets.
 // Setting this flag allows endpoints in a Service to loadbalance back to
 // themselves if they should try to access their own Service. Values:
 //
 // - "promiscuous-bridge": make the container bridge promiscuous.
 // - "hairpin-veth": set the hairpin flag on container veth interfaces.
 // - "none": do nothing.
 //
 // Generally, one must set `--hairpin-mode=hairpin-veth to` achieve hairpin NAT,
 // because promiscuous-bridge assumes the existence of a container bridge named cbr0.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may require a node reboot, depending on the network plugin.
 // Default: "promiscuous-bridge"
 // +optional
 HairpinMode string `json:"hairpinMode,omitempty"`
 // maxPods is the maximum number of Pods that can run on this Kubelet.
 // The value must be a non-negative integer.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // changes may cause Pods to fail admission on Kubelet restart, and may change
 // the value reported in Node.Status.Capacity[v1.ResourcePods], thus affecting
 // future scheduling decisions. Increasing this value may also decrease performance,
 // as more Pods can be packed into a single node.
 // Default: 110
 // +optional
 MaxPods int32 `json:"maxPods,omitempty"`
 // podCIDR is the CIDR to use for pod IP addresses, only used in standalone mode.
 // In cluster mode, this is obtained from the control plane.
 // Dynamic Kubelet Config (deprecated): This field should always be set to the empty default.
 // It should only set for standalone Kubelets, which cannot use Dynamic Kubelet Config.
 // Default: ""
 // +optional
 PodCIDR string `json:"podCIDR,omitempty"`
 // podPidsLimit is the maximum number of PIDs in any pod.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // lowering it may prevent container processes from forking after the change.
 // Default: -1
 // +optional
 PodPidsLimit *int64 `json:"podPidsLimit,omitempty"`
 // resolvConf is the resolver configuration file used as the basis
 // for the container DNS resolution configuration.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // changes will only take effect on Pods created after the update. Draining
 // the node is recommended before changing this field.
 // Default: "/etc/resolv.conf"
 // +optional
 ResolverConfig string `json:"resolvConf,omitempty"`
 // runOnce causes the Kubelet to check the API server once for pods,
 // run those in addition to the pods specified by static pod files, and exit.
 // Default: false
 // +optional
 RunOnce bool `json:"runOnce,omitempty"`
 // cpuCFSQuota enables CPU CFS quota enforcement for containers that
 // specify CPU limits.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // disabling it may reduce node stability.
 // Default: true
 // +optional
 CPUCFSQuota *bool `json:"cpuCFSQuota,omitempty"`
 // cpuCFSQuotaPeriod is the CPU CFS quota period value, `cpu.cfs_period_us`.
 // The value must be between 1 us and 1 second, inclusive.
 // Requires the CustomCPUCFSQuotaPeriod feature gate to be enabled.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // limits set for containers will result in different cpu.cfs_quota settings. This
 // will trigger container restarts on the node being reconfigured.
 // Default: "100ms"
 // +optional
 CPUCFSQuotaPeriod *metav1.Duration `json:"cpuCFSQuotaPeriod,omitempty"`
 // nodeStatusMaxImages caps the number of images reported in Node.status.images.
 // The value must be greater than -2.
 // Note: If -1 is specified, no cap will be applied. If 0 is specified, no image is returned.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // different values can be reported on node status.
 // Default: 50
 // +optional
 NodeStatusMaxImages *int32 `json:"nodeStatusMaxImages,omitempty"`
 // maxOpenFiles is Number of files that can be opened by Kubelet process.
 // The value must be a non-negative number.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact the ability of the Kubelet to interact with the node's filesystem.
 // Default: 1000000
 // +optional
 MaxOpenFiles int64 `json:"maxOpenFiles,omitempty"`
 // contentType is contentType of requests sent to apiserver.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact the ability for the Kubelet to communicate with the API server.
 // If the Kubelet loses contact with the API server due to a change to this field,
 // the change cannot be reverted via dynamic Kubelet config.
 // Default: "application/vnd.kubernetes.protobuf"
 // +optional
 ContentType string `json:"contentType,omitempty"`
 // kubeAPIQPS is the QPS to use while talking with kubernetes apiserver.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact scalability by changing the amount of traffic the Kubelet
 // sends to the API server.
 // Default: 5
 // +optional
 KubeAPIQPS *int32 `json:"kubeAPIQPS,omitempty"`
 // kubeAPIBurst is the burst to allow while talking with kubernetes API server.
 // This field cannot be a negative number.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact scalability by changing the amount of traffic the Kubelet
 // sends to the API server.
 // Default: 10
 // +optional
 KubeAPIBurst int32 `json:"kubeAPIBurst,omitempty"`
 // serializeImagePulls when enabled, tells the Kubelet to pull images one
 // at a time. We recommend *not* changing the default value on nodes that
 // run docker daemon with version < 1.9 or an Aufs storage backend.
 // Issue #10959 has more details.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact the performance of image pulls.
 // Default: true
 // +optional
 SerializeImagePulls *bool `json:"serializeImagePulls,omitempty"`
 // evictionHard is a map of signal names to quantities that defines hard eviction
 // thresholds. For example: `{"memory.available": "300Mi"}`.
 // To explicitly disable, pass a 0% or 100% threshold on an arbitrary resource.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may trigger or delay Pod evictions.
 // Default:
 // memory.available: "100Mi"
 // nodefs.available: "10%"
 // nodefs.inodesFree: "5%"
 // imagefs.available: "15%"
 // +optional
 EvictionHard map[string]string `json:"evictionHard,omitempty"`
 // evictionSoft is a map of signal names to quantities that defines soft eviction thresholds.
 // For example: `{"memory.available": "300Mi"}`.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may trigger or delay Pod evictions, and may change the allocatable reported
 // by the node.
 // Default: nil
 // +optional
 EvictionSoft map[string]string `json:"evictionSoft,omitempty"`
 // evictionSoftGracePeriod is a map of signal names to quantities that defines grace
 // periods for each soft eviction signal. For example: `{"memory.available": "30s"}`.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may trigger or delay Pod evictions.
 // Default: nil
 // +optional
 EvictionSoftGracePeriod map[string]string `json:"evictionSoftGracePeriod,omitempty"`
 // evictionPressureTransitionPeriod is the duration for which the kubelet has to wait
 // before transitioning out of an eviction pressure condition.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // lowering it may decrease the stability of the node when the node is overcommitted.
 // Default: "5m"
 // +optional
 EvictionPressureTransitionPeriod metav1.Duration `json:"evictionPressureTransitionPeriod,omitempty"`
 // evictionMaxPodGracePeriod is the maximum allowed grace period (in seconds) to use
 // when terminating pods in response to a soft eviction threshold being met. This value
 // effectively caps the Pod's terminationGracePeriodSeconds value during soft evictions.
 // Note: Due to issue #64530, the behavior has a bug where this value currently just
 // overrides the grace period during soft eviction, which can increase the grace
 // period from what is set on the Pod. This bug will be fixed in a future release.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // lowering it decreases the amount of time Pods will have to gracefully clean
 // up before being killed during a soft eviction.
 // Default: 0
 // +optional
 EvictionMaxPodGracePeriod int32 `json:"evictionMaxPodGracePeriod,omitempty"`
 // evictionMinimumReclaim is a map of signal names to quantities that defines minimum reclaims,
 // which describe the minimum amount of a given resource the kubelet will reclaim when
 // performing a pod eviction while that resource is under pressure.
 // For example: `{"imagefs.available": "2Gi"}`.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may change how well eviction can manage resource pressure.
 // Default: nil
 // +optional
 EvictionMinimumReclaim map[string]string `json:"evictionMinimumReclaim,omitempty"`
 // podsPerCore is the maximum number of pods per core. Cannot exceed maxPods.
 // The value must be a non-negative integer.
 // If 0, there is no limit on the number of Pods.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // changes may cause Pods to fail admission on Kubelet restart, and may change
 // the value reported in `Node.status.capacity.pods`, thus affecting
 // future scheduling decisions. Increasing this value may also decrease performance,
 // as more Pods can be packed into a single node.
 // Default: 0
 // +optional
 PodsPerCore int32 `json:"podsPerCore,omitempty"`
 // enableControllerAttachDetach enables the Attach/Detach controller to
 // manage attachment/detachment of volumes scheduled to this node, and
 // disables kubelet from executing any attach/detach operations.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // changing which component is responsible for volume management on a live node
 // may result in volumes refusing to detach if the node is not drained prior to
 // the update, and if Pods are scheduled to the node before the
 // volumes.kubernetes.io/controller-managed-attach-detach annotation is updated by the
 // Kubelet. In general, it is safest to leave this value set the same as local config.
 // Default: true
 // +optional
 EnableControllerAttachDetach *bool `json:"enableControllerAttachDetach,omitempty"`
 // protectKernelDefaults, if true, causes the Kubelet to error if kernel
 // flags are not as it expects. Otherwise the Kubelet will attempt to modify
 // kernel flags to match its expectation.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // enabling it may cause the Kubelet to crash-loop if the Kernel is not configured as
 // Kubelet expects.
 // Default: false
 // +optional
 ProtectKernelDefaults bool `json:"protectKernelDefaults,omitempty"`
 // makeIPTablesUtilChains, if true, causes the Kubelet ensures a set of iptables rules
 // are present on host.
 // These rules will serve as utility rules for various components, e.g. kube-proxy.
 // The rules will be created based on iptablesMasqueradeBit and iptablesDropBit.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // disabling it will prevent the Kubelet from healing locally misconfigured iptables rules.
 // Default: true
 // +optional
 MakeIPTablesUtilChains *bool `json:"makeIPTablesUtilChains,omitempty"`
 // iptablesMasqueradeBit is the bit of the iptables fwmark space to mark for SNAT.
 // Values must be within the range [0, 31]. Must be different from other mark bits.
 // Warning: Please match the value of the corresponding parameter in kube-proxy.
 // TODO: clean up IPTablesMasqueradeBit in kube-proxy.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it needs to be coordinated with other components, like kube-proxy, and the update
 // will only be effective if MakeIPTablesUtilChains is enabled.
 // Default: 14
 // +optional
 IPTablesMasqueradeBit *int32 `json:"iptablesMasqueradeBit,omitempty"`
 // iptablesDropBit is the bit of the iptables fwmark space to mark for dropping packets.
 // Values must be within the range [0, 31]. Must be different from other mark bits.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it needs to be coordinated with other components, like kube-proxy, and the update
 // will only be effective if MakeIPTablesUtilChains is enabled.
 // Default: 15
 // +optional
 IPTablesDropBit *int32 `json:"iptablesDropBit,omitempty"`
 // featureGates is a map of feature names to bools that enable or disable experimental
 // features. This field modifies piecemeal the built-in default values from
 // "k8s.io/kubernetes/pkg/features/kube_features.go".
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider the
 // documentation for the features you are enabling or disabling. While we
 // encourage feature developers to make it possible to dynamically enable
 // and disable features, some changes may require node reboots, and some
 // features may require careful coordination to retroactively disable.
 // Default: nil
 // +optional
 FeatureGates map[string]bool `json:"featureGates,omitempty"`
 // failSwapOn tells the Kubelet to fail to start if swap is enabled on the node.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // setting it to true will cause the Kubelet to crash-loop if swap is enabled.
 // Default: true
 // +optional
 FailSwapOn *bool `json:"failSwapOn,omitempty"`
 // memorySwap configures swap memory available to container workloads.
 // +featureGate=NodeSwap
 // +optional
 MemorySwap MemorySwapConfiguration `json:"memorySwap,omitempty"`
 // containerLogMaxSize is a quantity defining the maximum size of the container log
 // file before it is rotated. For example: "5Mi" or "256Ki".
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may trigger log rotation.
 // Default: "10Mi"
 // +optional
 ContainerLogMaxSize string `json:"containerLogMaxSize,omitempty"`
 // containerLogMaxFiles specifies the maximum number of container log files that can
 // be present for a container.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // lowering it may cause log files to be deleted.
 // Default: 5
 // +optional
 ContainerLogMaxFiles *int32 `json:"containerLogMaxFiles,omitempty"`
 // configMapAndSecretChangeDetectionStrategy is a mode in which ConfigMap and Secret
 // managers are running. Valid values include:
 //
 // - `Get`: kubelet fetches necessary objects directly from the API server;
 // - `Cache`: kubelet uses TTL cache for object fetched from the API server;
 // - `Watch`: kubelet uses watches to observe changes to objects that are in its interest.
 //
 // Default: "Watch"
 // +optional
 ConfigMapAndSecretChangeDetectionStrategy ResourceChangeDetectionStrategy `json:"configMapAndSecretChangeDetectionStrategy,omitempty"`

 /* the following fields are meant for Node Allocatable */

 // systemReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G)
 // pairs that describe resources reserved for non-kubernetes components.
 // Currently only cpu and memory are supported.
 // See http://kubernetes.io/docs/user-guide/compute-resources for more detail.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may not be possible to increase the reserved resources, because this
 // requires resizing cgroups. Always look for a NodeAllocatableEnforced event
 // after updating this field to ensure that the update was successful.
 // Default: nil
 // +optional
 SystemReserved map[string]string `json:"systemReserved,omitempty"`
 // kubeReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs
 // that describe resources reserved for kubernetes system components.
 // Currently cpu, memory and local storage for root file system are supported.
 // See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
 // for more details.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may not be possible to increase the reserved resources, because this
 // requires resizing cgroups. Always look for a NodeAllocatableEnforced event
 // after updating this field to ensure that the update was successful.
 // Default: nil
 // +optional
 KubeReserved map[string]string `json:"kubeReserved,omitempty"`
 // The reservedSystemCPUs option specifies the CPU list reserved for the host
 // level system threads and kubernetes related threads. This provide a "static"
 // CPU list rather than the "dynamic" list by systemReserved and kubeReserved.
 // This option does not support systemReservedCgroup or kubeReservedCgroup.
 ReservedSystemCPUs string `json:"reservedSystemCPUs,omitempty"`
 // showHiddenMetricsForVersion is the previous version for which you want to show
 // hidden metrics.
 // Only the previous minor version is meaningful, other values will not be allowed.
 // The format is `<major>.<minor>`, e.g.: `1.16`.
 // The purpose of this format is make sure you have the opportunity to notice
 // if the next release hides additional metrics, rather than being surprised
 // when they are permanently removed in the release after that.
 // Default: ""
 // +optional
 ShowHiddenMetricsForVersion string `json:"showHiddenMetricsForVersion,omitempty"`
 // systemReservedCgroup helps the kubelet identify absolute name of top level CGroup used
 // to enforce `systemReserved` compute resource reservation for OS system daemons.
 // Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md)
 // doc for more information.
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: ""
 // +optional
 SystemReservedCgroup string `json:"systemReservedCgroup,omitempty"`
 // kubeReservedCgroup helps the kubelet identify absolute name of top level CGroup used
 // to enforce `KubeReserved` compute resource reservation for Kubernetes node system daemons.
 // Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md)
 // doc for more information.
 // Dynamic Kubelet Config (deprecated): This field should not be updated without a full node
 // reboot. It is safest to keep this value the same as the local config.
 // Default: ""
 // +optional
 KubeReservedCgroup string `json:"kubeReservedCgroup,omitempty"`
 // This flag specifies the various Node Allocatable enforcements that Kubelet needs to perform.
 // This flag accepts a list of options. Acceptable options are `none`, `pods`,
 // `system-reserved` and `kube-reserved`.
 // If `none` is specified, no other options may be specified.
 // When `system-reserved` is in the list, systemReservedCgroup must be specified.
 // When `kube-reserved` is in the list, kubeReservedCgroup must be specified.
 // This field is supported only when `cgroupsPerQOS` is set to true.
 // Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md)
 // for more information.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // removing enforcements may reduce the stability of the node. Alternatively, adding
 // enforcements may reduce the stability of components which were using more than
 // the reserved amount of resources; for example, enforcing kube-reserved may cause
 // Kubelets to OOM if it uses more than the reserved resources, and enforcing system-reserved
 // may cause system daemons to OOM if they use more than the reserved resources.
 // Default: ["pods"]
 // +optional
 EnforceNodeAllocatable []string `json:"enforceNodeAllocatable,omitempty"`
 // A comma separated whitelist of unsafe sysctls or sysctl patterns (ending in `*`).
 // Unsafe sysctl groups are `kernel.shm*`, `kernel.msg*`, `kernel.sem`, `fs.mqueue.*`,
 // and `net.*`. For example: "`kernel.msg*,net.ipv4.route.min_pmtu`"
 // Default: []
 // +optional
 AllowedUnsafeSysctls []string `json:"allowedUnsafeSysctls,omitempty"`
 // volumePluginDir is the full path of the directory in which to search
 // for additional third party volume plugins.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that changing
 // the volumePluginDir may disrupt workloads relying on third party volume plugins.
 // Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
 // +optional
 VolumePluginDir string `json:"volumePluginDir,omitempty"`
 // providerID, if set, sets the unique ID of the instance that an external
 // provider (i.e. cloudprovider) can use to identify a specific node.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact the ability of the Kubelet to interact with cloud providers.
 // Default: ""
 // +optional
 ProviderID string `json:"providerID,omitempty"`
 // kernelMemcgNotification, if set, instructs the the kubelet to integrate with the
 // kernel memcg notification for determining if memory eviction thresholds are
 // exceeded rather than polling.
 // If DynamicKubeletConfig (deprecated; default off) is on, when
 // dynamically updating this field, consider that
 // it may impact the way Kubelet interacts with the kernel.
 // Default: false
 // +optional
 KernelMemcgNotification bool `json:"kernelMemcgNotification,omitempty"`
 // logging specifies the options of logging.
 // Refer to [Logs Options](https://github.com/kubernetes/component-base/blob/master/logs/options.go)
 // for more information.
 // Default:
 // Format: text
 // + optional
 Logging componentbaseconfigv1alpha1.LoggingConfiguration `json:"logging,omitempty"`
 // enableSystemLogHandler enables system logs via web interface host:port/logs/
 // Default: true
 // +optional
 EnableSystemLogHandler *bool `json:"enableSystemLogHandler,omitempty"`
 // shutdownGracePeriod specifies the total duration that the node should delay the
 // shutdown and total grace period for pod termination during a node shutdown.
 // Default: "0s"
 // +featureGate=GracefulNodeShutdown
 // +optional
 ShutdownGracePeriod metav1.Duration `json:"shutdownGracePeriod,omitempty"`
 // shutdownGracePeriodCriticalPods specifies the duration used to terminate critical
 // pods during a node shutdown. This should be less than shutdownGracePeriod.
 // For example, if shutdownGracePeriod=30s, and shutdownGracePeriodCriticalPods=10s,
 // during a node shutdown the first 20 seconds would be reserved for gracefully
 // terminating normal pods, and the last 10 seconds would be reserved for terminating
 // critical pods.
 // Default: "0s"
 // +featureGate=GracefulNodeShutdown
 // +optional
 ShutdownGracePeriodCriticalPods metav1.Duration `json:"shutdownGracePeriodCriticalPods,omitempty"`
 // reservedMemory specifies a comma-separated list of memory reservations for NUMA nodes.
 // The parameter makes sense only in the context of the memory manager feature.
 // The memory manager will not allocate reserved memory for container workloads.
 // For example, if you have a NUMA0 with 10Gi of memory and the reservedMemory was
 // specified to reserve 1Gi of memory at NUMA0, the memory manager will assume that
 // only 9Gi is available for allocation.
 // You can specify a different amount of NUMA node and memory types.
 // You can omit this parameter at all, but you should be aware that the amount of
 // reserved memory from all NUMA nodes should be equal to the amount of memory specified
 // by the [node allocatable](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable).
 // If at least one node allocatable parameter has a non-zero value, you will need
 // to specify at least one NUMA node.
 // Also, avoid specifying:
 //
 // 1. Duplicates, the same NUMA node, and memory type, but with a different value.
 // 2. zero limits for any memory type.
 // 3. NUMAs nodes IDs that do not exist under the machine.
 // 4. memory types except for memory and hugepages-<size>
 //
 // Default: nil
 // +optional
 ReservedMemory []MemoryReservation `json:"reservedMemory,omitempty"`
 // enableProfilingHandler enables profiling via web interface host:port/debug/pprof/
 // Default: true
 // +optional
 EnableProfilingHandler *bool `json:"enableProfilingHandler,omitempty"`
 // enableDebugFlagsHandler enables flags endpoint via web interface host:port/debug/flags/v
 // Default: true
 // +optional
 EnableDebugFlagsHandler *bool `json:"enableDebugFlagsHandler,omitempty"`
 // SeccompDefault enables the use of `RuntimeDefault` as the default seccomp profile for all workloads.
 // This requires the corresponding SeccompDefault feature gate to be enabled as well.
 // Default: false
 // +optional
 SeccompDefault *bool `json:"seccompDefault,omitempty"`
 // MemoryThrottlingFactor specifies the factor multiplied by the memory limit or node allocatable memory
 // when setting the cgroupv2 memory.high value to enforce MemoryQoS.
 // Decreasing this factor will set lower high limit for container cgroups and put heavier reclaim pressure
 // while increasing will put less reclaim pressure.
 // See http://kep.k8s.io/2570 for more details.
 // Default: 0.8
 // +featureGate=MemoryQoS
 // +optional
 MemoryThrottlingFactor *float64 `json:"memoryThrottlingFactor,omitempty"`
}

KubeletAuthentication

type KubeletAuthentication struct {
 // x509 contains settings related to x509 client certificate authentication.
 // +optional
 X509 KubeletX509Authentication `json:"x509"`
 // webhook contains settings related to webhook bearer token authentication.
 // +optional
 Webhook KubeletWebhookAuthentication `json:"webhook"`
 // anonymous contains settings related to anonymous authentication.
 // +optional
 Anonymous KubeletAnonymousAuthentication `json:"anonymous"`
}

KubeletAuthorization

type KubeletAuthorizationMode string

const (
 // KubeletAuthorizationModeAlwaysAllow authorizes all authenticated requests
 KubeletAuthorizationModeAlwaysAllow KubeletAuthorizationMode = "AlwaysAllow"
 // KubeletAuthorizationModeWebhook uses the SubjectAccessReview API to determine authorization
 KubeletAuthorizationModeWebhook KubeletAuthorizationMode = "Webhook"
)

type KubeletAuthorization struct {
 // mode is the authorization mode to apply to requests to the kubelet server.
 // Valid values are `AlwaysAllow` and `Webhook`.
 // Webhook mode uses the SubjectAccessReview API to determine authorization.
 // +optional
 Mode KubeletAuthorizationMode `json:"mode,omitempty"`

 // webhook contains settings related to Webhook authorization.
 // +optional
 Webhook KubeletWebhookAuthorization `json:"webhook"`
}

ResourceChangeDetectionStrategy

// ResourceChangeDetectionStrategy denotes a mode in which internal
// managers (secret, configmap) are discovering object changes.
type ResourceChangeDetectionStrategy string

// Enum settings for different strategies of kubelet managers.
const (
 // GetChangeDetectionStrategy is a mode in which kubelet fetches
 // necessary objects directly from apiserver.
 GetChangeDetectionStrategy ResourceChangeDetectionStrategy = "Get"
 // TTLCacheChangeDetectionStrategy is a mode in which kubelet uses
 // ttl cache for object directly fetched from apiserver.
 TTLCacheChangeDetectionStrategy ResourceChangeDetectionStrategy = "Cache"
 // WatchChangeDetectionStrategy is a mode in which kubelet uses
 // watches to observe changes to objects that are in its interest.
 WatchChangeDetectionStrategy ResourceChangeDetectionStrategy = "Watch"
 // RestrictedTopologyManagerPolicy is a mode in which kubelet only allows
 // pods with optimal NUMA node alignment for requested resources
 RestrictedTopologyManagerPolicy = "restricted"
 // BestEffortTopologyManagerPolicy is a mode in which kubelet will favour
 // pods with NUMA alignment of CPU and device resources.
 BestEffortTopologyManagerPolicy = "best-effort"
 // NoneTopologyManagerPolicy is a mode in which kubelet has no knowledge
 // of NUMA alignment of a pod's CPU and device resources.
 NoneTopologyManagerPolicy = "none"
 // SingleNumaNodeTopologyManagerPolicy is a mode in which kubelet only allows
 // pods with a single NUMA alignment of CPU and device resources.
 SingleNumaNodeTopologyManagerPolicy = "single-numa-node"
 // ContainerTopologyManagerScope represents that
 // topology policy is applied on a per-container basis.
 ContainerTopologyManagerScope = "container"
 // PodTopologyManagerScope represents that
 // topology policy is applied on a per-pod basis.
 PodTopologyManagerScope = "pod"
 // NoneMemoryManagerPolicy is a memory manager none policy, under the none policy
 // the memory manager will not pin containers memory of guaranteed pods
 NoneMemoryManagerPolicy = "None"
 // StaticMemoryManagerPolicy is a memory manager static policy, under the static policy
 // the memory manager will try to pin containers memory of guaranteed pods to the smallest
 // possible sub-set of NUMA nodes
 StaticMemoryManagerPolicy = "Static"
)

componentbaseconfigv1alpha1.LoggingConfiguration

代码路径：kubernetes-1.22.16/staging/src/k8s.io/component-base/config/types.go

// LoggingConfiguration contains logging options
// Refer [Logs Options](https://github.com/kubernetes/component-base/blob/master/logs/options.go) for more information.
type LoggingConfiguration struct {
 // Format Flag specifies the structure of log messages.
 // default value of format is `text`
 Format string
 // [Experimental] When enabled prevents logging of fields tagged as sensitive (passwords, keys, tokens).
 // Runtime log sanitization may introduce significant computation overhead and therefore should not be enabled in production.`)
 Sanitization bool
}

MemoryReservation

// MemoryReservation specifies the memory reservation of different types for each NUMA node
type MemoryReservation struct {
 NumaNode int32 `json:"numaNode"`
 Limits v1.ResourceList `json:"limits"`
}

Docs: Kube Proxy

Fri, 02 Dec 2022 17:30:52 +0800

简要概述

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration

代码路径：

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1beta3/types.go

配置示例

TODO;

数据结构

KubeProxyConfiguration

// KubeProxyConfiguration contains everything necessary to configure the
// Kubernetes proxy server.
type KubeProxyConfiguration struct {
 metav1.TypeMeta `json:",inline"`

 // featureGates is a map of feature names to bools that enable or disable alpha/experimental features.
 FeatureGates map[string]bool `json:"featureGates,omitempty"`

 // bindAddress is the IP address for the proxy server to serve on (set to 0.0.0.0
 // for all interfaces)
 BindAddress string `json:"bindAddress"`
 // healthzBindAddress is the IP address and port for the health check server to serve on,
 // defaulting to 0.0.0.0:10256
 HealthzBindAddress string `json:"healthzBindAddress"`
 // metricsBindAddress is the IP address and port for the metrics server to serve on,
 // defaulting to 127.0.0.1:10249 (set to 0.0.0.0 for all interfaces)
 MetricsBindAddress string `json:"metricsBindAddress"`
 // bindAddressHardFail, if true, kube-proxy will treat failure to bind to a port as fatal and exit
 BindAddressHardFail bool `json:"bindAddressHardFail"`
 // enableProfiling enables profiling via web interface on /debug/pprof handler.
 // Profiling handlers will be handled by metrics server.
 EnableProfiling bool `json:"enableProfiling"`
 // clusterCIDR is the CIDR range of the pods in the cluster. It is used to
 // bridge traffic coming from outside of the cluster. If not provided,
 // no off-cluster bridging will be performed.
 ClusterCIDR string `json:"clusterCIDR"`
 // hostnameOverride, if non-empty, will be used as the identity instead of the actual hostname.
 HostnameOverride string `json:"hostnameOverride"`
 // clientConnection specifies the kubeconfig file and client connection settings for the proxy
 // server to use when communicating with the apiserver.
 ClientConnection componentbaseconfigv1alpha1.ClientConnectionConfiguration `json:"clientConnection"`
 // iptables contains iptables-related configuration options.
 IPTables KubeProxyIPTablesConfiguration `json:"iptables"`
 // ipvs contains ipvs-related configuration options.
 IPVS KubeProxyIPVSConfiguration `json:"ipvs"`
 // oomScoreAdj is the oom-score-adj value for kube-proxy process. Values must be within
 // the range [-1000, 1000]
 OOMScoreAdj *int32 `json:"oomScoreAdj"`
 // mode specifies which proxy mode to use.
 Mode ProxyMode `json:"mode"`
 // portRange is the range of host ports (beginPort-endPort, inclusive) that may be consumed
 // in order to proxy service traffic. If unspecified (0-0) then ports will be randomly chosen.
 PortRange string `json:"portRange"`
 // udpIdleTimeout is how long an idle UDP connection will be kept open (e.g. '250ms', '2s').
 // Must be greater than 0. Only applicable for proxyMode=userspace.
 UDPIdleTimeout metav1.Duration `json:"udpIdleTimeout"`
 // conntrack contains conntrack-related configuration options.
 Conntrack KubeProxyConntrackConfiguration `json:"conntrack"`
 // configSyncPeriod is how often configuration from the apiserver is refreshed. Must be greater
 // than 0.
 ConfigSyncPeriod metav1.Duration `json:"configSyncPeriod"`
 // nodePortAddresses is the --nodeport-addresses value for kube-proxy process. Values must be valid
 // IP blocks. These values are as a parameter to select the interfaces where nodeport works.
 // In case someone would like to expose a service on localhost for local visit and some other interfaces for
 // particular purpose, a list of IP blocks would do that.
 // If set it to "127.0.0.0/8", kube-proxy will only select the loopback interface for NodePort.
 // If set it to a non-zero IP block, kube-proxy will filter that down to just the IPs that applied to the node.
 // An empty string slice is meant to select all network interfaces.
 NodePortAddresses []string `json:"nodePortAddresses"`
 // winkernel contains winkernel-related configuration options.
 Winkernel KubeProxyWinkernelConfiguration `json:"winkernel"`
 // ShowHiddenMetricsForVersion is the version for which you want to show hidden metrics.
 ShowHiddenMetricsForVersion string `json:"showHiddenMetricsForVersion"`
 // DetectLocalMode determines mode to use for detecting local traffic, defaults to LocalModeClusterCIDR
 DetectLocalMode LocalMode `json:"detectLocalMode"`
}

componentbaseconfigv1alpha1.ClientConnectionConfiguration

代码路径：kubernetes-1.22.16/staging/src/k8s.io/component-base/config/types.go

// ClientConnectionConfiguration contains details for constructing a client.
type ClientConnectionConfiguration struct {
 // kubeconfig is the path to a KubeConfig file.
 Kubeconfig string
 // acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the
 // default value of 'application/json'. This field will control all connections to the server used by a particular
 // client.
 AcceptContentTypes string
 // contentType is the content type used when sending data to the server from this client.
 ContentType string
 // qps controls the number of queries per second allowed for this connection.
 QPS float32
 // burst allows extra queries to accumulate when a client is exceeding its rate.
 Burst int32
}

KubeProxyIPTablesConfiguration

// KubeProxyIPTablesConfiguration contains iptables-related configuration
// details for the Kubernetes proxy server.
type KubeProxyIPTablesConfiguration struct {
 // masqueradeBit is the bit of the iptables fwmark space to use for SNAT if using
 // the pure iptables proxy mode. Values must be within the range [0, 31].
 MasqueradeBit *int32 `json:"masqueradeBit"`
 // masqueradeAll tells kube-proxy to SNAT everything if using the pure iptables proxy mode.
 MasqueradeAll bool `json:"masqueradeAll"`
 // syncPeriod is the period that iptables rules are refreshed (e.g. '5s', '1m',
 // '2h22m'). Must be greater than 0.
 SyncPeriod metav1.Duration `json:"syncPeriod"`
 // minSyncPeriod is the minimum period that iptables rules are refreshed (e.g. '5s', '1m',
 // '2h22m').
 MinSyncPeriod metav1.Duration `json:"minSyncPeriod"`
}

KubeProxyIPVSConfiguration

// KubeProxyIPVSConfiguration contains ipvs-related configuration
// details for the Kubernetes proxy server.
type KubeProxyIPVSConfiguration struct {
// syncPeriod is the period that ipvs rules are refreshed (e.g. '5s', '1m',
// '2h22m'). Must be greater than 0.
SyncPeriod metav1.Duration `json:"syncPeriod"`
// minSyncPeriod is the minimum period that ipvs rules are refreshed (e.g. '5s', '1m',
// '2h22m').
MinSyncPeriod metav1.Duration `json:"minSyncPeriod"`
// ipvs scheduler
Scheduler string `json:"scheduler"`
// excludeCIDRs is a list of CIDR's which the ipvs proxier should not touch
// when cleaning up ipvs services.
ExcludeCIDRs []string `json:"excludeCIDRs"`
// strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries
// from kube-ipvs0 interface
StrictARP bool `json:"strictARP"`
// tcpTimeout is the timeout value used for idle IPVS TCP sessions.
// The default value is 0, which preserves the current timeout value on the system.
TCPTimeout metav1.Duration `json:"tcpTimeout"`
// tcpFinTimeout is the timeout value used for IPVS TCP sessions after receiving a FIN.
// The default value is 0, which preserves the current timeout value on the system.
TCPFinTimeout metav1.Duration `json:"tcpFinTimeout"`
// udpTimeout is the timeout value used for IPVS UDP packets.
// The default value is 0, which preserves the current timeout value on the system.
UDPTimeout metav1.Duration `json:"udpTimeout"`
}

ProxyMode

// ProxyMode represents modes used by the Kubernetes proxy server.
//
// Currently, three modes of proxy are available in Linux platform: 'userspace' (older, going to be EOL), 'iptables'
// (newer, faster), 'ipvs'(newest, better in performance and scalability).
//
// Two modes of proxy are available in Windows platform: 'userspace'(older, stable) and 'kernelspace' (newer, faster).
//
// In Linux platform, if proxy mode is blank, use the best-available proxy (currently iptables, but may change in the
// future). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are
// insufficient, this always falls back to the userspace proxy. IPVS mode will be enabled when proxy mode is set to 'ipvs',
// and the fall back path is firstly iptables and then userspace.
//
// In Windows platform, if proxy mode is blank, use the best-available proxy (currently userspace, but may change in the
// future). If winkernel proxy is selected, regardless of how, but the Windows kernel can't support this mode of proxy,
// this always falls back to the userspace proxy.
type ProxyMode string

KubeProxyConntrackConfiguration

// KubeProxyConntrackConfiguration contains conntrack settings for
// the Kubernetes proxy server.
type KubeProxyConntrackConfiguration struct {
 // maxPerCore is the maximum number of NAT connections to track
 // per CPU core (0 to leave the limit as-is and ignore min).
 MaxPerCore *int32 `json:"maxPerCore"`
 // min is the minimum value of connect-tracking records to allocate,
 // regardless of conntrackMaxPerCore (set maxPerCore=0 to leave the limit as-is).
 Min *int32 `json:"min"`
 // tcpEstablishedTimeout is how long an idle TCP connection will be kept open
 // (e.g. '2s'). Must be greater than 0 to set.
 TCPEstablishedTimeout *metav1.Duration `json:"tcpEstablishedTimeout"`
 // tcpCloseWaitTimeout is how long an idle conntrack entry
 // in CLOSE_WAIT state will remain in the conntrack
 // table. (e.g. '60s'). Must be greater than 0 to set.
 TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"`
}

KubeProxyWinkernelConfiguration

// KubeProxyWinkernelConfiguration contains Windows/HNS settings for
// the Kubernetes proxy server.
type KubeProxyWinkernelConfiguration struct {
 // networkName is the name of the network kube-proxy will use
 // to create endpoints and policies
 NetworkName string `json:"networkName"`
 // sourceVip is the IP address of the source VIP endoint used for
 // NAT when loadbalancing
 SourceVip string `json:"sourceVip"`
 // enableDSR tells kube-proxy whether HNS policies should be created
 // with DSR
 EnableDSR bool `json:"enableDSR"`
}

LocalMode

// LocalMode represents modes to detect local traffic from the node
type LocalMode string

Docs: Kubectl

Thu, 14 Nov 2024 10:23:09 +0800

简要概述

TODO;

数据结构

Config

// Config holds the information needed to build connect to remote kubernetes clusters as a given user
// IMPORTANT if you add fields to this struct, please update IsConfigEmpty()
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type Config struct {
 // Legacy field from pkg/api/types.go TypeMeta.
 // TODO(jlowdermilk): remove this after eliminating downstream dependencies.
 // +k8s:conversion-gen=false
 // +optional
 Kind string `json:"kind,omitempty"`
 // Legacy field from pkg/api/types.go TypeMeta.
 // TODO(jlowdermilk): remove this after eliminating downstream dependencies.
 // +k8s:conversion-gen=false
 // +optional
 APIVersion string `json:"apiVersion,omitempty"`
 // Preferences holds general information to be use for cli interactions
 Preferences Preferences `json:"preferences"`
 // Clusters is a map of referencable names to cluster configs
 Clusters map[string]*Cluster `json:"clusters"`
 // AuthInfos is a map of referencable names to user configs
 AuthInfos map[string]*AuthInfo `json:"users"`
 // Contexts is a map of referencable names to context configs
 Contexts map[string]*Context `json:"contexts"`
 // CurrentContext is the name of the context that you would like to use by default
 CurrentContext string `json:"current-context"`
 // Extensions holds additional information. This is useful for extenders so that reads and writes don't clobber unknown fields
 // +optional
 Extensions map[string]runtime.Object `json:"extensions,omitempty"`
}

Preferences

// IMPORTANT if you add fields to this struct, please update IsConfigEmpty()
type Preferences struct {
 // +optional
 Colors bool `json:"colors,omitempty"`
 // Extensions holds additional information. This is useful for extenders so that reads and writes don't clobber unknown fields
 // +optional
 Extensions map[string]runtime.Object `json:"extensions,omitempty"`
}

Cluster

// Cluster contains information about how to communicate with a kubernetes cluster
type Cluster struct {
 // LocationOfOrigin indicates where this object came from. It is used for round tripping config post-merge, but never serialized.
 // +k8s:conversion-gen=false
 LocationOfOrigin string
 // Server is the address of the kubernetes cluster (https://hostname:port).
 Server string `json:"server"`
 // TLSServerName is used to check server certificate. If TLSServerName is empty, the hostname used to contact the server is used.
 // +optional
 TLSServerName string `json:"tls-server-name,omitempty"`
 // InsecureSkipTLSVerify skips the validity check for the server's certificate. This will make your HTTPS connections insecure.
 // +optional
 InsecureSkipTLSVerify bool `json:"insecure-skip-tls-verify,omitempty"`
 // CertificateAuthority is the path to a cert file for the certificate authority.
 // +optional
 CertificateAuthority string `json:"certificate-authority,omitempty"`
 // CertificateAuthorityData contains PEM-encoded certificate authority certificates. Overrides CertificateAuthority
 // +optional
 CertificateAuthorityData []byte `json:"certificate-authority-data,omitempty"`
 // ProxyURL is the URL to the proxy to be used for all requests made by this
 // client. URLs with "http", "https", and "socks5" schemes are supported. If
 // this configuration is not provided or the empty string, the client
 // attempts to construct a proxy configuration from http_proxy and
 // https_proxy environment variables. If these environment variables are not
 // set, the client does not attempt to proxy requests.
 //
 // socks5 proxying does not currently support spdy streaming endpoints (exec,
 // attach, port forward).
 // +optional
 ProxyURL string `json:"proxy-url,omitempty"`
 // DisableCompression allows client to opt-out of response compression for all requests to the server. This is useful
 // to speed up requests (specifically lists) when client-server network bandwidth is ample, by saving time on
 // compression (server-side) and decompression (client-side): https://github.com/kubernetes/kubernetes/issues/112296.
 // +optional
 DisableCompression bool `json:"disable-compression,omitempty"`
 // Extensions holds additional information. This is useful for extenders so that reads and writes don't clobber unknown fields
 // +optional
 Extensions map[string]runtime.Object `json:"extensions,omitempty"`
}

AuthInfo

// AuthInfo contains information that describes identity information. This is use to tell the kubernetes cluster who you are.
type AuthInfo struct {
 // LocationOfOrigin indicates where this object came from. It is used for round tripping config post-merge, but never serialized.
 // +k8s:conversion-gen=false
 LocationOfOrigin string
 // ClientCertificate is the path to a client cert file for TLS.
 // +optional
 ClientCertificate string `json:"client-certificate,omitempty"`
 // ClientCertificateData contains PEM-encoded data from a client cert file for TLS. Overrides ClientCertificate
 // +optional
 ClientCertificateData []byte `json:"client-certificate-data,omitempty"`
 // ClientKey is the path to a client key file for TLS.
 // +optional
 ClientKey string `json:"client-key,omitempty"`
 // ClientKeyData contains PEM-encoded data from a client key file for TLS. Overrides ClientKey
 // +optional
 ClientKeyData []byte `json:"client-key-data,omitempty" datapolicy:"security-key"`
 // Token is the bearer token for authentication to the kubernetes cluster.
 // +optional
 Token string `json:"token,omitempty" datapolicy:"token"`
 // TokenFile is a pointer to a file that contains a bearer token (as described above). If both Token and TokenFile are present, Token takes precedence.
 // +optional
 TokenFile string `json:"tokenFile,omitempty"`
 // Impersonate is the username to act-as.
 // +optional
 Impersonate string `json:"act-as,omitempty"`
 // ImpersonateUID is the uid to impersonate.
 // +optional
 ImpersonateUID string `json:"act-as-uid,omitempty"`
 // ImpersonateGroups is the groups to impersonate.
 // +optional
 ImpersonateGroups []string `json:"act-as-groups,omitempty"`
 // ImpersonateUserExtra contains additional information for impersonated user.
 // +optional
 ImpersonateUserExtra map[string][]string `json:"act-as-user-extra,omitempty"`
 // Username is the username for basic authentication to the kubernetes cluster.
 // +optional
 Username string `json:"username,omitempty"`
 // Password is the password for basic authentication to the kubernetes cluster.
 // +optional
 Password string `json:"password,omitempty" datapolicy:"password"`
 // AuthProvider specifies a custom authentication plugin for the kubernetes cluster.
 // +optional
 AuthProvider *AuthProviderConfig `json:"auth-provider,omitempty"`
 // Exec specifies a custom exec-based authentication plugin for the kubernetes cluster.
 // +optional
 Exec *ExecConfig `json:"exec,omitempty"`
 // Extensions holds additional information. This is useful for extenders so that reads and writes don't clobber unknown fields
 // +optional
 Extensions map[string]runtime.Object `json:"extensions,omitempty"`
}

Context

// Context is a tuple of references to a cluster (how do I communicate with a kubernetes cluster), a user (how do I identify myself), and a namespace (what subset of resources do I want to work with)
type Context struct {
 // LocationOfOrigin indicates where this object came from. It is used for round tripping config post-merge, but never serialized.
 // +k8s:conversion-gen=false
 LocationOfOrigin string
 // Cluster is the name of the cluster for this context
 Cluster string `json:"cluster"`
 // AuthInfo is the name of the authInfo for this context
 AuthInfo string `json:"user"`
 // Namespace is the default namespace to use on unspecified requests
 // +optional
 Namespace string `json:"namespace,omitempty"`
 // Extensions holds additional information. This is useful for extenders so that reads and writes don't clobber unknown fields
 // +optional
 Extensions map[string]runtime.Object `json:"extensions,omitempty"`
}

AuthProviderConfig

// AuthProviderConfig holds the configuration for a specified auth provider.
type AuthProviderConfig struct {
 Name string `json:"name"`
 // +optional
 Config map[string]string `json:"config,omitempty"`
}

ExecConfig

// ExecConfig specifies a command to provide client credentials. The command is exec'd
// and outputs structured stdout holding credentials.
//
// See the client.authentication.k8s.io API group for specifications of the exact input
// and output format
type ExecConfig struct {
 // Command to execute.
 Command string `json:"command"`
 // Arguments to pass to the command when executing it.
 // +optional
 Args []string `json:"args"`
 // Env defines additional environment variables to expose to the process. These
 // are unioned with the host's environment, as well as variables client-go uses
 // to pass argument to the plugin.
 // +optional
 Env []ExecEnvVar `json:"env"`

 // Preferred input version of the ExecInfo. The returned ExecCredentials MUST use
 // the same encoding version as the input.
 APIVersion string `json:"apiVersion,omitempty"`

 // This text is shown to the user when the executable doesn't seem to be
 // present. For example, `brew install foo-cli` might be a good InstallHint for
 // foo-cli on Mac OS systems.
 InstallHint string `json:"installHint,omitempty"`

 // ProvideClusterInfo determines whether or not to provide cluster information,
 // which could potentially contain very large CA data, to this exec plugin as a
 // part of the KUBERNETES_EXEC_INFO environment variable. By default, it is set
 // to false. Package k8s.io/client-go/tools/auth/exec provides helper methods for
 // reading this environment variable.
 ProvideClusterInfo bool `json:"provideClusterInfo"`

 // Config holds additional config data that is specific to the exec
 // plugin with regards to the cluster being authenticated to.
 //
 // This data is sourced from the clientcmd Cluster object's extensions[exec] field:
 //
 // clusters:
 // - name: my-cluster
 // cluster:
 // ...
 // extensions:
 // - name: client.authentication.k8s.io/exec # reserved extension name for per cluster exec config
 // extension:
 // audience: 06e3fbd18de8 # arbitrary config
 //
 // In some environments, the user config may be exactly the same across many clusters
 // (i.e. call this exec plugin) minus some details that are specific to each cluster
 // such as the audience. This field allows the per cluster config to be directly
 // specified with the cluster info. Using this field to store secret data is not
 // recommended as one of the prime benefits of exec plugins is that no secrets need
 // to be stored directly in the kubeconfig.
 // +k8s:conversion-gen=false
 Config runtime.Object

 // InteractiveMode determines this plugin's relationship with standard input. Valid
 // values are "Never" (this exec plugin never uses standard input), "IfAvailable" (this
 // exec plugin wants to use standard input if it is available), or "Always" (this exec
 // plugin requires standard input to function). See ExecInteractiveMode values for more
 // details.
 //
 // If APIVersion is client.authentication.k8s.io/v1alpha1 or
 // client.authentication.k8s.io/v1beta1, then this field is optional and defaults
 // to "IfAvailable" when unset. Otherwise, this field is required.
 // +optional
 InteractiveMode ExecInteractiveMode

 // StdinUnavailable indicates whether the exec authenticator can pass standard
 // input through to this exec plugin. For example, a higher level entity might be using
 // standard input for something else and therefore it would not be safe for the exec
 // plugin to use standard input. This is kept here in order to keep all of the exec configuration
 // together, but it is never serialized.
 // +k8s:conversion-gen=false
 StdinUnavailable bool

 // StdinUnavailableMessage is an optional message to be displayed when the exec authenticator
 // cannot successfully run this exec plugin because it needs to use standard input and
 // StdinUnavailable is true. For example, a process that is already using standard input to
 // read user instructions might set this to "used by my-program to read user instructions".
 // +k8s:conversion-gen=false
 StdinUnavailableMessage string
}

OPSaid – 集群部署

Docs: 阿里云

简要概述

云控制器

集群网络模式

方案一：在 VPC 添加路由

方案二：独立 VPC 子网

方案三：使用封装形式

注意事项

组件 CCM 正常但污点未移除

Docs: Init

简要概述

配置示例

数据结构

InitConfiguration

bootstraptokenv1.BootstrapToken

NodeRegistrationOptions

APIEndpoint

Docs: Etcd

简要概述

执行步骤

启动 containerd

启动 kubelet

生成 etcd ca

生成 etcd 配置

为每个etcd成员生成证书

传输各个etcd证书至各节点

生成 etcd pod

查看etcd集群状态

Docs: 快速开始

简要概述

前置说明

操作系统

主机拓扑

环境部署

ansible中控

负载均衡器

部署控制平面

步骤1：首次初始化前配置

步骤2：下载依赖的镜像包

步骤3：生成集群依赖证书

步骤4：系统公共组件安装

步骤5：部署控制平面节点

步骤6：验证控制平面组件

步骤7：部署集群基础组件

步骤8：本地配置存储集群

步骤9：控制平面相关维护

新增工作节点

步骤1：在ansible的host.ini中添加节点

步骤2：ansible中控节点可ssh节点

步骤3：系统公共组件安装

步骤4：添加集群工作节点

步骤5：验证允许节点加入

完成验收与后续

使用 calico 作为 CNI

Docs: Cluster

简要概述

数据结构

ClusterConfiguration

Etcd

Networking

APIServer

ControlPlaneComponent

DNS

HostPathMount

ImageMeta

Docs: 官方镜像

简要概述

控制平面

k8s 1.26 版本

k8s 1.22 版本

基本术语

Kubernetes镜像

下载Releasing包

获取二进制文件

解压各压缩包

导入容器镜像

pause镜像

etcd镜像

coredns镜像

步骤1：在ansible的`host.ini`中添加节点