OPSaid – 控制平面

Docs: kube-apiserver 配置

Mon, 19 Dec 2022 11:34:43 +0800

简要概述

一些关键参数说明。

参数解析

节点异常后 Pod 重建时间

配置以下参数，默认值为 300 秒：

--default-not-ready-toleration-seconds=300
--default-unreachable-toleration-seconds=300

同时开启 TaintNodesByCondition 控制器，默认也是启用的：

--enable-admission-plugins=TaintNodesByCondition

这样在创建生成 Pod 时，会自动添加以下 tolerations 分别表示对污点 “NotReady:NoExecute”、“Unreachable:NoExecute” 的容忍时长：

apiVersion: v1
kind: Pod
metadata:
spec:
 containers:
......
 tolerations:
 - effect: NoExecute
 key: node.kubernetes.io/not-ready
 operator: Exists
 tolerationSeconds: 300
 - effect: NoExecute
 key: node.kubernetes.io/unreachable
 operator: Exists
 tolerationSeconds: 300

以上说明当节点出现 “node.kubernetes.io/not-ready” 或 “node.kubernetes.io/unreachable” 时，该 Pod 还将在这个节点上运行多长时间，如默认集群节点判断不健康需要 40s 则以上发生 Pod 被驱逐需要 340s（40s + 300s）时间。

在集群级别应用 Pod 安全标准

涉及参数：

--admission-control-config-file

配置示例：

--admission-control-config-file=/etc/kubernetes/policies/admission-controls.yaml

cat <<EOF > /etc/kubernetes/policies/admission-controls.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
 configuration:
 apiVersion: pod-security.admission.config.k8s.io/v1
 kind: PodSecurityConfiguration
 defaults:
 enforce: "baseline"
 enforce-version: "latest"
 audit: "restricted"
 audit-version: "latest"
 warn: "restricted"
 warn-version: "latest"
 exemptions:
 usernames: []
 runtimeClasses: []
 namespaces: [kube-system]
EOF

这里使用 AdmissionConfiguration 配置了一个静态的 PodSecurity 准入控制器。说明如果违反 “baseline” 则不允许创建 Pod，如果违反 “restricted” 模式，则记录审计日志并在终端输出警告。具体配置可参考 Pod 安全标准。

Docs: kube-controller-manager 配置

Mon, 19 Dec 2022 11:34:43 +0800

简要概述

TODO;

参数解析

节点被标记为不健康的时间

涉及参数：

--node-monitor-grace-period=40s

在将一个 Node 标记为不健康之前允许其无响应的时长上限，必须比 kubelet 的 nodeStatusUpdateFrequency 大 N 倍，这里 N 指的是 kubelet 发送节点状态的重试次数。

为每个节点分配 cidr 地址

涉及参数：

--allocate-node-cidrs=true
--cluster-cidr=192.168.202.0/24
--node-cidr-mask-size=28

此时控制器将会按照 “cluster-cidr” 的网段为每个节点分配地址段，使用 “node-cidr-mask-size” 作为子网掩码并写入 Node 如：

# kubectl get node 192.168.0.2 -o yaml

apiVersion: v1
kind: Node
metadata:
 name: 192.168.0.2
spec:
 podCIDR: 192.168.202.0/28
 podCIDRs:
 - 192.168.202.0/28

如果 allocate-node-cidrs=false 时，为以下结果：

apiVersion: v1
kind: Node
metadata:
 name: 192.168.0.2
spec: {}

针对不同的 cni 实现不一定需要开启该功能，如使用 “cilium” 的 “ipam=kubernetes” 则需要开启。

Docs: kube-scheduler 配置

Mon, 01 Jan 0001 00:00:00 +0000

简要概述

kube-scheduler 主要配置已经均迁移至 KubeSchedulerConfiguration 不建议在启动命令行上配置。

KubeSchedulerConfiguration

github.com/kubernetes/kubernetes/src/k8s.io/kube-scheduler/config/v1/types.go

// KubeSchedulerConfiguration configures a scheduler
type KubeSchedulerConfiguration struct {
 metav1.TypeMeta `json:",inline"`

 // Parallelism defines the amount of parallelism in algorithms for scheduling a Pods. Must be greater than 0. Defaults to 16
 Parallelism *int32 `json:"parallelism,omitempty"`

 // LeaderElection defines the configuration of leader election client.
 LeaderElection componentbaseconfigv1alpha1.LeaderElectionConfiguration `json:"leaderElection"`

 // ClientConnection specifies the kubeconfig file and client connection
 // settings for the proxy server to use when communicating with the apiserver.
 ClientConnection componentbaseconfigv1alpha1.ClientConnectionConfiguration `json:"clientConnection"`

 // DebuggingConfiguration holds configuration for Debugging related features
 // TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfiguration
 componentbaseconfigv1alpha1.DebuggingConfiguration `json:",inline"`

 // PercentageOfNodesToScore is the percentage of all nodes that once found feasible
 // for running a pod, the scheduler stops its search for more feasible nodes in
 // the cluster. This helps improve scheduler's performance. Scheduler always tries to find
 // at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is.
 // Example: if the cluster size is 500 nodes and the value of this flag is 30,
 // then scheduler stops finding further feasible nodes once it finds 150 feasible ones.
 // When the value is 0, default percentage (5%--50% based on the size of the cluster) of the
 // nodes will be scored. It is overridden by profile level PercentageofNodesToScore.
 PercentageOfNodesToScore *int32 `json:"percentageOfNodesToScore,omitempty"`

 // PodInitialBackoffSeconds is the initial backoff for unschedulable pods.
 // If specified, it must be greater than 0. If this value is null, the default value (1s)
 // will be used.
 PodInitialBackoffSeconds *int64 `json:"podInitialBackoffSeconds,omitempty"`

 // PodMaxBackoffSeconds is the max backoff for unschedulable pods.
 // If specified, it must be greater than podInitialBackoffSeconds. If this value is null,
 // the default value (10s) will be used.
 PodMaxBackoffSeconds *int64 `json:"podMaxBackoffSeconds,omitempty"`

 // Profiles are scheduling profiles that kube-scheduler supports. Pods can
 // choose to be scheduled under a particular profile by setting its associated
 // scheduler name. Pods that don't specify any scheduler name are scheduled
 // with the "default-scheduler" profile, if present here.
 // +listType=map
 // +listMapKey=schedulerName
 Profiles []KubeSchedulerProfile `json:"profiles,omitempty"`

 // Extenders are the list of scheduler extenders, each holding the values of how to communicate
 // with the extender. These extenders are shared by all scheduler profiles.
 // +listType=set
 Extenders []Extender `json:"extenders,omitempty"`
}

OPSaid – 控制平面

Docs: kube-apiserver 配置

简要概述

参数解析

节点异常后 Pod 重建时间

在集群级别应用 Pod 安全标准

Docs: kube-controller-manager 配置

简要概述

参数解析

节点被标记为不健康的时间

为每个节点分配 cidr 地址

Docs: kube-scheduler 配置

简要概述

KubeSchedulerConfiguration

默认scheduler配置