OPSaid – 节点组件

Docs: 参数配置

Mon, 19 Dec 2022 11:34:43 +0800

简要概述

参数解析

命令行参数

外部组建参数

名称	默认值	来自组件	说明
add-dir-header	false	TODO;	在输出日志时，是否把代码文件所在目录一起提供，如：state/state_mem.go:36，否则仅输出：state_mem.go:36
alsologtostderr	-	klog	已弃用
application-metrics-count-limit	-	cadvisor	已废弃
azure-container-registry-config	-	azure	包含 Azure 容器镜像库配置信息的文件的路径
boot-id-file	-	cadvisor	默认来自：/proc/sys/kernel/random/boot_id
cloud-config	-	kubelet	已弃用
cloud-provider	-	kubelet	已弃用
cni-bin-dir	/opt/cni/bin	dockershim	已弃用 cni 的相关二进制地址
cni-cache-dir	/var/lib/cni/cache	dockershim	已弃用
cni-conf-dir	/etc/cni/net.d	dockershim	已弃用
container-hints	/etc/cadvisor/container_hints.json	cadvisor	已废弃
container-runtime	remote	kubelet	已废弃，目前支持 docker、remote，以后仅支持 remote
containerd	-	cadvisor	已废弃
containerd-namespace	-	cadvisor	已废弃
docker	-	cadvisor	已废弃
docker-endpoint	-	dockershim	已弃用
docker-env-metadata-whitelist	-	cadvisor	已废弃
docker-only	-	cadvisor	已废弃
docker-root	-	cadvisor	已废弃
docker-tls	-	cadvisor	已废弃
docker-tls-ca	-	cadvisor	已废弃
docker-tls-cert	-	cadvisor	已废弃
docker-tls-key	-	cadvisor	已废弃
dynamic-config-dir	-	kubelet	已废弃
enable-load-reader	-	cadvisor	已废弃
event-storage-age-limit	-	cadvisor	已废弃
event-storage-event-limit	-	cadvisor	已废弃
experimental-allocatable-ignore-eviction	false	kubelet	设置为 true 表示在计算节点可分配资源数量时忽略硬性逐出阈值设置
experimental-bootstrap-kubeconfig	-	kubelet	Use –bootstrap-kubeconfig
experimental-check-node-capabilities-before-mount	-	kubelet	已废弃
experimental-kernel-memcg-notification	-	kubelet	已废弃
experimental-mounter-path	-	kubelet	已废弃
global-housekeeping-interval	-	cadvisor	已废弃
image-pull-progress-deadline	-	dockershim	已弃用
keep-terminated-pod-volumes	-	kubelet	已废弃
log-backtrace-at	0	klog	已弃用
log-dir	-	klog	已弃用，如果此值为非空，则在所指定的目录中写入日志文件
log-file	-	klog	已弃用，如果此值非空，使用所给字符串作为日志文件名
log-file-max-size	1800	klog	已弃用，设置日志文件的最大值。单位为兆字节（M）。如果值为 0，则表示文件大小无限制
log-flush-frequency	5s	klog	两次日志刷新之间的最大秒数
logtostderr	true	klog	日志输出到 stderr 而不是文件
machine-id-file	-	cadvisor	/etc/machine-id,/var/lib/dbus/machine-id
master-service-namespace	-	kubelet	已弃用，kubelet 向 Pod 注入 Kubernetes 主控服务信息时使用的命名空间
maximum-dead-containers	-1	kubelet	已弃用，使用 eviction 代替
maximum-dead-containers-per-container	1	kubelet	已弃用，使用 eviction 代替
minimum-container-ttl-duration	-	kubelet	已弃用，使用 eviction 代替
network-plugin	-	dockershim	已弃用
network-plugin-mtu	-	dockershim	已弃用
non-masquerade-cidr	-	kubelet	已弃用
one-output	-	kubelet	已弃用
port	10250	kubelet	已弃用，kubelet 服务监听的本机端口号
really-crash-for-testing	-	kubelet	已弃用
register-schedulable	true	kubelet	已弃用
seccomp-profile-root	-	kubelet	已弃用
skip-headers	-	klog	已弃用
skip-log-headers	-	klog	已弃用
stderrthreshold	-	klog	已弃用
storage-driver-buffer-duration		-	cadvisor
storage-driver-db	-	cadvisor	已废弃
storage-driver-host	-	cadvisor	已废弃
storage-driver-password	-	cadvisor	已废弃
storage-driver-secure	-	cadvisor	已废弃
storage-driver-table	-	cadvisor	已废弃
storage-driver-user	-	cadvisor	已废弃
v	-	klog	日志级别详细程度的数值

组建内置参数

名称	默认值	KubeletConfiguration	说明
address	0.0.0.0	address	kubelet api 服务监听的地址
allowed-unsafe-sysctls	-	TODO;	允许不安全的 sysctl 设置
anonymous-auth	true	TODO;	是否可接受匿名请求，用户名为 system:anonymous，用户组为 system:unauthenticated
authentication-token-webhook	TODO	authentication.webhook.enabled	TODO；使用 TokenReview API 对持有者令牌进行身份认证
authentication-token-webhook-cache-ttl	2m0s	authentication.webhook.cacheTTL	对 Webhook 令牌认证组件所返回的响应的缓存时间
authorization-mode	AlwaysAllow 或 Webhook	authorization.mode	服务端鉴权模式，可选值：AlwaysAllow、Webhook，当设置 –config 时默认值为 Webhook
authorization-webhook-cache-authorized-ttl	5m0s	authorization.webhook.cacheAuthorizedTTL	对 Webhook 认证组件所返回的 “Authorized（已授权）” 应答的缓存时间
authorization-webhook-cache-unauthorized-ttl	30s	authorization.webhook.cacheUnauthorizedTTL	对 Webhook 认证组件所返回的 “Unauthorized（未授权）” 应答的缓存时间
bootstrap-kubeconfig	-	-	包含 apiserver 上配置的 known_tokens.csv 里面 token，用于获取 kubelet-client-current.pem 证书
cert-dir	/var/lib/kubelet/pki	-	kubelet.crt 与 kubelet.key 证书路径, 优先级低于 tls-cert-file 与 tls-private-key-file
cgroup-driver	cgroupfs	cgroupDriver	用来操作本机 cgroup 时使用的驱动程序。支持的选项包括 cgroupfs 和 systemd
cgroup-root	""	TODO	为 Pod 设置根 cgroup 默认将使用容器运行时的默认设置
cgroups-per-qos	true	TODO	启用创建 QoS cgroup 层次结构。此值为 true 时 kubelet 为 QoS 和 Pod 创建顶级的 cgroup
client-ca-file	-	authentication.x509.clientCAFile	TODO; 检查客户端证书，如 kube-apiserver 请求 kubelet ??
cluster-dns	-	clusterDNS	用于 Pod 中设置了 “dnsPolicy=ClusterFirst” 时为容器提供 DNS 服务
cluster-domain	cluster.local	clusterDomain	集群域名，如果设置了此值，kubelet 除了将主机的搜索域配置到所有容器之外，还会为其配置这里指定的域名
config	-	-	指向 KubeletConfiguration 配置文件，命令行参数会覆盖此文件中的配置
container-log-max-files	5	containerLogMaxFiles	设置容器的日志文件个数上限，必须不小于2且只能与 –container-runtime=remote 一起使用
container-log-max-size	10Mi	containerLogMaxSize	设置容器日志文件在轮换生成新文件时之前的最大值
container-runtime-endpoint	""	-	如：unix:///var/run/containerd/containerd.sock
contention-profiling	-	-	TODO; 启用锁竞争分析，前提是开启 pprof
cpu-cfs-quota	true	TODO	TODO; 为设置了 CPU 限制的容器启用 CPU CFS 配额保障
cpu-cfs-quota-period	100ms	TODO	TODO; 设置 CPU CFS 配额周期 cpu.cfs_period_us。默认使用 Linux 内核所设置的默认值
cpu-manager-policy	none	TODO	TODO; 要使用的 CPU 管理器策略。可选值包括：none 和 static
cpu-manager-policy-options	-	TODO	TODO; 一组用于微调其行为的 key=value CPU 管理器策略选项
cpu-manager-reconcile-period	10s	TODO	TODO; 设置 CPU 管理器的调和时间
enable-controller-attach-detach	true	TODO ｜启用 Attach/Detach 控制器来挂接和摘除调度到该节点的卷，同时禁用 kubelet 执行挂接和摘除操作
enable-debugging-handlers	true	TODO	启用服务器上用于日志收集和在本地运行容器和命令的端点
enable-server	true	TODO	启用 kubelet 服务器
enforce-node-allocatable	pods	TODO	用逗号分隔的列表，包含由 kubelet 强制执行的节点可分配资源级别
event-burst	10	TODO	事件记录的个数的突发峰值上限
event-qps	5	TODO	设置大于 0 的值表示限制每秒可生成的事件数量。设置为 0 表示不限制
eviction-hard	imagefs.available<15%,memory.available<100Mi,nodefs.available<10%	TODO	触发 Pod 驱逐操作的一组硬性门限
eviction-max-pod-grace-period	-	TODO	响应满足软性驱逐阈值（Soft Eviction Threshold）而终止 Pod 时使用的最长宽限期（以秒为单位
eviction-minimum-reclaim	-	TODO	当某资源压力过大时，kubelet 将执行 Pod 驱逐操作
eviction-pressure-transition-period	5m0s	TODO	在驱逐压力状况解除之前的最长等待时间
eviction-soft	-	TODO	设置一组驱逐阈值（例如：memory.available<1.5Gi）。如果在相应的宽限期内达到该阈值，则会触发 Pod 驱逐操作
eviction-soft-grace-period	-	TODO	设置一组驱逐宽限期（例如，memory.available=1m30s），对应于触发软性 Pod 驱逐操作之前软性驱逐阈值所需持续的时间长短
exit-on-lock-contention	-	-	设置为 true 表示当发生锁文件竞争时 kubelet 可以退出
experimental-logging-sanitization	-	TODO	对一些日志敏感字段做特殊处理
fail-swap-on	true	failSwapOn	设置为 true 表示如果主机启用了交换分区，kubelet 将直接失败
file-check-frequency	20s	TODO	检查配置文件中新数据的时间间隔
hairpin-mode	promiscuous-bridge	hairpinMode	设置 kubelet 执行 hairpin 网络地址转译的方式，该模式允许后端端点对其自身服务的访问能够再次经由负载均衡转发回自身，可选项包括 promiscuous-bridge、hairpin-veth 和 none
healthz-bind-address	127.0.0.1	healthzBindAddress	用于运行 healthz 服务器的 IP 地址
healthz-port	10248	healthzPort	本地 healthz 端点使用的端口（设置为 0 表示禁用）
hostname-override	-	-	将使用此字符串而不是实际的主机名作为节点标识
housekeeping-interval	10s	TODO	TODO
http-check-frequency	20s	TODO	HTTP 服务以获取新数据的时间间隔
image-credential-provider-bin-dir	-	-	TODO; 指向凭据提供组件可执行文件所在目录的路径
image-credential-provider-config	-	-	TODO; 指向凭据提供插件配置文件所在目录的路径
image-gc-high-threshold	85	TODO	镜像垃圾回收上限。磁盘使用空间达到该百分比时，镜像垃圾回收将持续工作。值必须在 [0，100] 范围内。要禁用镜像垃圾回收，请设置为 100
image-gc-low-threshold	80	TODO	镜像垃圾回收下限。磁盘使用空间在达到该百分比之前，镜像垃圾回收操作不会运行。值必须在 [0，100] 范围内，并且不得大于 high-threshold的值
image-service-endpoint	-	-	远程镜像服务的端点，未设置则同 container-runtime-endpoint
iptables-drop-bit	15	TODO	标记数据包将被丢弃的 fwmark 位设置。必须在 [0，31] 范围内
iptables-masquerade-bit	14	TODO	标记数据包将进行 SNAT 的 fwmark 空间位设置。必须在 [0，31] 范围内
kernel-memcg-notification	-	TODO	若启用，则 kubelet 将与内核中的 memcg 通知机制集成，不再使用轮询的方式来判定是否 Pod 达到内存驱逐阈值
kube-api-burst	10	TODO	每秒发送到 apiserver 的突发请求数量上限
kube-api-content-type	application/vnd.kubernetes.protobuf	TODO	发送到 apiserver 的请求的内容类型
kube-api-qps	5	TODO	与 apiserver 通信的每秒查询个数QPS，此值必须>=0，如果为0，则使用默认5。不包含事件和节点心跳 api，它们的速率限制是由一组不同的标志所控制
kube-reserved	-	kubeReserved	系统预留的资源配置，以一组 <资源名称>=<资源数量> 格式表示，支持：cpu、memory、ephemeral-storage
kube-reserved-cgroup	""	TODO	给出某个顶层 cgroup 绝对名称，该 cgroup 用于管理通过标志 –kube-reserved 为 kubernetes 组件所预留的计算资源
kubeconfig	-	-	指定如何连接到 API 服务器
kubelet-cgroups	-	TODO	用于创建和运行 kubelet 的 cgroup 的绝对名称
lock-file	-	-	kubelet 使用的锁文件的路径
log-json-info-buffer-size	0	TODO	实验性特性
log-json-split-stream	-	TODO	实验性特性
logging-format	text	TODO	设置日志文件格式。可以设置的格式有：“text”、“json”
make-iptables-util-chains	true	TODO	设置为 true 表示 kubelet 将确保 iptables 规则在主机上存在
manifest-url	-	TODO	用于访问要运行的其他 Pod 规范的 URL
manifest-url-header	-	TODO	请求 manifest-url 自定义的头
max-open-files	1000000	TODO	kubelet 进程可以打开的最大文件数量
max-pods	110	maxPods	此 kubelet 能运行的 Pod 最大数量
memory-manager-policy	None	TODO	内存管理器策略使用。可选值：‘None’、‘Static’
minimum-image-ttl-duration	2m0s	TODO	已结束的容器在被垃圾回收清理之前的最少存活时间。例如：‘300ms’、’10s’ 或者 ‘2h45m’
node-ip	-	-	节点的 IP 地址
node-labels	-	-	kubelet 在集群中注册本节点时设置的标签
node-status-max-images	50	TODO	在 node.status.images 中可以报告的最大镜像数量。如果指定为 -1，则不设上限
node-status-update-frequency	10s	TODO	指定 kubelet 向主控节点汇报节点状态的时间间隔。注意：更改此常量时请务必谨慎，它必须与节点控制器中的 nodeMonitorGracePeriod 一起使用
oom-score-adj	-999	oomScoreAdj	kubelet 进程的 oom-score-adj 参数值。有效范围为 [-1000，1000]
pod-cidr	-	TODO	用于给 Pod 分配 IP 地址的 CIDR 地址池，仅在独立运行模式下使用
pod-infra-container-image	k8s.gcr.io/pause:3.5	-	所指定的镜像不会被镜像垃圾收集器删除，依赖CRI
pod-manifest-path	-	staticPodPath	设置包含要运行的静态 Pod 的文件的路径
pod-max-pids	-1	TODO	设置每个 Pod 中的最大进程数目。如果为 -1，则 kubelet 使用节点可分配的 PID 容量作为默认值
pods-per-core	-	TODO	在每个处理器核上可运行的 Pod 数量
protect-kernel-defaults	-	TODO	TODO
provider-id	-	TODO	设置主机数据库中用来标识节点的唯一标识
qos-reserved	-	TODO	TODO
read-only-port	10255	TODO	kubelet 可以在没有身份验证/鉴权的情况下提供只读服务的端口（设置为 0 表示禁用）
register-node	true	TODO	向 API 服务器注册节点
register-with-taints	-	TODO	设置本节点的污点标记，格式为 =:，以逗号分隔
registry-burst	10	TODO	设置突发性镜像拉取的个数上限
registry-qps	5	TODO	如此值大于 0，可用来限制镜像仓库的 QPS 上限。设置为 0，表示不受限制
reserved-cpus	-	TODO	为系统和 Kubernetes 保留使用的 CPU，优先级高于 system-reserved 和 –kube-reskube-reserved
reserved-memory	-	TODO	TODO; 以逗号分隔的 NUMA 节点内存预留列表
resolv-conf	/etc/resolv.conf	TODO	名字解析服务的配置文件名，用作容器 DNS 解析配置的基础
root-dir	/var/lib/kubelet	-	设置用于管理 kubelet 文件的根目录
rotate-certificates	-	TODO	当客户端证书即将过期时 kubelet 自动从 kube-apiserver 请求新的证书进行轮换
rotate-server-certificates	-	TODO	当 kubelet 的服务证书即将过期时自动从 kube-apiserver 请求新的证书进行轮换
runonce	-	-	表示从本地清单或远程 URL 创建完 Pod 后立即退出 kubelet 进程
runtime-cgroups	-	-	TODO; 设置用于创建和运行容器运行时的 cgroup 的绝对名称
runtime-request-timeout	2m0s	TODO	设置除了长时间运行的请求（包括 pull、logs、exec 和 attach 等操作）之外的其他运行时请求的超时时间
seccomp-default	-	-	TODO; 启用 RuntimeDefault 作为所有工作负载的默认 seccomp 配置文件
serialize-image-pulls	true	serializeImagePulls	逐一拉取镜像
streaming-connection-idle-timeout	4h0m0s	TODO; 设置流连接在自动关闭之前可以空闲的最长时间。0 表示没有超时限制，与 kubelet 服务器的所有连接最长持续时间为 4 小时
sync-frequency	1m0s	TODO	在运行中的容器与其配置之间执行同步操作的最长时间间隔
system-cgroups	-	TODO	TODO
system-reserved	-	systemReserved	系统预留的资源配置，以一组资源名称=资源数量的格式表示
system-reserved-cgroup	""	TODO	TODO; 此标志给出一个顶层 cgroup 绝对名称，该 cgroup 用于管理非 kubernetes 组件
tls-cert-file	-	TODO	用于 HTTPS 认证，如未提供，则 kubelet 生成自签名证书和密钥，并将其保存到通过 –cert-dir 指定的目录
tls-cipher-suites	-	TODO	服务器端加密算法列表，以逗号分隔
tls-min-version	-	TODO	设置支持的最小 TLS 版本号，可选的版本号包括：VersionTLS10、 VersionTLS11、VersionTLS12 和 VersionTLS13
tls-private-key-file	-	TODO	同上
topology-manager-policy	none	TODO	设置拓扑管理策略（Topology Manager policy）。可选值包括：none、 best-effort、restricted 和 single-numa-node
topology-manager-scope	container	TODO	TODO; 拓扑提示信息使用范围
volume-plugin-dir	-	TODO	用来搜索第三方存储卷插件的目录
volume-stats-agg-period	1m0s	TODO	指定 kubelet 计算和缓存所有 Pod 和卷的磁盘用量总值的时间间隔。要禁用磁盘用量计算，请设置为 0

数据结构

KubeletConfiguration

// KubeletConfiguration contains the configuration for the Kubelet
type KubeletConfiguration struct {
 metav1.TypeMeta

 // enableServer enables Kubelet's secured server.
 // Note: Kubelet's insecure port is controlled by the readOnlyPort option.
 EnableServer bool
 // staticPodPath is the path to the directory containing local (static) pods to
 // run, or the path to a single static pod file.
 StaticPodPath string
 // syncFrequency is the max period between synchronizing running
 // containers and config
 SyncFrequency metav1.Duration
 // fileCheckFrequency is the duration between checking config files for
 // new data
 FileCheckFrequency metav1.Duration
 // httpCheckFrequency is the duration between checking http for new data
 HTTPCheckFrequency metav1.Duration
 // staticPodURL is the URL for accessing static pods to run
 StaticPodURL string
 // staticPodURLHeader is a map of slices with HTTP headers to use when accessing the podURL
 StaticPodURLHeader map[string][]string
 // address is the IP address for the Kubelet to serve on (set to 0.0.0.0
 // for all interfaces)
 Address string
 // port is the port for the Kubelet to serve on.
 Port int32
 // readOnlyPort is the read-only port for the Kubelet to serve on with
 // no authentication/authorization (set to 0 to disable)
 ReadOnlyPort int32
 // volumePluginDir is the full path of the directory in which to search
 // for additional third party volume plugins.
 VolumePluginDir string
 // providerID, if set, sets the unique id of the instance that an external provider (i.e. cloudprovider)
 // can use to identify a specific node
 ProviderID string
 // tlsCertFile is the file containing x509 Certificate for HTTPS. (CA cert,
 // if any, concatenated after server cert). If tlsCertFile and
 // tlsPrivateKeyFile are not provided, a self-signed certificate
 // and key are generated for the public address and saved to the directory
 // passed to the Kubelet's --cert-dir flag.
 TLSCertFile string
 // tlsPrivateKeyFile is the file containing x509 private key matching tlsCertFile
 TLSPrivateKeyFile string
 // TLSCipherSuites is the list of allowed cipher suites for the server.
 // Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
 TLSCipherSuites []string
 // TLSMinVersion is the minimum TLS version supported.
 // Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
 TLSMinVersion string
 // rotateCertificates enables client certificate rotation. The Kubelet will request a
 // new certificate from the certificates.k8s.io API. This requires an approver to approve the
 // certificate signing requests.
 RotateCertificates bool
 // serverTLSBootstrap enables server certificate bootstrap. Instead of self
 // signing a serving certificate, the Kubelet will request a certificate from
 // the certificates.k8s.io API. This requires an approver to approve the
 // certificate signing requests. The RotateKubeletServerCertificate feature
 // must be enabled.
 ServerTLSBootstrap bool
 // authentication specifies how requests to the Kubelet's server are authenticated
 Authentication KubeletAuthentication
 // authorization specifies how requests to the Kubelet's server are authorized
 Authorization KubeletAuthorization
 // registryPullQPS is the limit of registry pulls per second.
 // Set to 0 for no limit.
 RegistryPullQPS int32
 // registryBurst is the maximum size of bursty pulls, temporarily allows
 // pulls to burst to this number, while still not exceeding registryPullQPS.
 // Only used if registryPullQPS > 0.
 RegistryBurst int32
 // eventRecordQPS is the maximum event creations per second. If 0, there
 // is no limit enforced.
 EventRecordQPS int32
 // eventBurst is the maximum size of a burst of event creations, temporarily
 // allows event creations to burst to this number, while still not exceeding
 // eventRecordQPS. Only used if eventRecordQPS > 0.
 EventBurst int32
 // enableDebuggingHandlers enables server endpoints for log collection
 // and local running of containers and commands
 EnableDebuggingHandlers bool
 // enableContentionProfiling enables lock contention profiling, if enableDebuggingHandlers is true.
 EnableContentionProfiling bool
 // healthzPort is the port of the localhost healthz endpoint (set to 0 to disable)
 HealthzPort int32
 // healthzBindAddress is the IP address for the healthz server to serve on
 HealthzBindAddress string
 // oomScoreAdj is The oom-score-adj value for kubelet process. Values
 // must be within the range [-1000, 1000].
 OOMScoreAdj int32
 // clusterDomain is the DNS domain for this cluster. If set, kubelet will
 // configure all containers to search this domain in addition to the
 // host's search domains.
 ClusterDomain string
 // clusterDNS is a list of IP addresses for a cluster DNS server. If set,
 // kubelet will configure all containers to use this for DNS resolution
 // instead of the host's DNS servers.
 ClusterDNS []string
 // streamingConnectionIdleTimeout is the maximum time a streaming connection
 // can be idle before the connection is automatically closed.
 StreamingConnectionIdleTimeout metav1.Duration
 // nodeStatusUpdateFrequency is the frequency that kubelet computes node
 // status. If node lease feature is not enabled, it is also the frequency that
 // kubelet posts node status to master. In that case, be cautious when
 // changing the constant, it must work with nodeMonitorGracePeriod in nodecontroller.
 NodeStatusUpdateFrequency metav1.Duration
 // nodeStatusReportFrequency is the frequency that kubelet posts node
 // status to master if node status does not change. Kubelet will ignore this
 // frequency and post node status immediately if any change is detected. It is
 // only used when node lease feature is enabled.
 NodeStatusReportFrequency metav1.Duration
 // nodeLeaseDurationSeconds is the duration the Kubelet will set on its corresponding Lease.
 NodeLeaseDurationSeconds int32
 // imageMinimumGCAge is the minimum age for an unused image before it is
 // garbage collected.
 ImageMinimumGCAge metav1.Duration
 // imageGCHighThresholdPercent is the percent of disk usage after which
 // image garbage collection is always run. The percent is calculated as
 // this field value out of 100.
 ImageGCHighThresholdPercent int32
 // imageGCLowThresholdPercent is the percent of disk usage before which
 // image garbage collection is never run. Lowest disk usage to garbage
 // collect to. The percent is calculated as this field value out of 100.
 ImageGCLowThresholdPercent int32
 // How frequently to calculate and cache volume disk usage for all pods
 VolumeStatsAggPeriod metav1.Duration
 // KubeletCgroups is the absolute name of cgroups to isolate the kubelet in
 KubeletCgroups string
 // SystemCgroups is absolute name of cgroups in which to place
 // all non-kernel processes that are not already in a container. Empty
 // for no container. Rolling back the flag requires a reboot.
 SystemCgroups string
 // CgroupRoot is the root cgroup to use for pods.
 // If CgroupsPerQOS is enabled, this is the root of the QoS cgroup hierarchy.
 CgroupRoot string
 // Enable QoS based Cgroup hierarchy: top level cgroups for QoS Classes
 // And all Burstable and BestEffort pods are brought up under their
 // specific top level QoS cgroup.
 CgroupsPerQOS bool
 // driver that the kubelet uses to manipulate cgroups on the host (cgroupfs or systemd)
 CgroupDriver string
 // CPUManagerPolicy is the name of the policy to use.
 // Requires the CPUManager feature gate to be enabled.
 CPUManagerPolicy string
 // CPUManagerPolicyOptions is a set of key=value which allows to set extra options
 // to fine tune the behaviour of the cpu manager policies.
 // Requires both the "CPUManager" and "CPUManagerPolicyOptions" feature gates to be enabled.
 CPUManagerPolicyOptions map[string]string
 // CPU Manager reconciliation period.
 // Requires the CPUManager feature gate to be enabled.
 CPUManagerReconcilePeriod metav1.Duration
 // MemoryManagerPolicy is the name of the policy to use.
 // Requires the MemoryManager feature gate to be enabled.
 MemoryManagerPolicy string
 // TopologyManagerPolicy is the name of the policy to use.
 // Policies other than "none" require the TopologyManager feature gate to be enabled.
 TopologyManagerPolicy string
 // TopologyManagerScope represents the scope of topology hint generation
 // that topology manager requests and hint providers generate.
 // "pod" scope requires the TopologyManager feature gate to be enabled.
 // Default: "container"
 // +optional
 TopologyManagerScope string
 // Map of QoS resource reservation percentages (memory only for now).
 // Requires the QOSReserved feature gate to be enabled.
 QOSReserved map[string]string
 // runtimeRequestTimeout is the timeout for all runtime requests except long running
 // requests - pull, logs, exec and attach.
 RuntimeRequestTimeout metav1.Duration
 // hairpinMode specifies how the Kubelet should configure the container
 // bridge for hairpin packets.
 // Setting this flag allows endpoints in a Service to loadbalance back to
 // themselves if they should try to access their own Service. Values:
 // "promiscuous-bridge": make the container bridge promiscuous.
 // "hairpin-veth": set the hairpin flag on container veth interfaces.
 // "none": do nothing.
 // Generally, one must set --hairpin-mode=hairpin-veth to achieve hairpin NAT,
 // because promiscuous-bridge assumes the existence of a container bridge named cbr0.
 HairpinMode string
 // maxPods is the number of pods that can run on this Kubelet.
 MaxPods int32
 // The CIDR to use for pod IP addresses, only used in standalone mode.
 // In cluster mode, this is obtained from the master.
 PodCIDR string
 // The maximum number of processes per pod. If -1, the kubelet defaults to the node allocatable pid capacity.
 PodPidsLimit int64
 // ResolverConfig is the resolver configuration file used as the basis
 // for the container DNS resolution configuration.
 ResolverConfig string
 // RunOnce causes the Kubelet to check the API server once for pods,
 // run those in addition to the pods specified by static pod files, and exit.
 RunOnce bool
 // cpuCFSQuota enables CPU CFS quota enforcement for containers that
 // specify CPU limits
 CPUCFSQuota bool
 // CPUCFSQuotaPeriod sets the CPU CFS quota period value, cpu.cfs_period_us, defaults to 100ms
 CPUCFSQuotaPeriod metav1.Duration
 // maxOpenFiles is Number of files that can be opened by Kubelet process.
 MaxOpenFiles int64
 // nodeStatusMaxImages caps the number of images reported in Node.Status.Images.
 NodeStatusMaxImages int32
 // contentType is contentType of requests sent to apiserver.
 ContentType string
 // kubeAPIQPS is the QPS to use while talking with kubernetes apiserver
 KubeAPIQPS int32
 // kubeAPIBurst is the burst to allow while talking with kubernetes
 // apiserver
 KubeAPIBurst int32
 // serializeImagePulls when enabled, tells the Kubelet to pull images one at a time.
 SerializeImagePulls bool
 // Map of signal names to quantities that defines hard eviction thresholds. For example: {"memory.available": "300Mi"}.
 EvictionHard map[string]string
 // Map of signal names to quantities that defines soft eviction thresholds. For example: {"memory.available": "300Mi"}.
 EvictionSoft map[string]string
 // Map of signal names to quantities that defines grace periods for each soft eviction signal. For example: {"memory.available": "30s"}.
 EvictionSoftGracePeriod map[string]string
 // Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition.
 EvictionPressureTransitionPeriod metav1.Duration
 // Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met.
 EvictionMaxPodGracePeriod int32
 // Map of signal names to quantities that defines minimum reclaims, which describe the minimum
 // amount of a given resource the kubelet will reclaim when performing a pod eviction while
 // that resource is under pressure. For example: {"imagefs.available": "2Gi"}
 EvictionMinimumReclaim map[string]string
 // podsPerCore is the maximum number of pods per core. Cannot exceed MaxPods.
 // If 0, this field is ignored.
 PodsPerCore int32
 // enableControllerAttachDetach enables the Attach/Detach controller to
 // manage attachment/detachment of volumes scheduled to this node, and
 // disables kubelet from executing any attach/detach operations
 EnableControllerAttachDetach bool
 // protectKernelDefaults, if true, causes the Kubelet to error if kernel
 // flags are not as it expects. Otherwise the Kubelet will attempt to modify
 // kernel flags to match its expectation.
 ProtectKernelDefaults bool
 // If true, Kubelet ensures a set of iptables rules are present on host.
 // These rules will serve as utility for various components, e.g. kube-proxy.
 // The rules will be created based on IPTablesMasqueradeBit and IPTablesDropBit.
 MakeIPTablesUtilChains bool
 // iptablesMasqueradeBit is the bit of the iptables fwmark space to mark for SNAT
 // Values must be within the range [0, 31]. Must be different from other mark bits.
 // Warning: Please match the value of the corresponding parameter in kube-proxy.
 // TODO: clean up IPTablesMasqueradeBit in kube-proxy
 IPTablesMasqueradeBit int32
 // iptablesDropBit is the bit of the iptables fwmark space to mark for dropping packets.
 // Values must be within the range [0, 31]. Must be different from other mark bits.
 IPTablesDropBit int32
 // featureGates is a map of feature names to bools that enable or disable alpha/experimental
 // features. This field modifies piecemeal the built-in default values from
 // "k8s.io/kubernetes/pkg/features/kube_features.go".
 FeatureGates map[string]bool
 // Tells the Kubelet to fail to start if swap is enabled on the node.
 FailSwapOn bool
 // memorySwap configures swap memory available to container workloads.
 // +featureGate=NodeSwap
 // +optional
 MemorySwap MemorySwapConfiguration
 // A quantity defines the maximum size of the container log file before it is rotated. For example: "5Mi" or "256Ki".
 ContainerLogMaxSize string
 // Maximum number of container log files that can be present for a container.
 ContainerLogMaxFiles int32
 // ConfigMapAndSecretChangeDetectionStrategy is a mode in which config map and secret managers are running.
 ConfigMapAndSecretChangeDetectionStrategy ResourceChangeDetectionStrategy
 // A comma separated whitelist of unsafe sysctls or sysctl patterns (ending in *).
 // Unsafe sysctl groups are kernel.shm*, kernel.msg*, kernel.sem, fs.mqueue.*, and net.*.
 // These sysctls are namespaced but not allowed by default. For example: "kernel.msg*,net.ipv4.route.min_pmtu"
 // +optional
 AllowedUnsafeSysctls []string
 // kernelMemcgNotification if enabled, the kubelet will integrate with the kernel memcg
 // notification to determine if memory eviction thresholds are crossed rather than polling.
 KernelMemcgNotification bool

 /* the following fields are meant for Node Allocatable */

 // A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G,pid=100) pairs
 // that describe resources reserved for non-kubernetes components.
 // Currently only cpu and memory are supported.
 // See http://kubernetes.io/docs/user-guide/compute-resources for more detail.
 SystemReserved map[string]string
 // A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G,pid=100) pairs
 // that describe resources reserved for kubernetes system components.
 // Currently cpu, memory and local ephemeral storage for root file system are supported.
 // See http://kubernetes.io/docs/user-guide/compute-resources for more detail.
 KubeReserved map[string]string
 // This flag helps kubelet identify absolute name of top level cgroup used to enforce `SystemReserved` compute resource reservation for OS system daemons.
 // Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) doc for more information.
 SystemReservedCgroup string
 // This flag helps kubelet identify absolute name of top level cgroup used to enforce `KubeReserved` compute resource reservation for Kubernetes node system daemons.
 // Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) doc for more information.
 KubeReservedCgroup string
 // This flag specifies the various Node Allocatable enforcements that Kubelet needs to perform.
 // This flag accepts a list of options. Acceptable options are `pods`, `system-reserved` & `kube-reserved`.
 // Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) doc for more information.
 EnforceNodeAllocatable []string
 // This option specifies the cpu list reserved for the host level system threads and kubernetes related threads.
 // This provide a "static" CPU list rather than the "dynamic" list by system-reserved and kube-reserved.
 // This option overwrites CPUs provided by system-reserved and kube-reserved.
 ReservedSystemCPUs string
 // The previous version for which you want to show hidden metrics.
 // Only the previous minor version is meaningful, other values will not be allowed.
 // The format is <major>.<minor>, e.g.: '1.16'.
 // The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics,
 // rather than being surprised when they are permanently removed in the release after that.
 ShowHiddenMetricsForVersion string
 // Logging specifies the options of logging.
 // Refer [Logs Options](https://github.com/kubernetes/component-base/blob/master/logs/options.go) for more information.
 Logging componentbaseconfig.LoggingConfiguration
 // EnableSystemLogHandler enables /logs handler.
 EnableSystemLogHandler bool
 // ShutdownGracePeriod specifies the total duration that the node should delay the shutdown and total grace period for pod termination during a node shutdown.
 // Defaults to 0 seconds.
 // +featureGate=GracefulNodeShutdown
 // +optional
 ShutdownGracePeriod metav1.Duration
 // ShutdownGracePeriodCriticalPods specifies the duration used to terminate critical pods during a node shutdown. This should be less than ShutdownGracePeriod.
 // Defaults to 0 seconds.
 // For example, if ShutdownGracePeriod=30s, and ShutdownGracePeriodCriticalPods=10s, during a node shutdown the first 20 seconds would be reserved for gracefully terminating normal pods, and the last 10 seconds would be reserved for terminating critical pods.
 // +featureGate=GracefulNodeShutdown
 // +optional
 ShutdownGracePeriodCriticalPods metav1.Duration
 // ReservedMemory specifies a comma-separated list of memory reservations for NUMA nodes.
 // The parameter makes sense only in the context of the memory manager feature. The memory manager will not allocate reserved memory for container workloads.
 // For example, if you have a NUMA0 with 10Gi of memory and the ReservedMemory was specified to reserve 1Gi of memory at NUMA0,
 // the memory manager will assume that only 9Gi is available for allocation.
 // You can specify a different amount of NUMA node and memory types.
 // You can omit this parameter at all, but you should be aware that the amount of reserved memory from all NUMA nodes
 // should be equal to the amount of memory specified by the node allocatable features(https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable).
 // If at least one node allocatable parameter has a non-zero value, you will need to specify at least one NUMA node.
 // Also, avoid specifying:
 // 1. Duplicates, the same NUMA node, and memory type, but with a different value.
 // 2. zero limits for any memory type.
 // 3. NUMAs nodes IDs that do not exist under the machine.
 // 4. memory types except for memory and hugepages-<size>
 ReservedMemory []MemoryReservation
 // EnableProfiling enables /debug/pprof handler.
 EnableProfilingHandler bool
 // EnableDebugFlagsHandler enables/debug/flags/v handler.
 EnableDebugFlagsHandler bool
 // SeccompDefault enables the use of `RuntimeDefault` as the default seccomp profile for all workloads.
 SeccompDefault bool
 // MemoryThrottlingFactor specifies the factor multiplied by the memory limit or node allocatable memory
 // when setting the cgroupv2 memory.high value to enforce MemoryQoS.
 // Decreasing this factor will set lower high limit for container cgroups and put heavier reclaim pressure
 // while increasing will put less reclaim pressure.
 // See http://kep.k8s.io/2570 for more details.
 // Default: 0.8
 // +featureGate=MemoryQoS
 // +optional
 MemoryThrottlingFactor *float64
}

Docs: 参数配置

Mon, 01 Jan 0001 00:00:00 +0000

简要概述

主要用于控制 iptables、ipvs 在宿主上的网络流量。

配置示例

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
bindAddress: 0.0.0.0
bindAddressHardFail: false
clientConnection:
 acceptContentTypes: ""
 burst: 10
 contentType: application/vnd.kubernetes.protobuf
 kubeconfig: "/var/lib/kube-proxy/kubeconfig.conf"
 qps: 5
clusterCIDR: 192.168.201.0/24
configSyncPeriod: 15m0s
conntrack:
 maxPerCore: 32768
 min: 131072
 tcpCloseWaitTimeout: 1h0m0s
 tcpEstablishedTimeout: 24h0m0s
detectLocalMode: ""
enableProfiling: false
healthzBindAddress: 127.0.0.1:10256
hostnameOverride: ""
iptables:
 masqueradeAll: false
 masqueradeBit: 14
 minSyncPeriod: 1s
 syncPeriod: 30s
ipvs:
 excludeCIDRs: null
 minSyncPeriod: 0s
 scheduler: rr
 strictARP: false
 syncPeriod: 30s
 tcpFinTimeout: 0s
 tcpTimeout: 0s
 udpTimeout: 0s
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses:
- 192.168.201.0/24
oomScoreAdj: -999
portRange: ""
showHiddenMetricsForVersion: ""
udpIdleTimeout: 250ms
winkernel:
 enableDSR: false
 networkName: ""
 sourceVip: ""

参数解析

除命令行参数：–config, –write-config-to, and –cleanup 外，其他均建议通过 KubeProxyConfiguration 配置。

外部组件

–boot-id-file –log-backtrace-at –log-dir –log-file –log-file-max-size –log-flush-frequency –logtostderr –machine-id-file –one-output –profiling –skip-headers –skip-log-headers –stderrthreshold

内置组件

名称	KubeProxyConfiguration	默认值	说明
add-dir-header	-	false	同 kubelet
alsologtostderr	-	false	同上
bind-address	bindAddress	0.0.0.0	服务端监听的地址
bind-address-hard-fail	bindAddressHardFail	false	如果无法绑定端口，则将视为致命错误并退出
cleanup	-	false	清除 iptabels 与 ipvs 规则后退出
cluster-cidr	clusterCIDR	""	集群中 Pod 的 CIDR 范围，配置后将从该范围之外发送到集群 IP 的流量被伪装，从 Pod 发送到外部负载均衡器 IP 的流量将被重定向到相应的集群 IP
config	-	""	配置 KubeProxyConfiguration 文件的路径
config-sync-period	configSyncPeriod	15m0s	TODO; 来自 apiserver 的配置的刷新频率
conntrack-max-per-core	conntrack.maxPerCore	32768	每个 CPU 核跟踪的最大 NAT 连接数（0 表示保留当前限制并忽略 conntrack-min 设置）
conntrack-min	conntrack.min	131072	无论上面设置为多少，要分配的 conntrack 条目的最小数量（将 conntrack-max-per-core 设置为 0 即可保持当前的限制）
conntrack-tcp-timeout-close-wait	conntrack.tcpCloseWaitTimeout	1h0m0s	TODO; 处于 CLOSE_WAIT 状态的 TCP 连接的 NAT 超时
conntrack-tcp-timeout-established	conntrack.tcpEstablishedTimeout	24h0m0s	TODO; 已建立的 TCP 连接的空闲超时（0 保持当前设置）
detect-local-mode	detectLocalMode	""	TODO; 用于检测本地流量的模式，可配置：ClusterCIDR、NodeCIDR
feature-gates	-	""	开启的特性
healthz-bind-address	healthzBindAddress	0.0.0.0:10256	服务健康状态检查的 IP 地址和端口，设置为空表示禁用
hostname-override	hostnameOverride	""	如果非空，将使用此字符串而不是实际的主机名作为标识
iptables-masquerade-bit	iptables.masqueradeBit	14	TODO; 在使用纯 iptables 代理时，用来设置 fwmark 空间的 bit，标记需要 SNAT 的数据包。必须在 [0,31] 范围内。
iptables-min-sync-period	iptables.minSyncPeriod	1s	iptables 规则可以随着端点和服务的更改而刷新的最小间隔
iptables-sync-period	iptables.syncPeriod	30s	刷新 iptables 规则的最大间隔（例如 ‘5s’、‘1m’、‘2h22m’）。必须大于 0。
ipvs-exclude-cidrs	ipvs.excludeCIDRs	""	逗号分隔的 CIDR 列表，ipvs 代理在清理 IPVS 规则时不会此列表中的地址范围。
ipvs-min-sync-period	ipvs.minSyncPeriod	0s	ipvs 规则可以随着端点和服务的更改而刷新的最小间隔
ipvs-scheduler	ipvs.scheduler	""	ipvs 所选的调度器类型
ipvs-strict-arp	ipvs.strictARP	false	通过将 arp_ignore 设置为 1 并将 arp_announce 设置为 2 启用严格的 ARP
ipvs-sync-period	ipvs.syncPeriod	30s	刷新 ipvs 规则的最大间隔（例如 ‘5s’、‘1m’、‘2h22m’）。必须大于 0
ipvs-tcp-timeout	ipvs.tcpTimeout	0s	空闲 IPVS TCP 连接的超时时间，0 保持连接
ipvs-tcpfin-timeout	ipvs.tcpFinTimeout	0s	收到 FIN 数据包后，IPVS TCP 连接的超时，0 保持当前设置不变
ipvs-udp-timeout	ipvs.udpTimeout	0s	IPVS UDP 数据包的超时，0 保持当前设置不变
kube-api-burst	clientConnection.burst	10	与 kubernetes apiserver 通信的突发数量
kube-api-content-type	clientConnection.contentType	application/vnd.kubernetes.protobuf	发送到 apiserver 的请求的内容类型
kube-api-qps	clientConnection.qps	5	与 kubernetes apiserver 交互时使用的 QPS
kubeconfig	clientConnection.kubeconfig	""	连接 apiserver 的鉴权文件
masquerade-all	iptabels.masqueradeAll	false	如果使用纯 iptables 代理，则对通过服务集群 IP 发送的所有流量进行 SNAT（通常不需要）
master	-	""	Kubernetes API 服务器的地址，会覆盖 kubeconfig 中的值
metrics-bind-address	metricsBindAddress	127.0.0.1:10249	性能数据 /metrics 的地址
nodeport-addresses	nodePortAddresses	""	指定用于 NodePort 服务的地址，一个有效的 CIDR 地址段
oom-score-adj	oomScoreAdj	-999	kube-proxy 进程中的 oom-score-adj 值，必须在 [-1000,1000] 范围内
profiling	enableProfiling	false	如果为 true，则通过 Web 接口 /debug/pprof 启用性能分析
proxy-mode	mode	""	四种代理模式：userspace, iptables, ipvs, kernelspace，默认为 iptables
proxy-port-range	portRange	0-0	用来代理 nodePort 的主机端口范围，格式：beginPort-endPort，默认为 30000 - 32767
show-hidden-metrics-for-version	showHiddenMetricsForVersion	""	要显示隐藏指标的先前版本，仅先前的次要版本有意义，不允许其他值，如：“1.16”
udp-timeout	udpIdleTimeout	250ms	空闲 UDP 连接将保持打开的时长，必须大于 0，仅适用于 proxy-mode=userspace
write-config-to	-	-	将默认配置信息写入此文件并退出

数据结构

KubeProxyConfiguration

// KubeProxyConfiguration contains everything necessary to configure the
// Kubernetes proxy server.
type KubeProxyConfiguration struct {
 metav1.TypeMeta

 // featureGates is a map of feature names to bools that enable or disable alpha/experimental features.
 FeatureGates map[string]bool

 // bindAddress is the IP address for the proxy server to serve on (set to 0.0.0.0
 // for all interfaces)
 BindAddress string
 // healthzBindAddress is the IP address and port for the health check server to serve on,
 // defaulting to 0.0.0.0:10256
 HealthzBindAddress string
 // metricsBindAddress is the IP address and port for the metrics server to serve on,
 // defaulting to 127.0.0.1:10249 (set to 0.0.0.0 for all interfaces)
 MetricsBindAddress string
 // BindAddressHardFail, if true, kube-proxy will treat failure to bind to a port as fatal and exit
 BindAddressHardFail bool
 // enableProfiling enables profiling via web interface on /debug/pprof handler.
 // Profiling handlers will be handled by metrics server.
 EnableProfiling bool
 // clusterCIDR is the CIDR range of the pods in the cluster. It is used to
 // bridge traffic coming from outside of the cluster. If not provided,
 // no off-cluster bridging will be performed.
 ClusterCIDR string
 // hostnameOverride, if non-empty, will be used as the identity instead of the actual hostname.
 HostnameOverride string
 // clientConnection specifies the kubeconfig file and client connection settings for the proxy
 // server to use when communicating with the apiserver.
 ClientConnection componentbaseconfig.ClientConnectionConfiguration
 // iptables contains iptables-related configuration options.
 IPTables KubeProxyIPTablesConfiguration
 // ipvs contains ipvs-related configuration options.
 IPVS KubeProxyIPVSConfiguration
 // oomScoreAdj is the oom-score-adj value for kube-proxy process. Values must be within
 // the range [-1000, 1000]
 OOMScoreAdj *int32
 // mode specifies which proxy mode to use.
 Mode ProxyMode
 // portRange is the range of host ports (beginPort-endPort, inclusive) that may be consumed
 // in order to proxy service traffic. If unspecified (0-0) then ports will be randomly chosen.
 PortRange string
 // udpIdleTimeout is how long an idle UDP connection will be kept open (e.g. '250ms', '2s').
 // Must be greater than 0. Only applicable for proxyMode=userspace.
 UDPIdleTimeout metav1.Duration
 // conntrack contains conntrack-related configuration options.
 Conntrack KubeProxyConntrackConfiguration
 // configSyncPeriod is how often configuration from the apiserver is refreshed. Must be greater
 // than 0.
 ConfigSyncPeriod metav1.Duration
 // nodePortAddresses is the --nodeport-addresses value for kube-proxy process. Values must be valid
 // IP blocks. These values are as a parameter to select the interfaces where nodeport works.
 // In case someone would like to expose a service on localhost for local visit and some other interfaces for
 // particular purpose, a list of IP blocks would do that.
 // If set it to "127.0.0.0/8", kube-proxy will only select the loopback interface for NodePort.
 // If set it to a non-zero IP block, kube-proxy will filter that down to just the IPs that applied to the node.
 // An empty string slice is meant to select all network interfaces.
 NodePortAddresses []string
 // winkernel contains winkernel-related configuration options.
 Winkernel KubeProxyWinkernelConfiguration
 // ShowHiddenMetricsForVersion is the version for which you want to show hidden metrics.
 ShowHiddenMetricsForVersion string
 // DetectLocalMode determines mode to use for detecting local traffic, defaults to LocalModeClusterCIDR
 DetectLocalMode LocalMode
}

配置说明

clusterCIDR

如果 clusterCIDR 配置为 192.168.209.0/24，则会创建以下 iptables 规则：

-A KUBE-SERVICES ! -s 192.168.209.0/24 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ

conntrack

数量限制

查看当前系统支持最大的 conntrack 数量

cat /proc/sys/net/netfilter/nf_conntrack_max

或者

sysctl -a | grep net.netfilter.nf_conntrack_max

当 conntrack.maxPerCore=32768 且为 2个cpu核心的系统，则最大 nf_conntrack_max = 32768 * 2 = 65536，然后这个不应该小于 conntrack.min 设置的条目。

超时限制

其中 net.netfilter.nf_conntrack_tcp_timeout_close_wait 等待时间是让被动关闭方把该传的数据传完。如果程序写得不好，这里抛了未捕捉的异常，也许就走不到发 FIN 那步了，一直停在这里