OPSaid – Ingress NGINX

Docs: User guide

Sun, 11 Sep 2022 11:13:40 +0800

版本信息

以下配置说明均基于官方版本 1.31 做说明。

Support Versions table

Ingress-NGINX version	k8s supported version	Alpine Version	Nginx Version
v1.3.1	1.24, 1.23, 1.22, 1.21, 1.20	3.16.2	1.19.10†
v1.3.0	1.24, 1.23, 1.22, 1.21, 1.20	3.16.0	1.19.10†
v1.1.1	1.23, 1.22, 1.21, 1.20, 1.19	3.14.2	1.19.9†
v1.0.0	1.22, 1.21, 1.20, 1.19	3.13.5	1.20.1

安装部署

配置示例

参数解析

可用的Annotations

这里仅记录个人认为有比较有用处的几个注解，全部列表可以参考官方文档说明。

在ingress上annotation的键值对只能是字符串类型，数字或布尔类型需通过引号包含，如：“100”, “false”。默认键前缀是"nginx.ingress.kubernetes.io"，由参数"–annotations-prefix"控制。

名称	说明	默认值	可取值
nginx.ingress.kubernetes.io/app-root	当应用请求"/“时做302重定向的目标url	-	-
nginx.ingress.kubernetes.io/affinity	用于保持会话连接	-	cookie
nginx.ingress.kubernetes.io/affinity-mode	同上	balanced	“balanced” or “persistent”
nginx.ingress.kubernetes.io/affinity-canary-behavior	同上	sticky	“sticky” or “legacy”
nginx.ingress.kubernetes.io/session-cookie-name	同上	INGRESSCOOKIE	-
nginx.ingress.kubernetes.io/session-cookie-path	同上	/	-
nginx.ingress.kubernetes.io/session-cookie-secure	同上；只能用https协议发送给服务器	false	true, false
nginx.ingress.kubernetes.io/session-cookie-samesite	同上；Chrome 51 开始，浏览器的 Cookie 新增加了一个SameSite属性，用来防止 CSRF 攻击和用户追踪	-	“None”, “Lax”, “Strict”
nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none	同上	-	“true”, “false”
nginx.ingress.kubernetes.io/auth-type	认证	-	“basic” or “digest”
nginx.ingress.kubernetes.io/auth-secret	同上；存放认证"secret"资源的名称	-	-
nginx.ingress.kubernetes.io/auth-secret-type	同上；存放认证"secret"资源的格式	-	“auth-file”, “auth-map”
nginx.ingress.kubernetes.io/auth-realm	认证	-	-
nginx.ingress.kubernetes.io/canary	灰度、金丝雀发布	-	“true” or “false”
nginx.ingress.kubernetes.io/canary-by-header	同上，基于HTTP请求头的流量调度，这里对于请求头名称，对应值可取"always"或"never”	-	-
nginx.ingress.kubernetes.io/canary-by-header-value	同上，自定义匹配请求头的值，以代替"always"与"never"	-	-
nginx.ingress.kubernetes.io/canary-by-header-pattern	同上	-	-
nginx.ingress.kubernetes.io/canary-by-cookie	同上，基于Cookie的流量调度，这里对应cookie名称，对应值可取"always"或"never"	-	-
nginx.ingress.kubernetes.io/canary-weight	同上，流量调度到灰度副本的权重，填写数字如：30	-	-
nginx.ingress.kubernetes.io/canary-weight-total	同上，总权重是多少，配合上面使用	-	100
[nginx.ingress.kubernetes.io/client-body-buffer-size]	TODO; HTTP请求体缓冲区大小，如果请求超过这个值会写入nginx配置的temp_path做个缓冲	-	8k
[nginx.ingress.kubernetes.io/configuration-snippet]	TODO; 可自定义nginx配置，比较危险，建议关闭	-	-
[nginx.ingress.kubernetes.io/server-snippet]	TODO; 可自定义nginx配置，比较危险，建议关闭	-	-
[nginx.ingress.kubernetes.io/stream-snippet]	TODO; 可自定义nginx配置，比较危险，建议关闭	-	-
[nginx.ingress.kubernetes.io/custom-http-errors]	TODO; 设置 proxy-intercept-errors	-	-
[nginx.ingress.kubernetes.io/default-backend]	TODO；访问该ingress如果后端没有活跃地址，则转发至同空间下配置的服务名, 优先级高于全局配置	-	-
[nginx.ingress.kubernetes.io/enable-cors]	TODO; 开启跨域	-	-
[nginx.ingress.kubernetes.io/force-ssl-redirect]	TODO; 如果该ingress开启ssl，是否强制跳转http，通过http状态码308	-	-
[nginx.ingress.kubernetes.io/ssl-redirect]	TODO;	-	-
[nginx.ingress.kubernetes.io/from-to-www-redirect]	TODO;	-	-
[nginx.ingress.kubernetes.io/permanent-redirect]	TODO;	-	-
[nginx.ingress.kubernetes.io/permanent-redirect-code]	TODO;	-	-
[nginx.ingress.kubernetes.io/temporal-redirect]	TODO;	-	-
[nginx.ingress.kubernetes.io/preserve-trailing-slash]	TODO;	-	-
[nginx.ingress.kubernetes.io/enable-rewrite-log] TODO; 是否开启rewrite日志，默认为关闭	-	-
[nginx.ingress.kubernetes.io/rewrite-target]	TODO;	-	-
[nginx.ingress.kubernetes.io/http2-push-preload]	TODO; HTTP2推送预加载	-	-
[nginx.ingress.kubernetes.io/limit-connections]	TODO; 请求限制	-	-
[nginx.ingress.kubernetes.io/proxy-body-size]	TODO; 代理配置	-	-
[nginx.ingress.kubernetes.io/upstream-hash-by]	TODO; 代理后段算法	-	-
[nginx.ingress.kubernetes.io/load-balance]	TODO; 代理后段算法	-	-
[nginx.ingress.kubernetes.io/satisfy]	TODO; 不清楚用途	-	-
[nginx.ingress.kubernetes.io/server-alias]	TODO; 设置别名，不能冲突	-	-
[nginx.ingress.kubernetes.io/ssl-passthrough]	TODO;	“false”	“true” or “false”
[nginx.ingress.kubernetes.io/x-forwarded-prefix]	TODO;	-	-
[nginx.ingress.kubernetes.io/upstream-vhost]	TODO;	-	-
[nginx.ingress.kubernetes.io/enable-access-log]	TODO;	-	-
[nginx.ingress.kubernetes.io/enable-opentracing]	TODO;	-	-
[nginx.ingress.kubernetes.io/opentracing-trust-incoming-span]	TODO;	-	-
[nginx.ingress.kubernetes.io/enable-influxdb]	写入性能数据至influxdb，在k8s上使用prometheus，这个不建议开启	“false”	“true” or “false”
[nginx.ingress.kubernetes.io/influxdb-measurement]	同上	-	-
[nginx.ingress.kubernetes.io/influxdb-port]	同上	-	-
[nginx.ingress.kubernetes.io/influxdb-host]	同上	-	-
[nginx.ingress.kubernetes.io/influxdb-server-name]	同上	-	-
[nginx.ingress.kubernetes.io/use-regex]	TODO;	-	-
[nginx.ingress.kubernetes.io/enable-modsecurity]	开启modsecurity，仅在专用集群使用，业务集群不开启｜ - ｜ - ｜
[nginx.ingress.kubernetes.io/enable-owasp-core-rules]	同上	-	-
[nginx.ingress.kubernetes.io/modsecurity-transaction-id]	同上	-	-
[nginx.ingress.kubernetes.io/modsecurity-snippet]	同上	-	-
[nginx.ingress.kubernetes.io/mirror-request-body]	TODO；镜像	-	-
[nginx.ingress.kubernetes.io/mirror-target]	同上	-	-
[nginx.ingress.kubernetes.io/mirror-host]	同上	-	-

命令行参数

名称	说明	默认值	原始HELP
add_dir_header	TODO	TODO	If true, adds the file directory to the header of the log messages
alsologtostderr	错误输出终端，同时写入也文件	-	log to standard error as well as files (no effect when -logtostderr=true)
annotations-prefix	ingress使用注解的标签前缀	nginx.ingress.kubernetes.io	Prefix of the Ingress annotations specific to the NGINX controller. (default “nginx.ingress.kubernetes.io”)
apiserver-host	连接的api server地址	kubernetes.default.svc	Takes the form “protocol://address:port”. If not specified, it is assumed the program runs inside a Kubernetes cluster and local discovery is attempted.
certificate-authority	ca证书，验证api server的https	-	Path to a cert file for the certificate authority. This certificate is used only when the flag –apiserver-host is specified.
configmap	用于控制器全局的ConfigMap配置名	nginx-configuration	Name of the ConfigMap containing custom global configurations for the controller.
controller-class	TODO	k8s.io/ingress-nginx	Ingress Class Controller value this Ingress satisfies. The class of an Ingress object is set using the field IngressClassName in Kubernetes clusters version v1.19.0 or higher. The .spec.controller value of the IngressClass referenced in an Ingress Object should be the same value specified here to make this object be watched.
deep-inspect	TODO	true	Enables ingress object security deep inspector (default true)
default-backend-service	处理查询到的主机，流量转发的后段服务	$(POD_NAMESPACE)/ingress-nginx-defaultbackend	Service used to serve HTTP requests not matching any known server name (catch-all). Takes the form “namespace/name”. The controller configures NGINX to forward requests to the first port of this Service.
default-server-port	TODO	8181	Port to use for exposing the default server (catch-all).
default-ssl-certificate	TODO	-	Secret containing a SSL certificate to be used by the default HTTPS server
disable-catch-all	TODO	-	Disable support for catch-all Ingresses
disable-full-test	TODO；默认开启	-	Disable full test of all merged ingresses at the admission stage and tests the template of the ingress being created or updated
disable-svc-external-name	TODO	-	Disable support for Services of type ExternalName
dynamic-configuration-retries	TODO	15	Number of times to retry failed dynamic configuration before failing to sync an ingress.
election-id	解决多副本同时更新状态，使用的选举id锁	ingress-controller-leader	Election id to use for Ingress status updates.
enable-metrics	开启nginx的prometheus类型性能指标，影响性能，高并发下可以考虑关闭	true	Enables the collection of NGINX metrics
enable-ssl-chain-completion	TODO	-	Autocomplete SSL certificate chains with missing intermediate CA certificates. Certificates uploaded to Kubernetes must have the “Authority Information Access” X.509 v3 extension for this to succeed.
enable-ssl-passthrough	TODO	-	Enable SSL Passthrough.
health-check-path	健康检测的url地址	/healthz	URL path of the health check endpoint. Configured inside the NGINX status server. All requests received on the port defined by the healthz-port parameter are forwarded internally to this path.
health-check-timeout	健康检测超时时间，秒	10	Time limit, in seconds, for a probe to health-check-path to succeed.
healthz-host	健康检测主机地址	-	Address to bind the healthz endpoint.
healthz-port	健康检测端口	10254	Port to use for the healthz endpoint.
http-port	接收http协议流量的端口	80	Port to use for servicing HTTP traffic.
https-port	接收https协议流量的端口	80	Port to use for servicing HTTPS traffic.
ingress-class	已被废弃；TODO	-	Name of the ingress class this controller satisfies. The class of an Ingress object is set using the annotation “kubernetes.io/ingress.class”. The parameter –controller-class has precedence over this.
ingress-class-by-name	TODO	TODO	Define if Ingress Controller should watch for Ingress Class by Name together with Controller Class
internal-logger-address	TODO	127.0.0.1:11514	Address to be used when binding internal syslogger
kubeconfig	连接api server的信息	-	Path to a kubeconfig file containing authorization and API server information.
length-buckets	TODO	-	Set of buckets which will be used for prometheus histogram metrics such as RequestLength, ResponseLength. ([10.000000,20.000000,30.000000,40.000000,50.000000,60.000000,70.000000,80.000000,90.000000,100.000000])
log_backtrace_at	TODO	0	when logging hits line file:N, emit a stack trace
log_dir	日志目录	-	If non-empty, write log files in this directory (no effect when -logtostderr=true)
log_file	日志文件名	-	If non-empty, use this log file (no effect when -logtostderr=true)
log_file_max_size	日志文件最大值，单位：MB	1800MB	Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited.
logtostderr	日志输出到终端	true	log to standard error instead of files
maxmind-edition-ids	TODO	-	Maxmind edition ids to download GeoLite2 Databases. (default “GeoLite2-City,GeoLite2-ASN”)
maxmind-license-key	TODO	-	Maxmind license key to download GeoLite2 Databases.
maxmind-mirror	TODO	-	Maxmind mirror url (example: http://geoip.local/databases
maxmind-retries-count	TODO	1	Number of attempts to download the GeoIP DB.
maxmind-retries-timeout	TODO	0s	Maxmind downloading delay between 1st and 2nd attempt, 0s - do not retry to download if something went wrong.
metrics-per-host	TODO	true	Export metrics per-host
monitor-max-batch-size	TODO	10000	Max batch size of NGINX metrics
one_output	TODO	TODO	If true, only write logs to their native severity level
post-shutdown-grace-period	TODO	10	Seconds to wait after the nginx process has stopped before controller exits.
profiler-port	控制器对外提供的profiler端口地址	10245	Port to use for expose the ingress controller Go profiler when it is enabled.
profiling	是否开启pprof	true	Enable profiling via web interface host:port/debug/pprof/
publish-service	各个 ingress.status 的 Address 地址，监听命名空间下服务，格式为：“namespace/name”，一般仅在公有云环境使用	TODO	Service fronting the Ingress controller. Takes the form “namespace/name”. When used together with update-status, the controller mirrors the address of this service’s endpoints to the load-balancer status of all Ingress objects it satisfies.
publish-status-address	功能同"publish-service"，有人工预先分配	TODO	Customized address (or addresses, separated by comma) to set as the load-balancer status of Ingress objects this controller satisfies. Requires the update-status parameter.
report-node-internal-ip-address	功能同"publish-service"，使用当前 controler 部署节点的IP，一般在私有云使用	TODO	Set the load-balancer status of Ingress objects to internal Node addresses instead of external.
report-status-classes	TODO	TODO	Use status classes (2xx, 3xx, 4xx and 5xx) instead of status codes in metrics
shutdown-grace-period	TODO	TODO	Seconds to wait after receiving the shutdown signal, before stopping the nginx process.
size-buckets	TODO	TODO	Set of buckets which will be used for prometheus histogram metrics such as BytesSent (default [10.000000,100.000000,1000.000000,10000.000000,100000.000000,1000000.000000,10000000.000000])
skip_headers	TODO	TODO	If true, avoid header prefixes in the log messages
skip_log_headers	TODO	TODO	If true, avoid headers when opening log files (no effect when -logtostderr=true)
ssl-passthrough-proxy-port	TODO	442	Port to use internally for SSL Passthrough.
status-port	TODO	10246	Port to use for the lua HTTP endpoint configuration.
status-update-interval	TODO	60	Time interval in seconds in which the status should check if an update is required. Default is 60 seconds
stderrthreshold	TODO	2	logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false)
stream-port	TODO	10247	Port to use for the lua TCP/UDP endpoint configuration.
sync-period	TODO；默认关闭	-	Period at which the controller forces the repopulation of its local object stores.
sync-rate-limit	TODO	0.3	Define the sync frequency upper limit
tcp-services-configmap	nginx stream tcp代理	tcp-services	Name of the ConfigMap containing the definition of the TCP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form “namespace/name:port”, where “port” can either be a port number or name.
time-buckets	TODO	TODO	Set of buckets which will be used for prometheus histogram metrics such as RequestTime, ResponseTime (default [0.005000,0.010000,0.025000,0.050000,0.100000,0.250000,0.500000,1.000000,2.500000,5.000000,10.000000])
udp-services-configmap	nginx stream udp代理	udp-services	Name of the ConfigMap containing the definition of the UDP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form “namespace/name:port”, where “port” can either be a port name or number.
update-status	TODO	true	Update the load-balancer status of Ingress objects this controller satisfies. Requires setting the publish-service parameter to a valid Service reference.
update-status-on-shutdown	TODO	true	Update the load-balancer status of Ingress objects when the controller shuts down. Requires the update-status parameter.
v	日志级别	-	number for the log level verbosity
validating-webhook	准入控制器验证ingress的配置	TODO	The address to start an admission controller on to validate incoming ingresses. Takes the form “:port”. If not provided, no admission controller is started.
validating-webhook-certificate	准入控制器的证书	TODO	The path of the validating webhook certificate PEM.
validating-webhook-key	准入控制器的证书	TODO	The path of the validating webhook key PEM.
vmodule	TODO	TODO	comma-separated list of pattern=N settings for file-filtered logging
watch-ingress-without-class	TODO	TODO	Define if Ingress Controller should also watch for Ingresses without an IngressClass or the annotation specified
watch-namespace	TODO	TODO	Namespace the controller watches for updates to Kubernetes objects. This includes Ingresses, Services and all configuration resources. All namespaces are watched if this parameter is left empty.
watch-namespace-selector	TODO	TODO	Selector selects namespaces the controller watches for updates to Kubernetes objects.

ConfigMap 配置参数

可以通过这个定制 nginx.conf 文件内容，里面填写的键值对对照数据结构 config.go 配置，由于 “ConfigMap” 里面只能是 string 类型，所以任何值需使用引号包含起来，如："128"。如果是数组类型，则通过逗号分隔，如："192.168.30.0/24, 192.168.31.0/24"

kind: ConfigMap
apiVersion: v1
metadata:
 name: nginx-configuration
data:
 access-log-path: "/var/log/nginx/access.log"
 error-log-path: "/var/log/nginx/error.log"

名称	默认值	可取值	说明
allow-backend-server-header	“true”	“true” or “false”	是否以后端返回的 “Server” 响应头来代替nginx ingress的值
allow-snippet-annotations	“true”	“false” or “true”	开启ingress配置 *-snippet 更改 nginx.conf 内容，如果用户不可信则不开放
annotation-value-word-blocklist	""	字符串数组，见建议值	定义哪些关键字不能提供给用户配置 annotation 使用，有安全风险
hide-headers	""	字符串数组	定义哪些请求头不返回给客户端
access-log-params	""	access_log	nginx access_log 额外的一些参数
access-log-path	“/var/log/nginx/access.log” 指向 “/dev/stdout”	http 与 stream 共同使用	客户端访问日志
http-access-log-path	""	http 访问日志，优先级高	http 访问日志
stream-access-log-path	""	stream 访问日志，优先级高	http 访问日志
enable-access-log-for-default-backend	“false”	“true” or “false”	是否开启未匹配后端ingress的请求日志记录
error-log-path	“/var/log/nginx/error.log” 指向 “/dev/stderr”	""	nginx 错误日志
enable-modsecurity	“false”	“true” or “false”	是否开启 modsecurity 功能
enable-owasp-modsecurity-crs	“false”	“true” or “false”	同上
modsecurity-snippet	""	""	TODO; 同上
client-header-buffer-size	“1k”	""	客户端请求header缓存大小
client-header-timeout	“60”	""	-
client-body-buffer-size	“8k”	""	-
client-body-timeout	“60”	""	-
disable-access-log	“false”	“true” “false”	是否关闭请求日志
disable-ipv6	“false”	“true” or “false”	是否关闭ipv6监听
disable-ipv6-dns	“false”	“true” or “false”	关闭nginx对ipv6的解析
enable-underscores-in-headers	“false”	“true” or “false”	客户端提交下划线"_“的请求头是否忽略，不做传递至后端
enable-ocsp	“false”	“true” or “false”	TODO
ignore-invalid-headers	“true”	“true” or “false”	忽略异常请求头
retry-non-idempotent	“false”	“true” or “false”	对非幂等的请求（POST、LOCK、PATCH），如果后端错误将不进行重试
error-log-level	“notice”	见nginx error_log	错误日志级别
hsts	“true”	“true” or “false”	告诉浏览器必须使用https进行访问
hsts-include-subdomains	“true”	“true” or “false”	TODO
hsts-max-age	-	-	-
hsts-preload	“false”	-	-
keep-alive	“75”	"”	客户端与服务端保持连接时长，避免频繁建立tcp，设置为0则关闭
keep-alive-requests	“100”	""	TODO
large-client-header-buffers	“4 8k”	""
log-format-escape-json	“false”	“true” or “false”	如果日志以json格式输出，则对一些格式进行转换
log-format-upstream	见日志格式默认值	""	客户端请求日志格式
log-format-stream	见stream日志格式默认值	""	客户端请求日志格式
enable-multi-accept	“treu”	“true” or “false”	TODO
max-worker-connections	“16384”	""	TODO
max-worker-open-files	“0”	""	TODO
map-hash-bucket-size	“64”	""	TODO
proxy-real-ip-cidr	“0.0.0.0/0”	字符串数组	配置proxy中间层地址并过滤, 用于获取真实客户端IP
proxy-set-headers	""	""	自定义传输后段的请求头
server-name-hash-max-size	“1024”	""	TODO
server-name-hash-bucket-size	""	""	TODO
proxy-headers-hash-max-size	“512”	""	TODO
proxy-headers-hash-bucket-size	“64”	""	TODO
reuse-port	“true”	“true” or “false”	TODO
plugins	""	""	TODO
server-tokens	“false”	“true” or “false”	是否返回客户端nginx的版本
use-proxy-protocol	“false”	“true” or “false”	是否使用 PROXY 协议，可以获取客户端IP
proxy-protocol-header-timeout	“5s”	""	TODO
use-gzip	“false”	“true” or “false”	针对 HTTP 响应是否开启 gzip 压缩
gzip-level	“1”	""	gzip 压缩级别
use-geoip	“true”	“true” or “false”	开启 geoip
use-geoip2	“false”	“true” or “false”	开启 geoip
enable-brotli	“false”	“true” or “false”	使用 brotli 模块压缩
user-http2	“false”	“true” or “false”	开启 http2 支持
gzip-min-length	“256”	""	TODO
gzip-types	见gzip默认压缩类型	""	需要压缩的类型
worker-processes	“auto”	""	开启nginx的进程数，默认等于cpu核心数
load-balance	“round_robin”	“round_robin” or “ewma”	TODO;
upstream-keepalive-connections	-	-	-
proxy-read-timeout	60s	x	proxy读取等待后端响应体返回的最大时间
proxy-next-upstream-tries	3	x	proxy读取、发送后端如果超时，重试的次数
compute-full-forwarded-for	false	“true” or “false”	将 remote address 附加到 X-Forwarded-For,新增X-Original-Forwarded-For 记录旧值，一般用于解决 xff 伪造

ingress关闭建议值

"load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount,{,},',\""

日志格式默认值

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id

stream日志格式默认值

[$remote_addr] [$time_local] $protocol $status $bytes_sent $bytes_received $session_time

gzip默认压缩类型

application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component

各个配置例子

有三种方式可以自定义你的Ingress Nginx：

通过ConfigMap

更改全局的ConfigMap配置"nginx-configuration"，默认由参数"–configmap"控制。
通过Annotations

通过各个ingress的配置文件添加特定注解，控制各自的行为，注解前缀由"–annotations-prefix"控制。
自定义 nginx.tmpl 模版

如果以上两种方式还无法满足需要，可以通过更改nginx.conf的模版文件。

自定义Nginx模版

拷贝ingress-nginx镜像内的模版内容

/etc/nginx/template/nginx.tmpl

以nginx.tmpl创建ConfigMap资源

apiVersion: v1
kind: ConfigMap
metadata:
 name: nginx-template
 namespace: kube-system
data:
 nginx.tmpl: |
 {{ $all := . }}
 {{ $servers := .Servers }}
 {{ $cfg := .Cfg }}
 {{ $IsIPV6Enabled := .IsIPV6Enabled }}
 {{ $healthzURI := .HealthzURI }}
 {{ $backends := .Backends }}
 {{ $proxyHeaders := .ProxySetHeaders }}
 {{ $addHeaders := .AddHeaders }}

 # Configuration checksum: {{ $all.Cfg.Checksum }}
 ......

在Deployment或DaemonSet中加入模版

把官方镜像里的nginx.tmpl模版文件拷贝更改，并创建为ConfigMap资源挂载覆盖默认模版。

 volumeMounts:
 - name: nginx-template
 mountPath: /etc/nginx/template
 readOnly: true
......
 volumes:
 - name: nginx-template
 configMap:
 name: nginx-template
......

代理TCP与UDP流量

ingres是不包含tcp、udp这块流量规范，这里是ingress-nginx添加的功能。

创建tcp服务使用的ConfigMap

监听的cm名称由--tcp-services-configmap参数决定

kind: ConfigMap
apiVersion: v1
metadata:
 name: tcp-services
data:
 "1053": "kube-system/kube-dns:53"

开启1053端口，转发流量至"kube-system"命名空间下service为"kube-dns"的53端口。

在ingress-nginx自动生成的配置

/etc/nginx/nginx.conf

stream {
 ......
 server {
 preread_by_lua_block {
 ngx.var.proxy_upstream_name="tcp-kube-system-kube-dns-53";
 }

 listen 1053;

 proxy_timeout 600s;
 proxy_next_upstream on;
 proxy_next_upstream_timeout 600s;
 proxy_next_upstream_tries 3;

 proxy_pass upstream_balancer;
 }
 ......
}

app-root

ingress配置

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/app-root: "/test"

nginx.tmpl模版

{{ if not (empty $location.Rewrite.AppRoot) }}
if ($uri = /) {
 return 302 $scheme://$http_host{{ $location.Rewrite.AppRoot }};
}
{{ end }}

nginx.conf生成内容

server {
 if ($uri = /) {
 return 302 $scheme://$http_host/test;
 }
}

affinity

ingress配置

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/affinity: "cookie"
 nginx.ingress.kubernetes.io/session-cookie-name: "HELLO"

curl测试

< HTTP/1.1 200 OK
< Date: Mon, 12 Sep 2022 14:05:36 GMT
< Content-Type: text/html
< Content-Length: 4
< Connection: keep-alive
< Set-Cookie: HELLO=1662991537.226.193.550764|a6b1ec86db225f232b62b29085af45ff; Path=/; HttpOnly
< Last-Modified: Mon, 11 Dec 2017 03:18:39 GMT

具体逻辑由"rootfs/etc/nginx/lua/balancer/sticky.lua"实现。

auth

创建用户与密码文件

htpasswd -c http-auth.txt user1

# 用户: user1
# 密码: kubernetes
user1:$apr1$ue3Fu891$xnyqlJlwcM4iWtJrAN8ZW/

把 http-auth.txt 转化为 k8s secret 资源

cat http-auth.txt

当"nginx.ingress.kubernetes.io/auth-secret-type"为"auth-file"时，替换"auth"对应的值

apiVersion: v1
kind: Secret
metadata:
 name: basic-auth
stringData:
 auth: dXNlcjE6JGFwcjEkdWUzRnU4OTEkeG55cWxKbHdjTTRpV3RKckFOOFpXLwo=
type: Opaque

当"nginx.ingress.kubernetes.io/auth-secret-type"为"auth-map"时，替换下面"user1"行

apiVersion: v1
kind: Secret
metadata:
 name: basic-auth
stringData:
 user1: $apr1$ue3Fu891$xnyqlJlwcM4iWtJrAN8ZW/
type: Opaque

这里注意"secret"中的内容格式由"nginx.ingress.kubernetes.io/auth-secret-type"确定"。

canary

基于 HTTP 请求头

当仅设置 nginx.ingress.kubernetes.io/canary-by-header 请求头时，对应的值可取 “always” 或 “never” 分别表示流量至该副本或关闭。

如果需要自定义默认的 “always” 与 “never” 值，需在定义 nginx.ingress.kubernetes.io/canary-by-header-value 内容，然后请求匹配值才会调度。

日志格式

json日志

log-format-escape-json: "true"
log-format-upstream: '{"msec": "$msec", "remote_addr": "$remote_addr", "request_length": $request_length, "request_method": "$request_method", "host": "$host", "request_uri": "$request_uri", "server_name": "$server_name", "server_addr": "$server_addr", "server_port": $server_port, "uri": "$uri", "args": "$args", "server_protocol": "$server_protocol", "bytes_sent": $bytes_sent, "body_bytes_sent": $body_bytes_sent, "status": $status, "request_time": $request_time, "http_referer": "$http_referer", "http_user_agent": "$http_user_agent", "http_x_tr_user_id": "$http_x_tr_user_id", "http_x_tr_pv_id": "$http_x_tr_pv_id", "http_x_tr_request_id": "$req_id", "http_x_tr_rpc_id": "$http_x_tr_rpc_id", "proxy_upstream_name": "$proxy_upstream_name", "upstream_addr": "$upstream_addr", "upstream_response_length": "$upstream_response_length", "upstream_response_time": "$upstream_response_time", "upstream_status": "$upstream_status"}'

客户端通过snippet自定义nginx.conf配置

客户端是指通过在ingress中编写annotations可以很灵活的自定义nginx.conf配置，如果是不可信用户则相当危险。

功能开启或关闭是在 “nginx-configuration” 的configmap配置，“true"是开启状态：

kind: ConfigMap
apiVersion: v1
metadata:
 name: nginx-configuration
data:
 allow-snippet-annotations: "true"

可以通过以下几种注解分别自定义不同的配置：

nginx.ingress.kubernetes.io/configuration-snippet
nginx.ingress.kubernetes.io/server-snippet
nginx.ingress.kubernetes.io/stream-snippet
nginx.ingress.kubernetes.io/modsecurity-snippet

在各 “server” 下 “location” 自定义配置

ingress注解

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/configuration-snippet: |
 more_set_headers "x-tr-request-id: $req_id";

nginx.tmpl模版内容

http {
 ## start server {{ $server.Hostname }}
 server {
 ......
 location {{ $path }} {

 {{/* Add any additional configuration defined */}}
 {{ $location.ConfigurationSnippet }}

 }
 ......
 }
 ## end server {{ $server.Hostname }}
}

生成的内容

http {
 server {
 ......
 location / {

 more_set_headers "Request-Id: $req_id";

 }
 ......
 }
}

在各 “server” 下添加自定义配置

ingress注解

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/server-snippet: |
 set $agentflag 0;

 if ($http_user_agent ~* "(Mobile)" ){
 set $agentflag 1;
 }

 if ( $agentflag = 1 ) {
 return 301 https://m.a-demo-web.site;
 }

nginx.tmpl模版内容

http {
 ## start server {{ $server.Hostname }}
 server {
 ......

 {{ if not (empty $cfg.ServerSnippet) }}
 # Custom code snippet configured in the configuration configmap
 {{ $cfg.ServerSnippet }}
 {{ end }}

 ......
 }
 ## end server {{ $server.Hostname }}
}

生成的nginx.conf

http {
 server {

 # Custom code snippet configured for host api.a-demo-web.site
 set $agentflag 0;

 if ($http_user_agent ~* "(Mobile)" ){
 set $agentflag 1;
 }

 if ( $agentflag = 1 ) {
 return 301 https://m.a-demo-web.site;
 }

 }
}

在各 “stream” 下添加自定义配置

ingress注解

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/stream-snippet: |
 server {
 listen 8000;
 proxy_pass 127.0.0.1:80;
 }

nginx.tmpl模版内容

stream {
 # TCP services
 {{ range $tcpServer := .TCPBackends }}
 server {
 ....
 }
 {{ end }}

 # UDP services
 {{ range $udpServer := .UDPBackends }}
 server {
 ....
 }
 {{ end }}

 # Stream Snippets
 {{ range $snippet := .StreamSnippets }}
 {{ $snippet }}
 {{ end }}
}

在开启 “auth-url” 下添加自定义配置

ingress注解

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/auth-url: http://a-demo-web.site/external-auth
 nginx.ingress.kubernetes.io/auth-snippet: |
 proxy_set_header Foo-Header 42;

nginx.tmpl模版内容

http {
 ## start server {{ $server.Hostname }}
 server {

 {{ if $authPath }}
 location = {{ $authPath }} {
 internal;

 {{ if not (empty $externalAuth.AuthSnippet) }}
 {{ $externalAuth.AuthSnippet }}
 {{ end }}

 }

 }
 ## end server {{ $server.Hostname }}
}

在开启 “modsecurity” 下添加自定义配置

ingress注解

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/enable-modsecurity: "true"
 nginx.ingress.kubernetes.io/modsecurity-snippet: |
 SecRuleEngine On
 SecDebugLog /tmp/modsec_debug.log

nginx.tmpl模版内容

http {
 ## start server {{ $server.Hostname }}
 server {
 ......
 location {{ $path }} {

 {{ buildModSecurityForLocation $all.Cfg $location }}

 }
 ......
 }
 ## end server {{ $server.Hostname }}
}

# 各各站点的配置生成
# internal/ingress/controller/template/template.go

func buildModSecurityForLocation(cfg config.Configuration, location *ingress.Location) string {
 isMSEnabledInLoc := location.ModSecurity.Enable
 isMSEnableSetInLoc := location.ModSecurity.EnableSet
 isMSEnabled := cfg.EnableModsecurity

 ......

 if location.ModSecurity.Snippet != "" {
 buffer.WriteString(fmt.Sprintf(`modsecurity_rules '
%v
';
`, location.ModSecurity.Snippet))
 }

 return buffer.String()
}

生成的nginx.conf

http {
 server {
 ......
 set $proxy_alternative_upstream_name "";

 modsecurity on;
 modsecurity_rules '
 SecRuleEngine On
 SecDebugLog /tmp/modsec_debug.log

 ';

 client_max_body_size 20m;
 ......
 }
}

服务端通过snippet自定义nginx.conf配置

服务端是通过"ingress-controller"启动参数--configmap=$(POD_NAMESPACE)/nginx-configuration所关联的"ConfigMap"资源来实现。

可以通过以下几种注解分别自定义不同的配置：

main-snippet
http-snippet
server-snippet
stream-snippet
location-snippet

......
daemon off;

# main-snippet
{{ if not (empty $cfg.MainSnippet) }}
{{ $cfg.MainSnippet }}
{{ end }}

events {
}

http {

 # http-snippet
 {{ if not (empty $cfg.HTTPSnippet) }}
 # Custom code snippet configured in the configuration configmap
 {{ $cfg.HTTPSnippet }}
 {{ end }}

 ## start server {{ $server.Hostname }}
 server {
 ......
 # 该模版内同样包含一个"server snippet"，由客户端通过注解提交
 # 以下属于 {{ template "SERVER" serverConfig $all $server }} 内容

 {{ if not (empty $server.ServerSnippet) }}
 # Custom code snippet configured for host {{ $server.Hostname }}
 {{ $server.ServerSnippet }}
 {{ end }}

 location {{ $path }} {

 # location-snippet
 {{ if not (empty $all.Cfg.LocationSnippet) }}
 # Custom code snippet configured in the configuration configmap
 {{ $all.Cfg.LocationSnippet }}
 {{ end }}
 }
 # 以上属于 {{ template "SERVER" serverConfig $all $server }} 内容

 # server-snippet
 {{ if not (empty $cfg.ServerSnippet) }}
 # Custom code snippet configured in the configuration configmap
 {{ $cfg.ServerSnippet }}
 {{ end }}

 {{ template "CUSTOM_ERRORS" (buildCustomErrorDeps "upstream-default-backend" $cfg.CustomHTTPErrors $all.EnableMetrics $cfg.EnableModsecurity) }}
 }
 ## end server {{ $server.Hostname }}
}

stream {
 ......

 # stream-snippet
 # 客户端通过ingress提交注解也在次
 # Stream Snippets
 {{ range $snippet := .StreamSnippets }}
 {{ $snippet }}
 {{ end }}
}
......

监听的几个端口

启动监听的端口

端口	协议	参数	用途描述
80	http	http-port	xxx
443	https	https-port	xx
8181	http	default-server-port	默认服务
442	https	ssl-passthrough-proxy-port	xxx
10254	http	healthz-port	/metrics, /healthz
10245	http	profiler-port	仅可在 127.0.0.1 监听； /debug/pprof/
10246	http	status-port	仅可在 127.0.0.1 监听；/healthz, /is-dynamic-lb-initialized, /nginx_status, /configuration
10247	tcp	stream-port	仅可在 127.0.0.1 监听；用于更新 tcp-services 与 udp-services 后端

rp := []int{
n.cfg.ListenPorts.HTTP,
n.cfg.ListenPorts.HTTPS,
n.cfg.ListenPorts.SSLProxy,
n.cfg.ListenPorts.Health,
n.cfg.ListenPorts.Default,
nginx.ProfilerPort,
nginx.StatusPort,
nginx.StreamPort,
}

default-server-port

在nginx.conf模版中实现

# backend for when default-backend-service is not configured or it does not have endpoints
server {
 listen 8181 default_server reuseport backlog=4096;

 set $proxy_upstream_name "internal";

 access_log off;

 location / {
 return 404;
 }
}

profiler-port

在go代码中实现

if conf.EnableProfiling {
 go metrics.RegisterProfiler("127.0.0.1", nginx.ProfilerPort)
}

status-port

在nginx.conf模版中实现

# default server, used for NGINX healthcheck and access to nginx stats
server {
 listen 127.0.0.1:{{ .StatusPort }};
 set $proxy_upstream_name "internal";

 keepalive_timeout 0;
 gzip off;

 access_log off;

 location {{ $healthzURI }} {
 return 200;
 }

 location /is-dynamic-lb-initialized {
 ......
 }

 location {{ .StatusPath }} {
 stub_status on;
 }

 location /configuration {
 ......
 }

 location / {
 content_by_lua_block {
 ngx.exit(ngx.HTTP_NOT_FOUND)
 }
 }
}

stream-port

在go代码中实现

streams := make([]ingress.Backend, 0)

for _, ep := range TCPEndpoints {
 var service *apiv1.Service
 if ep.Service != nil {
 service = &apiv1.Service{Spec: ep.Service.Spec}
 }

 key := fmt.Sprintf("tcp-%v-%v-%v", ep.Backend.Namespace, ep.Backend.Name, ep.Backend.Port.String())
 streams = append(streams, ingress.Backend{
 Name: key,
 Endpoints: ep.Endpoints,
 Port: intstr.FromInt(ep.Port),
 Service: service,
 })
}

for _, ep := range UDPEndpoints {
 var service *apiv1.Service
 if ep.Service != nil {
 service = &apiv1.Service{Spec: ep.Service.Spec}
 }

 key := fmt.Sprintf("udp-%v-%v-%v", ep.Backend.Namespace, ep.Backend.Name, ep.Backend.Port.String())
 streams = append(streams, ingress.Backend{
 Name: key,
 Endpoints: ep.Endpoints,
 Port: intstr.FromInt(ep.Port),
 Service: service,
 })
}

buf, err := json.Marshal(streams)

hostPort := net.JoinHostPort("127.0.0.1", fmt.Sprintf("%v", nginx.StreamPort))
conn, err := net.Dial("tcp", hostPort)

_, err = conn.Write(buf)

配置404服务后端

如果请求域名未在集群中配置，则把流量返回至指定后端服务，比如官方默认的服务：

建议不要开启，当该后端服务不可用时，客户端需等待超时时间后才关闭，并以504返回；客户等待的时间为：proxy-read-timeout * proxy-next-upstream-tries

kube-system/ingress-nginx-defaultbackend

镜像与代码

默认镜像

gcr.io/google-containers/defaultbackend:1.4

源代码

https://github.com/kubernetes/ingress-gce/tree/master/cmd/404-server

default backend 原始代码仓库为 ingress-nginx 之后被移到 ingress-gce 下面，参考

controller 开启

在 /nginx-ingress-controller 添加 --default-backend-service 参数指定 namespace 下的 service

args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/ingress-nginx-defaultbackend

响应示例：未配置任何404服务

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

响应示例：转发默认404服务

default backend - 404

响应示例：已配置404服务，但后端不可用

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>

需等待服务端配置的超时时间结束。

POD内核优化

未调整前

/etc/nginx $ sysctl -a | grep kernel.core_uses_pid
kernel.core_uses_pid = 0
/etc/nginx $
/etc/nginx $ sysctl -a | grep net.core.somaxconn
net.core.somaxconn = 4096
/etc/nginx $
/etc/nginx $ sysctl -a | grep net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768 60999
/etc/nginx $ ^C

URL重定向

需求

针对请求地址 api.a-demo-web.site/commons/cmdb/v1/host/search 到后端需变更为 /host/search，也就是移除前缀路径 /commons/cmdb/v1/。

实现

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 annotations:
 nginx.ingress.kubernetes.io/rewrite-target: "/$1"
 name: commons-api
spec:
 ingressClassName: nginx
 rules:
 - host: api.a-demo-web.site
 http:
 paths:
 - path: /commons/cmdb/v1/(.*)
 backend:
 serviceName: commons-cmdb-v1
 servicePort: 10080
 pathType: Prefix

主要添加 “rewrite-target” 注解，与 path 设置为正则，这里需注意下 ingress 的 “apiVersion”，不同版本间风格存在差异。

防止 xff 伪造

功能需求

客户端使用 “X-Forwarded-For” 自定了一些 ip 并提交了请求，此时服务端正常情况下会记录整个 ip 列表。

curl -v -H "X-Forwarded-For: 1.1.1.1, 2.2.2.2" https://www.baidu.com

服务端将会记录：“1.1.1.1, 2.2.2.2, 代理IP1, 代理IP2, ……"，此时希望能排除客户端自定的ip，期望为：“代理IP1, 代理IP2, ……”

步骤实现

forwarded-for-header: "X-Forwarded-For"
use-forwarded-headers: "true"
compute-full-forwarded-for: "true"
proxy-real-ip-cidr: ""

配置 “compute-full-forwarded-for” 值为 “true”，则在 ingress-nginx 这里会把当前 “X-Forwarded-For” 请求头的值拷贝并新增至 “X-Original-Forwarded-For”，同时重制 “X-Forwarded-For” 并把此时 nginx 认为的 remote addr 写入覆盖该头。

原理分析

http {

......

 {{ if and $cfg.UseForwardedHeaders $cfg.ComputeFullForwardedFor }}
 # We can't use $proxy_add_x_forwarded_for because the realip module
 # replaces the remote_addr too soon
 map $http_x_forwarded_for $full_x_forwarded_for {
 {{ if $all.Cfg.UseProxyProtocol }}
 default "$http_x_forwarded_for, $proxy_protocol_addr";
 '' "$proxy_protocol_addr";
 {{ else }}
 default "$http_x_forwarded_for, $realip_remote_addr";
 '' "$realip_remote_addr";
 {{ end}}
 }

......

}

{{/* definition of server-template to avoid repetitions with server-alias */}}
{{ define "SERVER" }}

......

 {{ if $authPath }}
 location = {{ $authPath }} {

 {{ if and $all.Cfg.UseForwardedHeaders $all.Cfg.ComputeFullForwardedFor }}
 proxy_set_header X-Forwarded-For $full_x_forwarded_for;
 {{ else }}
 proxy_set_header X-Forwarded-For $remote_addr;
 {{ end }}

 }
 {{ end }}

 location {{ $path }} {

 # 如果同时设置了 "use-forwarded-headers" 与 "compute-full-forwarded-for" 则不进行 xff 的替换。
 {{ if and $all.Cfg.UseForwardedHeaders $all.Cfg.ComputeFullForwardedFor }}
 {{ $proxySetHeader }} X-Forwarded-For $full_x_forwarded_for;
 {{ else }}
 {{ $proxySetHeader }} X-Forwarded-For $remote_addr;
 {{ end }}


 }

......

{{ end }}

Docs: Nginx Template

Thu, 22 Sep 2022 11:08:05 +0800

Ingress Nginx 模版内容

基于 controller-v1.3.1 版本

模版内容

{{ $all := . }}
{{ $servers := .Servers }}
{{ $cfg := .Cfg }}
{{ $IsIPV6Enabled := .IsIPV6Enabled }}
{{ $healthzURI := .HealthzURI }}
{{ $backends := .Backends }}
{{ $proxyHeaders := .ProxySetHeaders }}
{{ $addHeaders := .AddHeaders }}

# Configuration checksum: {{ $all.Cfg.Checksum }}

# setup custom paths that do not require root access
pid {{ .PID }};

{{ if $cfg.UseGeoIP2 }}
load_module /etc/nginx/modules/ngx_http_geoip2_module.so;
{{ end }}

{{ if $cfg.EnableBrotli }}
load_module /etc/nginx/modules/ngx_http_brotli_filter_module.so;
load_module /etc/nginx/modules/ngx_http_brotli_static_module.so;
{{ end }}

{{ if (shouldLoadInfluxDBModule $servers) }}
load_module /etc/nginx/modules/ngx_http_influxdb_module.so;
{{ end }}

{{ if (shouldLoadAuthDigestModule $servers) }}
load_module /etc/nginx/modules/ngx_http_auth_digest_module.so;
{{ end }}

{{ if (shouldLoadModSecurityModule $cfg $servers) }}
load_module /etc/nginx/modules/ngx_http_modsecurity_module.so;
{{ end }}

{{ if (shouldLoadOpentracingModule $cfg $servers) }}
load_module /etc/nginx/modules/ngx_http_opentracing_module.so;
{{ end }}

daemon off;

worker_processes {{ $cfg.WorkerProcesses }};
{{ if gt (len $cfg.WorkerCPUAffinity) 0 }}
worker_cpu_affinity {{ $cfg.WorkerCPUAffinity }};
{{ end }}

worker_rlimit_nofile {{ $cfg.MaxWorkerOpenFiles }};

{{/* http://nginx.org/en/docs/ngx_core_module.html#worker_shutdown_timeout */}}
{{/* avoid waiting too long during a reload */}}
worker_shutdown_timeout {{ $cfg.WorkerShutdownTimeout }} ;

{{ if not (empty $cfg.MainSnippet) }}
{{ $cfg.MainSnippet }}
{{ end }}

events {
 multi_accept {{ if $cfg.EnableMultiAccept }}on{{ else }}off{{ end }};
 worker_connections {{ $cfg.MaxWorkerConnections }};
 use epoll;
 {{ range $index , $v := $cfg.DebugConnections }}
 debug_connection {{ $v }};
 {{ end }}
}

http {
 lua_package_path "/etc/nginx/lua/?.lua;;";

 {{ buildLuaSharedDictionaries $cfg $servers }}

 init_by_lua_block {
 collectgarbage("collect")

 -- init modules
 local ok, res

 ok, res = pcall(require, "lua_ingress")
 if not ok then
 error("require failed: " .. tostring(res))
 else
 lua_ingress = res
 lua_ingress.set_config({{ configForLua $all }})
 end

 ok, res = pcall(require, "configuration")
 if not ok then
 error("require failed: " .. tostring(res))
 else
 configuration = res
 configuration.prohibited_localhost_port = '{{ .StatusPort }}'
 end

 ok, res = pcall(require, "balancer")
 if not ok then
 error("require failed: " .. tostring(res))
 else
 balancer = res
 end

 {{ if $all.EnableMetrics }}
 ok, res = pcall(require, "monitor")
 if not ok then
 error("require failed: " .. tostring(res))
 else
 monitor = res
 end
 {{ end }}

 ok, res = pcall(require, "certificate")
 if not ok then
 error("require failed: " .. tostring(res))
 else
 certificate = res
 certificate.is_ocsp_stapling_enabled = {{ $cfg.EnableOCSP }}
 end

 ok, res = pcall(require, "plugins")
 if not ok then
 error("require failed: " .. tostring(res))
 else
 plugins = res
 end
 -- load all plugins that'll be used here
 plugins.init({ {{ range $idx, $plugin := $cfg.Plugins }}{{ if $idx }},{{ end }}{{ $plugin | quote }}{{ end }} })
 }

 init_worker_by_lua_block {
 lua_ingress.init_worker()
 balancer.init_worker()
 {{ if $all.EnableMetrics }}
 monitor.init_worker({{ $all.MonitorMaxBatchSize }})
 {{ end }}

 plugins.run()
 }

 {{/* Enable the real_ip module only if we use either X-Forwarded headers or Proxy Protocol. */}}
 {{/* we use the value of the real IP for the geo_ip module */}}
 {{ if or (or $cfg.UseForwardedHeaders $cfg.UseProxyProtocol) $cfg.EnableRealIp }}
 {{ if $cfg.UseProxyProtocol }}
 real_ip_header proxy_protocol;
 {{ else }}
 real_ip_header {{ $cfg.ForwardedForHeader }};
 {{ end }}

 real_ip_recursive on;
 {{ range $trusted_ip := $cfg.ProxyRealIPCIDR }}
 set_real_ip_from {{ $trusted_ip }};
 {{ end }}
 {{ end }}

 {{ if $all.Cfg.EnableModsecurity }}
 modsecurity on;

 {{ if (not (empty $all.Cfg.ModsecuritySnippet)) }}
 modsecurity_rules '
 {{ $all.Cfg.ModsecuritySnippet }}
 ';
 {{ else }}
 modsecurity_rules_file /etc/nginx/modsecurity/modsecurity.conf;
 {{ end }}

 {{ if $all.Cfg.EnableOWASPCoreRules }}
 modsecurity_rules_file /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf;
 {{ end }}

 {{ end }}

 {{ if $cfg.UseGeoIP }}
 {{/* databases used to determine the country depending on the client IP address */}}
 {{/* http://nginx.org/en/docs/http/ngx_http_geoip_module.html */}}
 {{/* this is require to calculate traffic for individual country using GeoIP in the status page */}}
 geoip_country /etc/nginx/geoip/GeoIP.dat;
 geoip_city /etc/nginx/geoip/GeoLiteCity.dat;
 geoip_org /etc/nginx/geoip/GeoIPASNum.dat;
 geoip_proxy_recursive on;
 {{ end }}

 {{ if $cfg.UseGeoIP2 }}
 # https://github.com/leev/ngx_http_geoip2_module#example-usage

 {{ range $index, $file := $all.MaxmindEditionFiles }}
 {{ if eq $file "GeoLite2-Country.mmdb" }}
 geoip2 /etc/nginx/geoip/GeoLite2-Country.mmdb {
 $geoip2_country_code source=$remote_addr country iso_code;
 $geoip2_country_name source=$remote_addr country names en;
 $geoip2_continent_code source=$remote_addr continent code;
 $geoip2_continent_name source=$remote_addr continent names en;
 }
 {{ end }}

 {{ if eq $file "GeoIP2-Country.mmdb" }}
 geoip2 /etc/nginx/geoip/GeoIP2-Country.mmdb {
 $geoip2_country_code source=$remote_addr country iso_code;
 $geoip2_country_name source=$remote_addr country names en;
 $geoip2_continent_code source=$remote_addr continent code;
 $geoip2_continent_name source=$remote_addr continent names en;
 }
 {{ end }}

 {{ if eq $file "GeoLite2-City.mmdb" }}
 geoip2 /etc/nginx/geoip/GeoLite2-City.mmdb {
 $geoip2_city_country_code source=$remote_addr country iso_code;
 $geoip2_city_country_name source=$remote_addr country names en;
 $geoip2_city source=$remote_addr city names en;
 $geoip2_postal_code source=$remote_addr postal code;
 $geoip2_dma_code source=$remote_addr location metro_code;
 $geoip2_latitude source=$remote_addr location latitude;
 $geoip2_longitude source=$remote_addr location longitude;
 $geoip2_time_zone source=$remote_addr location time_zone;
 $geoip2_region_code source=$remote_addr subdivisions 0 iso_code;
 $geoip2_region_name source=$remote_addr subdivisions 0 names en;
 $geoip2_subregion_code source=$remote_addr subdivisions 1 iso_code;
 $geoip2_subregion_name source=$remote_addr subdivisions 1 names en;
 }
 {{ end }}

 {{ if eq $file "GeoIP2-City.mmdb" }}
 geoip2 /etc/nginx/geoip/GeoIP2-City.mmdb {
 $geoip2_city_country_code source=$remote_addr country iso_code;
 $geoip2_city_country_name source=$remote_addr country names en;
 $geoip2_city source=$remote_addr city names en;
 $geoip2_postal_code source=$remote_addr postal code;
 $geoip2_dma_code source=$remote_addr location metro_code;
 $geoip2_latitude source=$remote_addr location latitude;
 $geoip2_longitude source=$remote_addr location longitude;
 $geoip2_time_zone source=$remote_addr location time_zone;
 $geoip2_region_code source=$remote_addr subdivisions 0 iso_code;
 $geoip2_region_name source=$remote_addr subdivisions 0 names en;
 $geoip2_subregion_code source=$remote_addr subdivisions 1 iso_code;
 $geoip2_subregion_name source=$remote_addr subdivisions 1 names en;
 }
 {{ end }}

 {{ if eq $file "GeoLite2-ASN.mmdb" }}
 geoip2 /etc/nginx/geoip/GeoLite2-ASN.mmdb {
 $geoip2_asn source=$remote_addr autonomous_system_number;
 $geoip2_org source=$remote_addr autonomous_system_organization;
 }
 {{ end }}

 {{ if eq $file "GeoIP2-ASN.mmdb" }}
 geoip2 /etc/nginx/geoip/GeoIP2-ASN.mmdb {
 $geoip2_asn source=$remote_addr autonomous_system_number;
 $geoip2_org source=$remote_addr autonomous_system_organization;
 }
 {{ end }}

 {{ if eq $file "GeoIP2-ISP.mmdb" }}
 geoip2 /etc/nginx/geoip/GeoIP2-ISP.mmdb {
 $geoip2_isp source=$remote_addr isp;
 $geoip2_isp_org source=$remote_addr organization;
 $geoip2_asn source=$remote_addr default=0 autonomous_system_number;
 }
 {{ end }}

 {{ if eq $file "GeoIP2-Connection-Type.mmdb" }}
 geoip2 /etc/nginx/geoip/GeoIP2-Connection-Type.mmdb {
 $geoip2_connection_type connection_type;
 }
 {{ end }}

 {{ if eq $file "GeoIP2-Anonymous-IP.mmdb" }}
 geoip2 /etc/nginx/geoip/GeoIP2-Anonymous-IP.mmdb {
 $geoip2_is_anon source=$remote_addr is_anonymous;
 $geoip2_is_anonymous source=$remote_addr default=0 is_anonymous;
 $geoip2_is_anonymous_vpn source=$remote_addr default=0 is_anonymous_vpn;
 $geoip2_is_hosting_provider source=$remote_addr default=0 is_hosting_provider;
 $geoip2_is_public_proxy source=$remote_addr default=0 is_public_proxy;
 $geoip2_is_tor_exit_node source=$remote_addr default=0 is_tor_exit_node;
 }
 {{ end }}

 {{ end }}

 {{ end }}

 aio threads;
 aio_write on;

 tcp_nopush on;
 tcp_nodelay on;

 log_subrequest on;

 reset_timedout_connection on;

 keepalive_timeout {{ $cfg.KeepAlive }}s;
 keepalive_requests {{ $cfg.KeepAliveRequests }};

 client_body_temp_path /tmp/nginx/client-body;
 fastcgi_temp_path /tmp/nginx/fastcgi-temp;
 proxy_temp_path /tmp/nginx/proxy-temp;
 ajp_temp_path /tmp/nginx/ajp-temp;

 client_header_buffer_size {{ $cfg.ClientHeaderBufferSize }};
 client_header_timeout {{ $cfg.ClientHeaderTimeout }}s;
 large_client_header_buffers {{ $cfg.LargeClientHeaderBuffers }};
 client_body_buffer_size {{ $cfg.ClientBodyBufferSize }};
 client_body_timeout {{ $cfg.ClientBodyTimeout }}s;

 http2_max_field_size {{ $cfg.HTTP2MaxFieldSize }};
 http2_max_header_size {{ $cfg.HTTP2MaxHeaderSize }};
 http2_max_requests {{ $cfg.HTTP2MaxRequests }};
 http2_max_concurrent_streams {{ $cfg.HTTP2MaxConcurrentStreams }};

 types_hash_max_size 2048;
 server_names_hash_max_size {{ $cfg.ServerNameHashMaxSize }};
 server_names_hash_bucket_size {{ $cfg.ServerNameHashBucketSize }};
 map_hash_bucket_size {{ $cfg.MapHashBucketSize }};

 proxy_headers_hash_max_size {{ $cfg.ProxyHeadersHashMaxSize }};
 proxy_headers_hash_bucket_size {{ $cfg.ProxyHeadersHashBucketSize }};

 variables_hash_bucket_size {{ $cfg.VariablesHashBucketSize }};
 variables_hash_max_size {{ $cfg.VariablesHashMaxSize }};

 underscores_in_headers {{ if $cfg.EnableUnderscoresInHeaders }}on{{ else }}off{{ end }};
 ignore_invalid_headers {{ if $cfg.IgnoreInvalidHeaders }}on{{ else }}off{{ end }};

 limit_req_status {{ $cfg.LimitReqStatusCode }};
 limit_conn_status {{ $cfg.LimitConnStatusCode }};

 {{ buildOpentracing $cfg $servers }}

 include /etc/nginx/mime.types;
 default_type {{ $cfg.DefaultType }};

 {{ if $cfg.EnableBrotli }}
 brotli on;
 brotli_comp_level {{ $cfg.BrotliLevel }};
 brotli_min_length {{ $cfg.BrotliMinLength }};
 brotli_types {{ $cfg.BrotliTypes }};
 {{ end }}

 {{ if $cfg.UseGzip }}
 gzip on;
 gzip_comp_level {{ $cfg.GzipLevel }};
 gzip_http_version 1.1;
 gzip_min_length {{ $cfg.GzipMinLength}};
 gzip_types {{ $cfg.GzipTypes }};
 gzip_proxied any;
 gzip_vary on;
 {{ end }}

 # Custom headers for response
 {{ range $k, $v := $addHeaders }}
 more_set_headers {{ printf "%s: %s" $k $v | quote }};
 {{ end }}

 server_tokens {{ if $cfg.ShowServerTokens }}on{{ else }}off{{ end }};
 {{ if not $cfg.ShowServerTokens }}
 more_clear_headers Server;
 {{ end }}

 # disable warnings
 uninitialized_variable_warn off;

 # Additional available variables:
 # $namespace
 # $ingress_name
 # $service_name
 # $service_port
 log_format upstreaminfo {{ if $cfg.LogFormatEscapeJSON }}escape=json {{ end }}'{{ $cfg.LogFormatUpstream }}';

 {{/* map urls that should not appear in access.log */}}
 {{/* http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log */}}
 map $request_uri $loggable {
 {{ range $reqUri := $cfg.SkipAccessLogURLs }}
 {{ $reqUri }} 0;{{ end }}
 default 1;
 }

 {{ if or $cfg.DisableAccessLog $cfg.DisableHTTPAccessLog }}
 access_log off;
 {{ else }}
 {{ if $cfg.EnableSyslog }}
 access_log syslog:server={{ $cfg.SyslogHost }}:{{ $cfg.SyslogPort }} upstreaminfo if=$loggable;
 {{ else }}
 access_log {{ or $cfg.HttpAccessLogPath $cfg.AccessLogPath }} upstreaminfo {{ $cfg.AccessLogParams }} if=$loggable;
 {{ end }}
 {{ end }}

 {{ if $cfg.EnableSyslog }}
 error_log syslog:server={{ $cfg.SyslogHost }}:{{ $cfg.SyslogPort }} {{ $cfg.ErrorLogLevel }};
 {{ else }}
 error_log {{ $cfg.ErrorLogPath }} {{ $cfg.ErrorLogLevel }};
 {{ end }}

 {{ buildResolvers $cfg.Resolver $cfg.DisableIpv6DNS }}

 # See https://www.nginx.com/blog/websocket-nginx
 map $http_upgrade $connection_upgrade {
 default upgrade;
 {{ if (gt $cfg.UpstreamKeepaliveConnections 0) }}
 # See http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
 '' '';
 {{ else }}
 '' close;
 {{ end }}
 }

 # Reverse proxies can detect if a client provides a X-Request-ID header, and pass it on to the backend server.
 # If no such header is provided, it can provide a random value.
 map $http_x_request_id $req_id {
 default $http_x_request_id;
 {{ if $cfg.GenerateRequestID }}
 "" $request_id;
 {{ end }}
 }

 {{ if and $cfg.UseForwardedHeaders $cfg.ComputeFullForwardedFor }}
 # We can't use $proxy_add_x_forwarded_for because the realip module
 # replaces the remote_addr too soon
 map $http_x_forwarded_for $full_x_forwarded_for {
 {{ if $all.Cfg.UseProxyProtocol }}
 default "$http_x_forwarded_for, $proxy_protocol_addr";
 '' "$proxy_protocol_addr";
 {{ else }}
 default "$http_x_forwarded_for, $realip_remote_addr";
 '' "$realip_remote_addr";
 {{ end}}
 }

 {{ end }}

 # Create a variable that contains the literal $ character.
 # This works because the geo module will not resolve variables.
 geo $literal_dollar {
 default "$";
 }

 server_name_in_redirect off;
 port_in_redirect off;

 ssl_protocols {{ $cfg.SSLProtocols }};

 ssl_early_data {{ if $cfg.SSLEarlyData }}on{{ else }}off{{ end }};

 # turn on session caching to drastically improve performance
 {{ if $cfg.SSLSessionCache }}
 ssl_session_cache shared:SSL:{{ $cfg.SSLSessionCacheSize }};
 ssl_session_timeout {{ $cfg.SSLSessionTimeout }};
 {{ end }}

 # allow configuring ssl session tickets
 ssl_session_tickets {{ if $cfg.SSLSessionTickets }}on{{ else }}off{{ end }};

 {{ if not (empty $cfg.SSLSessionTicketKey ) }}
 ssl_session_ticket_key /etc/nginx/tickets.key;
 {{ end }}

 # slightly reduce the time-to-first-byte
 ssl_buffer_size {{ $cfg.SSLBufferSize }};

 {{ if not (empty $cfg.SSLCiphers) }}
 # allow configuring custom ssl ciphers
 ssl_ciphers '{{ $cfg.SSLCiphers }}';
 ssl_prefer_server_ciphers on;
 {{ end }}

 {{ if not (empty $cfg.SSLDHParam) }}
 # allow custom DH file http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_dhparam
 ssl_dhparam {{ $cfg.SSLDHParam }};
 {{ end }}

 ssl_ecdh_curve {{ $cfg.SSLECDHCurve }};

 # PEM sha: {{ $cfg.DefaultSSLCertificate.PemSHA }}
 ssl_certificate {{ $cfg.DefaultSSLCertificate.PemFileName }};
 ssl_certificate_key {{ $cfg.DefaultSSLCertificate.PemFileName }};

 {{ if gt (len $cfg.CustomHTTPErrors) 0 }}
 proxy_intercept_errors on;
 {{ end }}

 {{ range $errCode := $cfg.CustomHTTPErrors }}
 error_page {{ $errCode }} = @custom_upstream-default-backend_{{ $errCode }};{{ end }}

 proxy_ssl_session_reuse on;

 {{ if $cfg.AllowBackendServerHeader }}
 proxy_pass_header Server;
 {{ end }}

 {{ range $header := $cfg.HideHeaders }}proxy_hide_header {{ $header }};
 {{ end }}

 {{ if not (empty $cfg.HTTPSnippet) }}
 # Custom code snippet configured in the configuration configmap
 {{ $cfg.HTTPSnippet }}
 {{ end }}

 upstream upstream_balancer {
 ### Attention!!!
 #
 # We no longer create "upstream" section for every backend.
 # Backends are handled dynamically using Lua. If you would like to debug
 # and see what backends ingress-nginx has in its memory you can
 # install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin.
 # Once you have the plugin you can use "kubectl ingress-nginx backends" command to
 # inspect current backends.
 #
 ###

 server 0.0.0.1; # placeholder

 balancer_by_lua_block {
 balancer.balance()
 }

 {{ if (gt $cfg.UpstreamKeepaliveConnections 0) }}
 keepalive {{ $cfg.UpstreamKeepaliveConnections }};
 keepalive_time {{ $cfg.UpstreamKeepaliveTime }};
 keepalive_timeout {{ $cfg.UpstreamKeepaliveTimeout }}s;
 keepalive_requests {{ $cfg.UpstreamKeepaliveRequests }};
 {{ end }}
 }

 {{ range $rl := (filterRateLimits $servers ) }}
 # Ratelimit {{ $rl.Name }}
 geo $remote_addr $whitelist_{{ $rl.ID }} {
 default 0;
 {{ range $ip := $rl.Whitelist }}
 {{ $ip }} 1;{{ end }}
 }

 # Ratelimit {{ $rl.Name }}
 map $whitelist_{{ $rl.ID }} $limit_{{ $rl.ID }} {
 0 {{ $cfg.LimitConnZoneVariable }};
 1 "";
 }
 {{ end }}

 {{/* build all the required rate limit zones. Each annotation requires a dedicated zone */}}
 {{/* 1MB -> 16 thousand 64-byte states or about 8 thousand 128-byte states */}}
 {{ range $zone := (buildRateLimitZones $servers) }}
 {{ $zone }}
 {{ end }}

 # Cache for internal auth checks
 proxy_cache_path /tmp/nginx/nginx-cache-auth levels=1:2 keys_zone=auth_cache:10m max_size=128m inactive=30m use_temp_path=off;

 # Global filters
 {{ range $ip := $cfg.BlockCIDRs }}deny {{ trimSpace $ip }};
 {{ end }}

 {{ if gt (len $cfg.BlockUserAgents) 0 }}
 map $http_user_agent $block_ua {
 default 0;

 {{ range $ua := $cfg.BlockUserAgents }}{{ trimSpace $ua }} 1;
 {{ end }}
 }
 {{ end }}

 {{ if gt (len $cfg.BlockReferers) 0 }}
 map $http_referer $block_ref {
 default 0;

 {{ range $ref := $cfg.BlockReferers }}{{ trimSpace $ref }} 1;
 {{ end }}
 }
 {{ end }}

 {{/* Build server redirects (from/to www) */}}
 {{ range $redirect := .RedirectServers }}
 ## start server {{ $redirect.From }}
 server {
 server_name {{ $redirect.From }};

 {{ buildHTTPListener $all $redirect.From }}
 {{ buildHTTPSListener $all $redirect.From }}

 ssl_certificate_by_lua_block {
 certificate.call()
 }

 {{ if gt (len $cfg.BlockUserAgents) 0 }}
 if ($block_ua) {
 return 403;
 }
 {{ end }}
 {{ if gt (len $cfg.BlockReferers) 0 }}
 if ($block_ref) {
 return 403;
 }
 {{ end }}

 set_by_lua_block $redirect_to {
 local request_uri = ngx.var.request_uri
 if string.sub(request_uri, -1) == "/" then
 request_uri = string.sub(request_uri, 1, -2)
 end

 {{ if $cfg.UseForwardedHeaders }}
 local redirectScheme
 if not ngx.var.http_x_forwarded_proto then
 redirectScheme = ngx.var.scheme
 else
 redirectScheme = ngx.var.http_x_forwarded_proto
 end
 {{ else }}
 local redirectScheme = ngx.var.scheme
 {{ end }}

 {{ if ne $all.ListenPorts.HTTPS 443 }}
 {{ $redirect_port := (printf ":%v" $all.ListenPorts.HTTPS) }}
 return string.format("%s://%s%s%s", redirectScheme, "{{ $redirect.To }}", "{{ $redirect_port }}", request_uri)
 {{ else }}
 return string.format("%s://%s%s", redirectScheme, "{{ $redirect.To }}", request_uri)
 {{ end }}
 }

 return {{ $all.Cfg.HTTPRedirectCode }} $redirect_to;
 }
 ## end server {{ $redirect.From }}
 {{ end }}

 {{ range $server := $servers }}
 {{ range $location := $server.Locations }}
 {{ $applyGlobalAuth := shouldApplyGlobalAuth $location $all.Cfg.GlobalExternalAuth.URL }}
 {{ $applyAuthUpstream := shouldApplyAuthUpstream $location $all.Cfg }}
 {{ if and (eq $applyAuthUpstream true) (eq $applyGlobalAuth false) }}
 ## start auth upstream {{ $server.Hostname }}{{ $location.Path }}
 upstream {{ buildAuthUpstreamName $location $server.Hostname }} {
 {{- $externalAuth := $location.ExternalAuth }}
 server {{ extractHostPort $externalAuth.URL }};

 keepalive {{ $externalAuth.KeepaliveConnections }};
 keepalive_requests {{ $externalAuth.KeepaliveRequests }};
 keepalive_timeout {{ $externalAuth.KeepaliveTimeout }}s;
 }
 ## end auth upstream {{ $server.Hostname }}{{ $location.Path }}
 {{ end }}
 {{ end }}
 {{ end }}

 {{ range $server := $servers }}
 ## start server {{ $server.Hostname }}
 server {
 server_name {{ buildServerName $server.Hostname }} {{range $server.Aliases }}{{ . }} {{ end }};

 {{ if gt (len $cfg.BlockUserAgents) 0 }}
 if ($block_ua) {
 return 403;
 }
 {{ end }}
 {{ if gt (len $cfg.BlockReferers) 0 }}
 if ($block_ref) {
 return 403;
 }
 {{ end }}

 {{ template "SERVER" serverConfig $all $server }}

 {{ if not (empty $cfg.ServerSnippet) }}
 # Custom code snippet configured in the configuration configmap
 {{ $cfg.ServerSnippet }}
 {{ end }}

 {{ template "CUSTOM_ERRORS" (buildCustomErrorDeps "upstream-default-backend" $cfg.CustomHTTPErrors $all.EnableMetrics $cfg.EnableModsecurity) }}
 }
 ## end server {{ $server.Hostname }}

 {{ end }}

 # backend for when default-backend-service is not configured or it does not have endpoints
 server {
 listen {{ $all.ListenPorts.Default }} default_server {{ if $all.Cfg.ReusePort }}reuseport{{ end }} backlog={{ $all.BacklogSize }};
 {{ if $IsIPV6Enabled }}listen [::]:{{ $all.ListenPorts.Default }} default_server {{ if $all.Cfg.ReusePort }}reuseport{{ end }} backlog={{ $all.BacklogSize }};{{ end }}
 set $proxy_upstream_name "internal";

 access_log off;

 location / {
 return 404;
 }
 }

 # default server, used for NGINX healthcheck and access to nginx stats
 server {
 listen 127.0.0.1:{{ .StatusPort }};
 set $proxy_upstream_name "internal";

 keepalive_timeout 0;
 gzip off;

 access_log off;

 {{ if $cfg.EnableOpentracing }}
 opentracing off;
 {{ end }}

 location {{ $healthzURI }} {
 return 200;
 }

 location /is-dynamic-lb-initialized {
 content_by_lua_block {
 local configuration = require("configuration")
 local backend_data = configuration.get_backends_data()
 if not backend_data then
 ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
 return
 end

 ngx.say("OK")
 ngx.exit(ngx.HTTP_OK)
 }
 }

 location {{ .StatusPath }} {
 stub_status on;
 }

 location /configuration {
 client_max_body_size {{ luaConfigurationRequestBodySize $cfg }};
 client_body_buffer_size {{ luaConfigurationRequestBodySize $cfg }};
 proxy_buffering off;

 content_by_lua_block {
 configuration.call()
 }
 }

 location / {
 content_by_lua_block {
 ngx.exit(ngx.HTTP_NOT_FOUND)
 }
 }
 }
}

stream {
 lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;;";

 lua_shared_dict tcp_udp_configuration_data 5M;

 init_by_lua_block {
 collectgarbage("collect")

 -- init modules
 local ok, res

 ok, res = pcall(require, "configuration")
 if not ok then
 error("require failed: " .. tostring(res))
 else
 configuration = res
 end

 ok, res = pcall(require, "tcp_udp_configuration")
 if not ok then
 error("require failed: " .. tostring(res))
 else
 tcp_udp_configuration = res
 tcp_udp_configuration.prohibited_localhost_port = '{{ .StatusPort }}'

 end

 ok, res = pcall(require, "tcp_udp_balancer")
 if not ok then
 error("require failed: " .. tostring(res))
 else
 tcp_udp_balancer = res
 end
 }

 init_worker_by_lua_block {
 tcp_udp_balancer.init_worker()
 }

 lua_add_variable $proxy_upstream_name;

 log_format log_stream '{{ $cfg.LogFormatStream }}';

 {{ if or $cfg.DisableAccessLog $cfg.DisableStreamAccessLog }}
 access_log off;
 {{ else }}
 access_log {{ or $cfg.StreamAccessLogPath $cfg.AccessLogPath }} log_stream {{ $cfg.AccessLogParams }};
 {{ end }}


 error_log {{ $cfg.ErrorLogPath }} {{ $cfg.ErrorLogLevel }};
 {{ if $cfg.EnableRealIp }}
 {{ range $trusted_ip := $cfg.ProxyRealIPCIDR }}
 set_real_ip_from {{ $trusted_ip }};
 {{ end }}
 {{ end }}

 upstream upstream_balancer {
 server 0.0.0.1:1234; # placeholder

 balancer_by_lua_block {
 tcp_udp_balancer.balance()
 }
 }

 server {
 listen 127.0.0.1:{{ .StreamPort }};

 access_log off;

 content_by_lua_block {
 tcp_udp_configuration.call()
 }
 }

 # TCP services
 {{ range $tcpServer := .TCPBackends }}
 server {
 preread_by_lua_block {
 ngx.var.proxy_upstream_name="tcp-{{ $tcpServer.Backend.Namespace }}-{{ $tcpServer.Backend.Name }}-{{ $tcpServer.Backend.Port }}";
 }

 {{ range $address := $all.Cfg.BindAddressIpv4 }}
 listen {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
 {{ else }}
 listen {{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
 {{ end }}
 {{ if $IsIPV6Enabled }}
 {{ range $address := $all.Cfg.BindAddressIpv6 }}
 listen {{ $address }}:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
 {{ else }}
 listen [::]:{{ $tcpServer.Port }}{{ if $tcpServer.Backend.ProxyProtocol.Decode }} proxy_protocol{{ end }};
 {{ end }}
 {{ end }}
 proxy_timeout {{ $cfg.ProxyStreamTimeout }};
 proxy_next_upstream {{ if $cfg.ProxyStreamNextUpstream }}on{{ else }}off{{ end }};
 proxy_next_upstream_timeout {{ $cfg.ProxyStreamNextUpstreamTimeout }};
 proxy_next_upstream_tries {{ $cfg.ProxyStreamNextUpstreamTries }};

 proxy_pass upstream_balancer;
 {{ if $tcpServer.Backend.ProxyProtocol.Encode }}
 proxy_protocol on;
 {{ end }}
 }
 {{ end }}

 # UDP services
 {{ range $udpServer := .UDPBackends }}
 server {
 preread_by_lua_block {
 ngx.var.proxy_upstream_name="udp-{{ $udpServer.Backend.Namespace }}-{{ $udpServer.Backend.Name }}-{{ $udpServer.Backend.Port }}";
 }

 {{ range $address := $all.Cfg.BindAddressIpv4 }}
 listen {{ $address }}:{{ $udpServer.Port }} udp;
 {{ else }}
 listen {{ $udpServer.Port }} udp;
 {{ end }}
 {{ if $IsIPV6Enabled }}
 {{ range $address := $all.Cfg.BindAddressIpv6 }}
 listen {{ $address }}:{{ $udpServer.Port }} udp;
 {{ else }}
 listen [::]:{{ $udpServer.Port }} udp;
 {{ end }}
 {{ end }}
 proxy_responses {{ $cfg.ProxyStreamResponses }};
 proxy_timeout {{ $cfg.ProxyStreamTimeout }};
 proxy_next_upstream {{ if $cfg.ProxyStreamNextUpstream }}on{{ else }}off{{ end }};
 proxy_next_upstream_timeout {{ $cfg.ProxyStreamNextUpstreamTimeout }};
 proxy_next_upstream_tries {{ $cfg.ProxyStreamNextUpstreamTries }};
 proxy_pass upstream_balancer;
 }
 {{ end }}

 # Stream Snippets
 {{ range $snippet := .StreamSnippets }}
 {{ $snippet }}
 {{ end }}
}

{{/* definition of templates to avoid repetitions */}}
{{ define "CUSTOM_ERRORS" }}
 {{ $enableMetrics := .EnableMetrics }}
 {{ $modsecurityEnabled := .ModsecurityEnabled }}
 {{ $upstreamName := .UpstreamName }}
 {{ range $errCode := .ErrorCodes }}
 location @custom_{{ $upstreamName }}_{{ $errCode }} {
 internal;

 # Ensure that modsecurity will not run on custom error pages or they might be blocked
 {{ if $modsecurityEnabled }}
 modsecurity off;
 {{ end }}

 proxy_intercept_errors off;

 proxy_set_header X-Code {{ $errCode }};
 proxy_set_header X-Format $http_accept;
 proxy_set_header X-Original-URI $request_uri;
 proxy_set_header X-Namespace $namespace;
 proxy_set_header X-Ingress-Name $ingress_name;
 proxy_set_header X-Service-Name $service_name;
 proxy_set_header X-Service-Port $service_port;
 proxy_set_header X-Request-ID $req_id;
 proxy_set_header X-Forwarded-For $remote_addr;
 proxy_set_header Host $best_http_host;

 set $proxy_upstream_name {{ $upstreamName | quote }};

 rewrite (.*) / break;

 proxy_pass http://upstream_balancer;
 log_by_lua_block {
 {{ if $enableMetrics }}
 monitor.call()
 {{ end }}
 }
 }
 {{ end }}
{{ end }}

{{/* CORS support from https://michielkalkman.com/snippets/nginx-cors-open-configuration.html */}}
{{ define "CORS" }}
 {{ $cors := .CorsConfig }}
 # Cors Preflight methods needs additional options and different Return Code
 {{ if $cors.CorsAllowOrigin }}
 {{ buildCorsOriginRegex $cors.CorsAllowOrigin }}
 {{ end }}
 if ($request_method = 'OPTIONS') {
 set $cors ${cors}options;
 }

 if ($cors = "true") {
 more_set_headers 'Access-Control-Allow-Origin: $http_origin';
 {{ if $cors.CorsAllowCredentials }} more_set_headers 'Access-Control-Allow-Credentials: {{ $cors.CorsAllowCredentials }}'; {{ end }}
 more_set_headers 'Access-Control-Allow-Methods: {{ $cors.CorsAllowMethods }}';
 more_set_headers 'Access-Control-Allow-Headers: {{ $cors.CorsAllowHeaders }}';
 {{ if not (empty $cors.CorsExposeHeaders) }} more_set_headers 'Access-Control-Expose-Headers: {{ $cors.CorsExposeHeaders }}'; {{ end }}
 more_set_headers 'Access-Control-Max-Age: {{ $cors.CorsMaxAge }}';
 }

 if ($cors = "trueoptions") {
 more_set_headers 'Access-Control-Allow-Origin: $http_origin';
 {{ if $cors.CorsAllowCredentials }} more_set_headers 'Access-Control-Allow-Credentials: {{ $cors.CorsAllowCredentials }}'; {{ end }}
 more_set_headers 'Access-Control-Allow-Methods: {{ $cors.CorsAllowMethods }}';
 more_set_headers 'Access-Control-Allow-Headers: {{ $cors.CorsAllowHeaders }}';
 {{ if not (empty $cors.CorsExposeHeaders) }} more_set_headers 'Access-Control-Expose-Headers: {{ $cors.CorsExposeHeaders }}'; {{ end }}
 more_set_headers 'Access-Control-Max-Age: {{ $cors.CorsMaxAge }}';
 more_set_headers 'Content-Type: text/plain charset=UTF-8';
 more_set_headers 'Content-Length: 0';
 return 204;
 }
{{ end }}

{{/* definition of server-template to avoid repetitions with server-alias */}}
{{ define "SERVER" }}
 {{ $all := .First }}
 {{ $server := .Second }}

 {{ buildHTTPListener $all $server.Hostname }}
 {{ buildHTTPSListener $all $server.Hostname }}

 set $proxy_upstream_name "-";

 {{ if not ( empty $server.CertificateAuth.MatchCN ) }}
 {{ if gt (len $server.CertificateAuth.MatchCN) 0 }}
 if ( $ssl_client_s_dn !~ {{ $server.CertificateAuth.MatchCN }} ) {
 return 403 "client certificate unauthorized";
 }
 {{ end }}
 {{ end }}

 {{ if eq $server.Hostname "_" }}
 ssl_reject_handshake {{ if $all.Cfg.SSLRejectHandshake }}on{{ else }}off{{ end }};
 {{ end }}

 ssl_certificate_by_lua_block {
 certificate.call()
 }

 {{ if not (empty $server.AuthTLSError) }}
 # {{ $server.AuthTLSError }}
 return 403;
 {{ else }}

 {{ if not (empty $server.CertificateAuth.CAFileName) }}
 # PEM sha: {{ $server.CertificateAuth.CASHA }}
 ssl_client_certificate {{ $server.CertificateAuth.CAFileName }};
 ssl_verify_client {{ $server.CertificateAuth.VerifyClient }};
 ssl_verify_depth {{ $server.CertificateAuth.ValidationDepth }};

 {{ if not (empty $server.CertificateAuth.CRLFileName) }}
 # PEM sha: {{ $server.CertificateAuth.CRLSHA }}
 ssl_crl {{ $server.CertificateAuth.CRLFileName }};
 {{ end }}

 {{ if not (empty $server.CertificateAuth.ErrorPage)}}
 error_page 495 496 = {{ $server.CertificateAuth.ErrorPage }};
 {{ end }}
 {{ end }}

 {{ if not (empty $server.ProxySSL.CAFileName) }}
 # PEM sha: {{ $server.ProxySSL.CASHA }}
 proxy_ssl_trusted_certificate {{ $server.ProxySSL.CAFileName }};
 proxy_ssl_ciphers {{ $server.ProxySSL.Ciphers }};
 proxy_ssl_protocols {{ $server.ProxySSL.Protocols }};
 proxy_ssl_verify {{ $server.ProxySSL.Verify }};
 proxy_ssl_verify_depth {{ $server.ProxySSL.VerifyDepth }};
 {{ if not (empty $server.ProxySSL.ProxySSLName) }}
 proxy_ssl_name {{ $server.ProxySSL.ProxySSLName }};
 proxy_ssl_server_name {{ $server.ProxySSL.ProxySSLServerName }};
 {{ end }}
 {{ end }}

 {{ if not (empty $server.ProxySSL.PemFileName) }}
 proxy_ssl_certificate {{ $server.ProxySSL.PemFileName }};
 proxy_ssl_certificate_key {{ $server.ProxySSL.PemFileName }};
 {{ end }}

 {{ if not (empty $server.SSLCiphers) }}
 ssl_ciphers {{ $server.SSLCiphers }};
 {{ end }}

 {{ if not (empty $server.SSLPreferServerCiphers) }}
 ssl_prefer_server_ciphers {{ $server.SSLPreferServerCiphers }};
 {{ end }}

 {{ if not (empty $server.ServerSnippet) }}
 # Custom code snippet configured for host {{ $server.Hostname }}
 {{ $server.ServerSnippet }}
 {{ end }}

 {{ range $errorLocation := (buildCustomErrorLocationsPerServer $server) }}
 {{ template "CUSTOM_ERRORS" (buildCustomErrorDeps $errorLocation.UpstreamName $errorLocation.Codes $all.EnableMetrics $all.Cfg.EnableModsecurity) }}
 {{ end }}

 {{ buildMirrorLocations $server.Locations }}

 {{ $enforceRegex := enforceRegexModifier $server.Locations }}
 {{ range $location := $server.Locations }}
 {{ $path := buildLocation $location $enforceRegex }}
 {{ $proxySetHeader := proxySetHeader $location }}
 {{ $authPath := buildAuthLocation $location $all.Cfg.GlobalExternalAuth.URL }}
 {{ $applyGlobalAuth := shouldApplyGlobalAuth $location $all.Cfg.GlobalExternalAuth.URL }}
 {{ $applyAuthUpstream := shouldApplyAuthUpstream $location $all.Cfg }}

 {{ $externalAuth := $location.ExternalAuth }}
 {{ if eq $applyGlobalAuth true }}
 {{ $externalAuth = $all.Cfg.GlobalExternalAuth }}
 {{ end }}

 {{ if not (empty $location.Rewrite.AppRoot) }}
 if ($uri = /) {
 return 302 $scheme://$http_host{{ $location.Rewrite.AppRoot }};
 }
 {{ end }}

 {{ if $authPath }}
 location = {{ $authPath }} {
 internal;

 {{ if (or $all.Cfg.EnableOpentracing $location.Opentracing.Enabled) }}
 opentracing on;
 opentracing_propagate_context;
 {{ end }}

 # Ensure that modsecurity will not run on an internal location as this is not accessible from outside
 {{ if $all.Cfg.EnableModsecurity }}
 modsecurity off;
 {{ end }}

 {{ if $externalAuth.AuthCacheKey }}
 set $tmp_cache_key '{{ $server.Hostname }}{{ $authPath }}{{ $externalAuth.AuthCacheKey }}';
 set $cache_key '';

 rewrite_by_lua_block {
 ngx.var.cache_key = ngx.encode_base64(ngx.sha1_bin(ngx.var.tmp_cache_key))
 }

 proxy_cache auth_cache;

 {{- range $dur := $externalAuth.AuthCacheDuration }}
 proxy_cache_valid {{ $dur }};
 {{- end }}

 proxy_cache_key "$cache_key";
 {{ end }}

 # ngx_auth_request module overrides variables in the parent request,
 # therefore we have to explicitly set this variable again so that when the parent request
 # resumes it has the correct value set for this variable so that Lua can pick backend correctly
 set $proxy_upstream_name {{ buildUpstreamName $location | quote }};

 proxy_pass_request_body off;
 proxy_set_header Content-Length "";
 proxy_set_header X-Forwarded-Proto "";
 proxy_set_header X-Request-ID $req_id;

 {{ if $externalAuth.Method }}
 proxy_method {{ $externalAuth.Method }};
 proxy_set_header X-Original-URI $request_uri;
 proxy_set_header X-Scheme $pass_access_scheme;
 {{ end }}

 proxy_set_header Host {{ $externalAuth.Host }};
 proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
 proxy_set_header X-Original-Method $request_method;
 proxy_set_header X-Sent-From "nginx-ingress-controller";
 proxy_set_header X-Real-IP $remote_addr;
 {{ if and $all.Cfg.UseForwardedHeaders $all.Cfg.ComputeFullForwardedFor }}
 proxy_set_header X-Forwarded-For $full_x_forwarded_for;
 {{ else }}
 proxy_set_header X-Forwarded-For $remote_addr;
 {{ end }}

 {{ if $externalAuth.RequestRedirect }}
 proxy_set_header X-Auth-Request-Redirect {{ $externalAuth.RequestRedirect }};
 {{ else }}
 proxy_set_header X-Auth-Request-Redirect $request_uri;
 {{ end }}

 {{ if $externalAuth.AuthCacheKey }}
 proxy_buffering "on";
 {{ else }}
 proxy_buffering {{ $location.Proxy.ProxyBuffering }};
 {{ end }}
 proxy_buffer_size {{ $location.Proxy.BufferSize }};
 proxy_buffers {{ $location.Proxy.BuffersNumber }} {{ $location.Proxy.BufferSize }};
 proxy_request_buffering {{ $location.Proxy.RequestBuffering }};

 proxy_ssl_server_name on;
 proxy_pass_request_headers on;
 {{ if isValidByteSize $location.Proxy.BodySize true }}
 client_max_body_size {{ $location.Proxy.BodySize }};
 {{ end }}
 {{ if isValidByteSize $location.ClientBodyBufferSize false }}
 client_body_buffer_size {{ $location.ClientBodyBufferSize }};
 {{ end }}

 # Pass the extracted client certificate to the auth provider
 {{ if not (empty $server.CertificateAuth.CAFileName) }}
 {{ if $server.CertificateAuth.PassCertToUpstream }}
 proxy_set_header ssl-client-cert $ssl_client_escaped_cert;
 {{ end }}
 proxy_set_header ssl-client-verify $ssl_client_verify;
 proxy_set_header ssl-client-subject-dn $ssl_client_s_dn;
 proxy_set_header ssl-client-issuer-dn $ssl_client_i_dn;
 {{ end }}

 {{- range $line := buildAuthProxySetHeaders $externalAuth.ProxySetHeaders}}
 {{ $line }}
 {{- end }}

 {{ if not (empty $externalAuth.AuthSnippet) }}
 {{ $externalAuth.AuthSnippet }}
 {{ end }}

 {{ if and (eq $applyAuthUpstream true) (eq $applyGlobalAuth false) }}
 {{ $authUpstreamName := buildAuthUpstreamName $location $server.Hostname }}
 # The target is an upstream with HTTP keepalive, that is why the
 # Connection header is cleared and the HTTP version is set to 1.1 as
 # the Nginx documentation suggests:
 # http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
 proxy_http_version 1.1;
 proxy_set_header Connection "";
 set $target {{ changeHostPort $externalAuth.URL $authUpstreamName }};
 {{ else }}
 proxy_http_version {{ $location.Proxy.ProxyHTTPVersion }};
 set $target {{ $externalAuth.URL }};
 {{ end }}
 proxy_pass $target;
 }
 {{ end }}

 {{ if isLocationAllowed $location }}
 {{ if $externalAuth.SigninURL }}
 location {{ buildAuthSignURLLocation $location.Path $externalAuth.SigninURL }} {
 internal;

 add_header Set-Cookie $auth_cookie;

 # Ensure that modsecurity will not run on an internal location as this is not accessible from outside
 {{ if $all.Cfg.EnableModsecurity }}
 modsecurity off;
 {{ end }}

 return 302 {{ buildAuthSignURL $externalAuth.SigninURL $externalAuth.SigninURLRedirectParam }};
 }
 {{ end }}
 {{ end }}

 location {{ $path }} {
 {{ $ing := (getIngressInformation $location.Ingress $server.Hostname $location.IngressPath) }}
 set $namespace {{ $ing.Namespace | quote}};
 set $ingress_name {{ $ing.Rule | quote }};
 set $service_name {{ $ing.Service | quote }};
 set $service_port {{ $ing.ServicePort | quote }};
 set $location_path {{ $ing.Path | escapeLiteralDollar | quote }};
 set $global_rate_limit_exceeding n;

 {{ buildOpentracingForLocation $all.Cfg.EnableOpentracing $all.Cfg.OpentracingTrustIncomingSpan $location }}

 {{ if $location.Mirror.Source }}
 mirror {{ $location.Mirror.Source }};
 mirror_request_body {{ $location.Mirror.RequestBody }};
 {{ end }}

 rewrite_by_lua_block {
 lua_ingress.rewrite({{ locationConfigForLua $location $all }})
 balancer.rewrite()
 plugins.run()
 }

 # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
 # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
 # other authentication method such as basic auth or external auth useless - all requests will be allowed.
 #access_by_lua_block {
 #}

 header_filter_by_lua_block {
 lua_ingress.header()
 plugins.run()
 }

 body_filter_by_lua_block {
 plugins.run()
 }

 log_by_lua_block {
 balancer.log()
 {{ if $all.EnableMetrics }}
 monitor.call()
 {{ end }}

 plugins.run()
 }

 {{ if not $location.Logs.Access }}
 access_log off;
 {{ end }}

 {{ if $location.Logs.Rewrite }}
 rewrite_log on;
 {{ end }}

 {{ if $location.HTTP2PushPreload }}
 http2_push_preload on;
 {{ end }}

 port_in_redirect {{ if $location.UsePortInRedirects }}on{{ else }}off{{ end }};

 set $balancer_ewma_score -1;
 set $proxy_upstream_name {{ buildUpstreamName $location | quote }};
 set $proxy_host $proxy_upstream_name;
 set $pass_access_scheme $scheme;

 {{ if $all.Cfg.UseProxyProtocol }}
 set $pass_server_port $proxy_protocol_server_port;
 {{ else }}
 set $pass_server_port $server_port;
 {{ end }}

 set $best_http_host $http_host;
 set $pass_port $pass_server_port;

 set $proxy_alternative_upstream_name "";

 {{ buildModSecurityForLocation $all.Cfg $location }}

 {{ if isLocationAllowed $location }}
 {{ if gt (len $location.Whitelist.CIDR) 0 }}
 {{ range $ip := $location.Whitelist.CIDR }}
 allow {{ $ip }};{{ end }}
 deny all;
 {{ end }}

 {{ if not (isLocationInLocationList $location $all.Cfg.NoAuthLocations) }}
 {{ if $authPath }}
 # this location requires authentication
 {{ if and (eq $applyAuthUpstream true) (eq $applyGlobalAuth false) }}
 set $auth_cookie '';
 add_header Set-Cookie $auth_cookie;
 {{- range $line := buildAuthResponseHeaders $proxySetHeader $externalAuth.ResponseHeaders true }}
 {{ $line }}
 {{- end }}
 # `auth_request` module does not support HTTP keepalives in upstream block:
 # https://trac.nginx.org/nginx/ticket/1579
 access_by_lua_block {
 local res = ngx.location.capture('{{ $authPath }}', { method = ngx.HTTP_GET, body = '' })
 if res.status == ngx.HTTP_OK then
 ngx.var.auth_cookie = res.header['Set-Cookie']
 {{- range $line := buildAuthUpstreamLuaHeaders $externalAuth.ResponseHeaders }}
 {{ $line }}
 {{- end }}
 return
 end
 if res.status == ngx.HTTP_FORBIDDEN then
 ngx.exit(res.status)
 end
 ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
 }
 {{ else }}
 auth_request {{ $authPath }};
 auth_request_set $auth_cookie $upstream_http_set_cookie;
 {{ if $externalAuth.AlwaysSetCookie }}
 add_header Set-Cookie $auth_cookie always;
 {{ else }}
 add_header Set-Cookie $auth_cookie;
 {{ end }}
 {{- range $line := buildAuthResponseHeaders $proxySetHeader $externalAuth.ResponseHeaders false }}
 {{ $line }}
 {{- end }}
 {{ end }}
 {{ end }}

 {{ if $externalAuth.SigninURL }}
 set_escape_uri $escaped_request_uri $request_uri;
 error_page 401 = {{ buildAuthSignURLLocation $location.Path $externalAuth.SigninURL }};
 {{ end }}

 {{ if $location.BasicDigestAuth.Secured }}
 {{ if eq $location.BasicDigestAuth.Type "basic" }}
 auth_basic {{ $location.BasicDigestAuth.Realm | quote }};
 auth_basic_user_file {{ $location.BasicDigestAuth.File }};
 {{ else }}
 auth_digest {{ $location.BasicDigestAuth.Realm | quote }};
 auth_digest_user_file {{ $location.BasicDigestAuth.File }};
 {{ end }}
 {{ $proxySetHeader }} Authorization "";
 {{ end }}
 {{ end }}

 {{/* if the location contains a rate limit annotation, create one */}}
 {{ $limits := buildRateLimit $location }}
 {{ range $limit := $limits }}
 {{ $limit }}{{ end }}

 {{ if $location.CorsConfig.CorsEnabled }}
 {{ template "CORS" $location }}
 {{ end }}

 {{ buildInfluxDB $location.InfluxDB }}

 {{ if isValidByteSize $location.Proxy.BodySize true }}
 client_max_body_size {{ $location.Proxy.BodySize }};
 {{ end }}
 {{ if isValidByteSize $location.ClientBodyBufferSize false }}
 client_body_buffer_size {{ $location.ClientBodyBufferSize }};
 {{ end }}

 {{/* By default use vhost as Host to upstream, but allow overrides */}}
 {{ if not (eq $proxySetHeader "grpc_set_header") }}
 {{ if not (empty $location.UpstreamVhost) }}
 {{ $proxySetHeader }} Host {{ $location.UpstreamVhost | quote }};
 {{ else }}
 {{ $proxySetHeader }} Host $best_http_host;
 {{ end }}
 {{ end }}

 # Pass the extracted client certificate to the backend
 {{ if not (empty $server.CertificateAuth.CAFileName) }}
 {{ if $server.CertificateAuth.PassCertToUpstream }}
 {{ $proxySetHeader }} ssl-client-cert $ssl_client_escaped_cert;
 {{ end }}
 {{ $proxySetHeader }} ssl-client-verify $ssl_client_verify;
 {{ $proxySetHeader }} ssl-client-subject-dn $ssl_client_s_dn;
 {{ $proxySetHeader }} ssl-client-issuer-dn $ssl_client_i_dn;
 {{ end }}

 # Allow websocket connections
 {{ $proxySetHeader }} Upgrade $http_upgrade;
 {{ if $location.Connection.Enabled}}
 {{ $proxySetHeader }} Connection {{ $location.Connection.Header }};
 {{ else }}
 {{ $proxySetHeader }} Connection $connection_upgrade;
 {{ end }}

 {{ $proxySetHeader }} X-Request-ID $req_id;
 {{ $proxySetHeader }} X-Real-IP $remote_addr;
 {{ if and $all.Cfg.UseForwardedHeaders $all.Cfg.ComputeFullForwardedFor }}
 {{ $proxySetHeader }} X-Forwarded-For $full_x_forwarded_for;
 {{ else }}
 {{ $proxySetHeader }} X-Forwarded-For $remote_addr;
 {{ end }}
 {{ $proxySetHeader }} X-Forwarded-Host $best_http_host;
 {{ $proxySetHeader }} X-Forwarded-Port $pass_port;
 {{ $proxySetHeader }} X-Forwarded-Proto $pass_access_scheme;
 {{ $proxySetHeader }} X-Forwarded-Scheme $pass_access_scheme;
 {{ if $all.Cfg.ProxyAddOriginalURIHeader }}
 {{ $proxySetHeader }} X-Original-URI $request_uri;
 {{ end }}
 {{ $proxySetHeader }} X-Scheme $pass_access_scheme;

 # Pass the original X-Forwarded-For
 {{ $proxySetHeader }} X-Original-Forwarded-For {{ buildForwardedFor $all.Cfg.ForwardedForHeader }};

 # mitigate HTTPoxy Vulnerability
 # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
 {{ $proxySetHeader }} Proxy "";

 # Custom headers to proxied server
 {{ range $k, $v := $all.ProxySetHeaders }}
 {{ $proxySetHeader }} {{ $k }} {{ $v | quote }};
 {{ end }}

 proxy_connect_timeout {{ $location.Proxy.ConnectTimeout }}s;
 proxy_send_timeout {{ $location.Proxy.SendTimeout }}s;
 proxy_read_timeout {{ $location.Proxy.ReadTimeout }}s;

 proxy_buffering {{ $location.Proxy.ProxyBuffering }};
 proxy_buffer_size {{ $location.Proxy.BufferSize }};
 proxy_buffers {{ $location.Proxy.BuffersNumber }} {{ $location.Proxy.BufferSize }};
 {{ if isValidByteSize $location.Proxy.ProxyMaxTempFileSize true }}
 proxy_max_temp_file_size {{ $location.Proxy.ProxyMaxTempFileSize }};
 {{ end }}
 proxy_request_buffering {{ $location.Proxy.RequestBuffering }};
 proxy_http_version {{ $location.Proxy.ProxyHTTPVersion }};

 proxy_cookie_domain {{ $location.Proxy.CookieDomain }};
 proxy_cookie_path {{ $location.Proxy.CookiePath }};

 # In case of errors try the next upstream server before returning an error
 proxy_next_upstream {{ buildNextUpstream $location.Proxy.NextUpstream $all.Cfg.RetryNonIdempotent }};
 proxy_next_upstream_timeout {{ $location.Proxy.NextUpstreamTimeout }};
 proxy_next_upstream_tries {{ $location.Proxy.NextUpstreamTries }};

 {{/* Add any additional configuration defined */}}
 {{ $location.ConfigurationSnippet }}

 {{ if not (empty $all.Cfg.LocationSnippet) }}
 # Custom code snippet configured in the configuration configmap
 {{ $all.Cfg.LocationSnippet }}
 {{ end }}

 {{/* if we are sending the request to a custom default backend, we add the required headers */}}
 {{ if (hasPrefix $location.Backend "custom-default-backend-") }}
 proxy_set_header X-Code 503;
 proxy_set_header X-Format $http_accept;
 proxy_set_header X-Namespace $namespace;
 proxy_set_header X-Ingress-Name $ingress_name;
 proxy_set_header X-Service-Name $service_name;
 proxy_set_header X-Service-Port $service_port;
 proxy_set_header X-Request-ID $req_id;
 {{ end }}

 {{ if $location.Satisfy }}
 satisfy {{ $location.Satisfy }};
 {{ end }}

 {{/* if a location-specific error override is set, add the proxy_intercept here */}}
 {{ if $location.CustomHTTPErrors }}
 # Custom error pages per ingress
 proxy_intercept_errors on;
 {{ end }}

 {{ range $errCode := $location.CustomHTTPErrors }}
 error_page {{ $errCode }} = @custom_{{ $location.DefaultBackendUpstreamName }}_{{ $errCode }};{{ end }}

 {{ if (eq $location.BackendProtocol "FCGI") }}
 include /etc/nginx/fastcgi_params;
 {{ end }}
 {{- if $location.FastCGI.Index -}}
 fastcgi_index {{ $location.FastCGI.Index | quote }};
 {{- end -}}
 {{ range $k, $v := $location.FastCGI.Params }}
 fastcgi_param {{ $k }} {{ $v | quote }};
 {{ end }}

 {{ if not (empty $location.Redirect.URL) }}
 return {{ $location.Redirect.Code }} {{ $location.Redirect.URL }};
 {{ end }}

 {{ buildProxyPass $server.Hostname $all.Backends $location }}
 {{ if (or (eq $location.Proxy.ProxyRedirectFrom "default") (eq $location.Proxy.ProxyRedirectFrom "off")) }}
 proxy_redirect {{ $location.Proxy.ProxyRedirectFrom }};
 {{ else if not (eq $location.Proxy.ProxyRedirectTo "off") }}
 proxy_redirect {{ $location.Proxy.ProxyRedirectFrom }} {{ $location.Proxy.ProxyRedirectTo }};
 {{ end }}
 {{ else }}
 # Location denied. Reason: {{ $location.Denied | quote }}
 return 503;
 {{ end }}
 {{ if not (empty $location.ProxySSL.CAFileName) }}
 # PEM sha: {{ $location.ProxySSL.CASHA }}
 proxy_ssl_trusted_certificate {{ $location.ProxySSL.CAFileName }};
 proxy_ssl_ciphers {{ $location.ProxySSL.Ciphers }};
 proxy_ssl_protocols {{ $location.ProxySSL.Protocols }};
 proxy_ssl_verify {{ $location.ProxySSL.Verify }};
 proxy_ssl_verify_depth {{ $location.ProxySSL.VerifyDepth }};
 {{ end }}

 {{ if not (empty $location.ProxySSL.ProxySSLName) }}
 proxy_ssl_name {{ $location.ProxySSL.ProxySSLName }};
 {{ end }}
 {{ if not (empty $location.ProxySSL.ProxySSLServerName) }}
 proxy_ssl_server_name {{ $location.ProxySSL.ProxySSLServerName }};
 {{ end }}

 {{ if not (empty $location.ProxySSL.PemFileName) }}
 proxy_ssl_certificate {{ $location.ProxySSL.PemFileName }};
 proxy_ssl_certificate_key {{ $location.ProxySSL.PemFileName }};
 {{ end }}
 }
 {{ end }}
 {{ end }}

 {{ if eq $server.Hostname "_" }}
 # health checks in cloud providers require the use of port {{ $all.ListenPorts.HTTP }}
 location {{ $all.HealthzURI }} {
 {{ if $all.Cfg.EnableOpentracing }}
 opentracing off;
 {{ end }}

 access_log off;
 return 200;
 }

 # this is required to avoid error if nginx is being monitored
 # with an external software (like sysdig)
 location /nginx_status {
 {{ if $all.Cfg.EnableOpentracing }}
 opentracing off;
 {{ end }}

 {{ range $v := $all.NginxStatusIpv4Whitelist }}
 allow {{ $v }};
 {{ end }}
 {{ if $all.IsIPV6Enabled -}}
 {{ range $v := $all.NginxStatusIpv6Whitelist }}
 allow {{ $v }};
 {{ end }}
 {{ end -}}
 deny all;

 access_log off;
 stub_status on;
 }

 {{ end }}

{{ end }}