mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
fix(ingress-nginx): raise client keepalive_time above the 1h default (#5250)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, security (a fix that comes through a security advisory), docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Fixes merged to `main` reach the release lines the backport batch runs for automatically, so do not open backport PRs by hand: https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md --> ## What this PR does ingress-nginx exposes no ConfigMap key for the client-side `keepalive_time`, so nginx keeps its built-in 1h default: every client connection older than an hour is closed, and on HTTP/2 that is a `GOAWAY` (`ErrCode=NO_ERROR`). A client that multiplexes a long operation over one connection loses the request in flight. We hit it with a kopia repository maintenance run by Velero against a SeaweedFS S3 endpoint published through this controller: the run opened its connection at 18:57:05, finished the heavy part, and failed at 19:57:38 while listing blobs with `http2: server sent GOAWAY and closed the connection`, because kopia does not retry that listing. A shorter maintenance on the previous day succeeded on the same repository. This sets `keepalive_time 12h` through `http-snippet` in the controller ConfigMap. Idle connections are still closed after `keep-alive` (75s), so only connections that are actually in use live longer. No cozystack template sets `http-snippet` today; the tenant ingress HelmRelease only sets other `controller.config` keys, which Helm merges with this default, and a test covers that merge. Applied by hand on a production cluster before this PR: the controllers reloaded cleanly and S3 traffic was unaffected. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(ingress-nginx): raise the client keepalive_time from the nginx 1h default to 12h, so long-lived HTTP/2 connections (for example kopia maintenance against S3) are no longer cut by a GOAWAY after one hour ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * The ingress controller now configures NGINX to allow HTTP connections to remain open for up to 12 hours, supporting long-running operations such as S3 transfers. * **Tests** * Added checks confirming the setting remains in place when PROXY protocol and whitelist configuration are enabled. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
2 files changed
+33
No files matched your search
@@ -0,0 +1,26 @@
|
||||
suite: ingress-nginx client keepalive_time
|
||||
release:
|
||||
name: ingress-nginx
|
||||
namespace: cozy-ingress-nginx
|
||||
templates:
|
||||
- charts/ingress-nginx/templates/controller-configmap.yaml
|
||||
tests:
|
||||
- it: raises the client keepalive_time above the nginx 1h default
|
||||
asserts:
|
||||
- equal:
|
||||
path: data.http-snippet
|
||||
value: keepalive_time 12h;
|
||||
- it: keeps the snippet when a tenant ingress sets its own config keys
|
||||
# The tenant ingress HelmRelease passes controller.config for whitelist,
|
||||
# Cloudflare and PROXY protocol; Helm merges maps, so those keys must not
|
||||
# drop the default snippet.
|
||||
set:
|
||||
ingress-nginx.controller.config.use-proxy-protocol: "true"
|
||||
ingress-nginx.controller.config.whitelist-source-range: 10.0.0.0/8
|
||||
asserts:
|
||||
- equal:
|
||||
path: data.http-snippet
|
||||
value: keepalive_time 12h;
|
||||
- equal:
|
||||
path: data.use-proxy-protocol
|
||||
value: "true"
|
||||
@@ -47,6 +47,13 @@ ingress-nginx:
|
||||
upstream-keepalive-timeout: "60"
|
||||
upstream-keepalive-connections: "320"
|
||||
ssl-session-tickets: "true"
|
||||
# Client-side keepalive_time has no ConfigMap key, so nginx keeps its
|
||||
# 1h default and sends GOAWAY to any HTTP/2 connection older than that.
|
||||
# A long S3 operation (kopia maintenance behind Velero) multiplexes its
|
||||
# requests over one connection, and the in-flight object listing fails
|
||||
# instead of being retried. Idle connections still close after
|
||||
# keep-alive (75s).
|
||||
http-snippet: "keepalive_time 12h;"
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
|
||||
Reference in new issue
Block a user