<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Backports #3887 to `release-1.6`. The fix reached `main` only: `v1.6.3`
and `v1.6.4` both still ship the bootstrap-token Job as a
`post-install,post-upgrade` Helm hook, and `git tag --contains` on the
merge commit returns `v1.7.0-alpha.1` and nothing from the 1.6 line.
Tracked as #4539.
#3875 describes the deadlock this removes: the chart upgrade waits for
`MachineDeployment` and the child HelmReleases, workers cannot register
without the `bootstrap-token-<id>` Secret in the guest cluster, and the
hook that creates that Secret only runs once the wait has already
succeeded. The wait cannot succeed without nodes, so the upgrade times
out and Flux retries the same cycle indefinitely.
It is deterministic rather than flaky, and it takes out every tenant
Kubernetes cluster on the platform at once. What makes the backport
worth filing separately: anyone hitting this on 1.6.3 finds #3875 closed
in the tracker, upgrades to 1.6.4 expecting the fix, and lands in
exactly the same state.
Observed on a production cluster before the manual workaround: machines
stuck in `Provisioned` with no `NODENAME`, VMIs `Running` with IPs
assigned, guest clusters reporting `No resources found` for nodes, and
no bootstrap-token Secret anywhere in the guest `kube-system`. Creating
the Secret by hand brought nodes up within 90 seconds on each affected
tenant.
`git cherry-pick --mainline 1` of the merge commit applied cleanly with
no conflicts, and nothing was changed on top: the diff here is identical
to what merged on `main`.
Validation, `helm unittest` in `packages/apps/kubernetes` including
`tests/talos_templates_test.yaml` which this change extends:
```
Charts: 1 passed, 1 total
Test Suites: 19 passed, 19 total
Tests: 189 passed, 189 total
```
### Screenshots
Not applicable, no UI changes.
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix(kubernetes): create the tenant bootstrap-token Secret outside the Helm hook, so Talos workers can register during a chart upgrade instead of deadlocking against the wait that needs them
```
## What this PR does
Backport of #4612 to `release-1.6`, together with its follow-up #4789.
All three commits are cherry-picked. Two of them needed their
`ApplicationDetailPage.test.tsx` cases added by hand, because on this
branch that file has none of the virtual machine tab tests they sit next
to on main.
FoundationDB keeps its connection string in a ConfigMap that tenants may
read, and the console had no view for ConfigMaps. The application page
gets a ConfigMaps tab for the ConfigMaps named in the application's
resource map, with a copy button for each value. Each read uses an
exact-name field selector, so no general ConfigMap access is needed. The
FoundationDB chart now renders its resource map with monitoring disabled
too, and tenant roles may read it. If the resource map stops being
readable after a successful read, the tab goes away instead of keeping
the old names.
### Validation
- All 6 FoundationDB chart tests pass, and so do all 338 console tests
in 52 files and the TypeScript check.
- Tests cover finding the ConfigMaps through the resource map, reading
only the named ones in the active namespace, copying a value, a failed
copy, a denied read, malformed resource maps, and a resource map that
answers 403 or 404 after a successful read. The four tests for the last
case fail without the #4789 commit.
### Screenshots
The tab is the same as in #4612, which has a screenshot of it.
### Downstream repositories
Walked the trigger map against the diff. It changes a FoundationDB chart
template and its test, and the console, with no values, schema, CRD or
`hack/` file, and the console is vendored in this repository. No
companion repository needs an update.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(dashboard): backport FoundationDB connection ConfigMaps to release-1.6
```
## What this PR does
backport of #4779 to `release-1.6`. the template change is cherry-picked
unchanged from a6cf0e5ce. one test needed an edit for this branch, see
below.
the bug is the same here, and v1.6.4 ships this exact template. with
`bootstrap.enabled: true` the chart renders a
`PerconaServerMongoDBRestore` without `spec.backupSource.s3.bucket`,
which the operator crd requires, so the api server rejects it and the
release fails. the restore now takes the bucket and prefix from the same
helpers as the backup storage, so it reads from where the backups are
written. both places quote the values, the backup storage still leaves
out an empty prefix. a `destinationPath` that names no bucket fails at
render time.
this branch has no `backup.insecureSkipTLSVerify`. the backup storage
and the restore both hardcode `false`, so
`tests/restore-storage_test.yaml` checks `false` on both sides here
instead of setting the value to `true`.
- `helm unittest` passes 109 of 109 tests in 8 suites. before this
change it was 101 in 7.
- against the release-1.6 templates 7 of the 8 new cases fail, the same
7 as on main.
- `helm template` of the `PerconaServerMongoDB` gives the same picture
as on main for 8 `destinationPath` forms. 5 render the same storage, the
two bucket root forms only stop rendering a `prefix: null` the api
server prunes anyway, and `s3://2024/10` goes from ints the crd rejects
to strings.
- offline validation against the crd on this branch matches main. all 8
forms fail with `spec.backupSource.s3.bucket: Required value` before the
fix, and after it 7 pass and `s3://` stops at render time.
- `make generate` with cozyvalues-gen v1.6.0, the version this branch's
ci pins, leaves every generated file unchanged.
- `packages/apps/mongodb/Makefile` has no `test` target on this branch,
so ci does not run these suites here. i ran them by hand with
helm-unittest 1.1.2.
- no restore was run against a live cluster.
on upgrade, releases with `bootstrap.enabled: false` render the same
manifests as before. a release that still has `bootstrap.enabled: true`
creates its restore on the first reconcile after the upgrade, and the
operator starts restoring into the cluster as it is at that moment. the
release note says so.
### Screenshots
not a ui change.
### Downstream repositories
same diff as #4779, where the trigger map was walked file by file. no
entry matches.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(mongodb): backport the bootstrap restore fix to 1.6. a mongodb app created with bootstrap.enabled now gets a PerconaServerMongoDBRestore with the s3 bucket and prefix of backup.destinationPath, so the api server accepts it and the operator looks for the backup where the backup storage wrote it.
on upgrade, an app that still has bootstrap.enabled set to true, for example after an earlier restore attempt that failed on the missing bucket, creates its restore on the first reconcile, and the operator restores the backup into the cluster as it is at that moment. set bootstrap.enabled to false before the upgrade on every mongodb app that should not be restored.
```
## What this PR does
Backport of #4724 to `release-1.6`, including the memory-request cap
required by CloudNativePG when memory is overcommitted.
The chart renders no `shared_buffers`, and CloudNativePG 1.28.2, the
operator version on this branch, does not set one either, so PostgreSQL
starts with its built-in 128MB. That is the whole memory limit on
`t1.nano` and half of it on `t1.micro`, the chart's default preset, and
a dump, index scans over a large table or a replica catching up after a
failover can fill the pool and get the instance OOMKilled. Below a 512Mi
memory limit the chart defaults `shared_buffers` to a quarter of the
limit with a 16MB floor, capped at the rendered memory request rounded
down to whole MiB. From 512Mi up nothing is rendered and those clusters
are unchanged. An explicit `shared_buffers` in `postgresql.parameters`
still wins.
On upgrade every cluster below 512Mi that does not set `shared_buffers`
gets the new value. With the CloudNativePG defaults the chart keeps,
`unsupervised` and `restart`, the operator restarts the replicas first
and then the primary in place, so writes pause for one PostgreSQL
restart, and a single-instance cluster is unavailable for that time.
### Validation
- The commit applies cleanly to `release-1.6` at `da98ddc08`.
- `helm unittest` passes all 50 tests in 7 suites on this branch. The
added cases cover memory overcommit, fractional MiB requests, a request
below the 16MB floor, explicit values and the unchanged 512Mi threshold.
Four cases fail against the previous PR template.
- `helm template` of the chart on this branch renders `t1.small` exactly
as before, and the default preset differs only by `shared_buffers:
"64MB"`.
- `make generate` with cozyvalues-gen v1.6.0, the version this branch's
CI pins, leaves the generated files unchanged.
### Screenshots
Not a UI change.
### Downstream repositories
Same diff as #4724, where the trigger map was walked file by file. No
entry matches.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(postgres): backport the small-preset shared_buffers default to 1.6. Without an explicit shared_buffers and below a 512Mi memory limit, PostgreSQL now gets a quarter of the limit with a 16MB floor, capped at the memory request, instead of its built-in 128MB, so a full buffer pool no longer takes the whole limit on t1.nano or half of it on t1.micro. Affected clusters restart their instances once on upgrade, the primary last.
```
## What this PR does
Backport of #4675 to `release-1.6`, cherry-picked without changes from
`15b7092f3ada7c26fbaa7c67be26e59ced73992f`. #4675 is merged in main now.
With `users` configured and `jetstream.enabled: true`, the application
chart enables JetStream on the server but generates account `A` without
it. Authentication and ordinary NATS messaging work, while
`$JS.API.INFO` returns error 10039, "JetStream not enabled for account".
The generated account now gets `jetstream: true` when persistent
messaging is enabled and the account does not set JetStream itself.
Tenant-supplied limits, explicit disablement, other accounts and the
existing generated-user merge are kept. nats-server matches config keys
case-insensitively and walks the account map in no fixed order, so a
`JetStream` key in any casing counts as set. Otherwise the account would
carry both keys and the server would apply either one, changing from
restart to restart.
Existing instances with `users` pick this up on upgrade. The rendered
NATS config changes, so the StatefulSet rolls its pods once, and account
`A` gets JetStream without account limits unless
`config.merge.accounts.A.jetstream` sets them.
### Validation
- The commit applies cleanly to `release-1.6` at `da98ddc08`.
- `helm unittest` passes all 65 application-chart tests on this branch.
- The run on NATS Server 2.11.10 is described in #4675.
### Screenshots
Not a UI change.
### Downstream repositories
The diff changes an application template and chart tests only. Public
values, schemas, output resource names and tooling contracts are
unchanged. The website NATS documentation contains no account-level
workaround to remove. No downstream trigger applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(nats): enable JetStream on the account generated for users when jetstream.enabled is true, keeping any explicit account-level JetStream setting. Instances with users roll their NATS pods once on upgrade.
```
With the static seal the key sits in a Secret in the same cluster as the
data it unwraps, so the barrier is only as strong as access to Secrets
in that namespace, to etcd and to backups. The README now says so before
the reader picks a seal, and it says what the key identifier is for,
since OpenBAO picks the key for stored data by it.
Tenant roles grant nothing on Secrets and no pods/log, so a cluster
administrator creates the key Secret and reads the log line that
confirms a re-wrap.
The StatefulSet uses the OnDelete strategy, so neither a seal migration
nor the apiserver egress label reaches a running pod until it is
replaced. The README describes the migration from Shamir step by step,
following the OpenBAO seal migration guide, and says that pods created
before the label have to be replaced once.
The migration starts with a backup. The standalone file backend is not
transactional, so its volume is copied with the server stopped or
snapshotted atomically. The upgrade that sets allowMigration is the
point of no return for the chart, and the README says how a cluster
administrator backs out of it before any pod is replaced.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 3ea7a95108)
The upstream chart passes the server config through tpl, and its start
script replaces upper case placeholders such as HOSTNAME anywhere in the
copied file. A key identifier with template delimiters would run as a
template inside helm-controller, which runs as cluster-admin, and an
upper case one could come out different on every pod. Both key
identifiers now take lowercase alphanumerics, hyphens, dots and
underscores.
The Secret names become pod volume names of the form userconfig-<name>,
which Kubernetes validates as DNS labels. Both name fields now take a
DNS label of at most 52 characters, and the volume entries are quoted.
The apps API only defaults against the generated schema, so Helm
enforces these patterns when it installs or upgrades the release.
A previous Secret equal to the current one mounts the same volume twice,
which the API server rejects for the StatefulSet, and OpenBAO refuses to
start when equal key identifiers name different key material. Both
cases now fail at render. The deepcopy functions for the Seal type are
regenerated as well.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit bfbb5ac75f)
The OpenBAO app has no auto-unseal (#2787). Every restart of a server,
after a node drain, an image bump or an OOM kill, leaves it sealed until
an operator enters the Shamir key shares into each replica. In HA mode
the pods also hang at start (#2793). Cozystack's allow-to-apiserver
policy only admits egress from pods labelled
policy.cozystack.io/allow-to-apiserver, the app never set that label,
and service_registration "kubernetes" cannot reach the API server
without it.
A new seal value selects shamir, which stays the default, or OpenBAO's
seal "static". With static the chart renders the static seal stanza into
the standalone and the Raft config and mounts the key Secret named in
seal.secretName through the upstream chart's server.extraVolumes. The
chart never generates, stores or rotates the key, because a missed
lookup would replace it and leave the data unreadable.
seal.previousSecretName and seal.previousKeyId carry the n-1 key that
the static seal accepts during a rotation. The server pods always carry
the apiserver egress label.
Changing the seal of an initialized instance is a seal migration that
OpenBAO performs under bao operator unseal -migrate. The chart reads the
seal in use from the config ConfigMap the previous release rendered and
refuses a different seal.type unless seal.allowMigration acknowledges
the migration. Moving away from static needs the old stanza kept with
disabled = "true", which the chart does not render, so that direction is
refused regardless. The README documents the key Secret, the rotation
with the pod replacement that the OnDelete update strategy requires, and
the log line that confirms the re-wrap.
Signed-off-by: txmazing <txmazing@gmail.com>
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 0c52dedbe2)
CloudNativePG rejects clusters whose shared_buffers exceeds the memory
request. Memory overcommit can put that request below both a quarter of
the limit and the 16MB floor.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 7af129b6a1)
The chart renders no shared_buffers, so PostgreSQL starts with its 128MB
default whatever the container gets. A full pool is the whole memory
limit on t1.nano and half of it on t1.micro, the default preset. The
pool is shared memory that the kernel cannot reclaim without swap, so
when a dump, index scans over a large table or a replica catching up
after a failover fill it, the instance is OOMKilled.
Instances below a 512Mi memory limit now default to a quarter of the
limit, at least 16MB. From 512Mi up a quarter is at least 128MB, so
those instances render exactly as before. shared_buffers needs a
restart, so on upgrade CloudNativePG restarts every instance below
512Mi that does not set it explicitly.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 0b39d7ea89)
With bootstrap.enabled the chart rendered a PerconaServerMongoDBRestore
without backupSource.s3.bucket. The operator CRD requires that field, so
the API server rejected the restore and the HelmRelease failed. The prefix
was missing too, and it matters just as much. For a backupSource restore
the operator builds the PBM storage from backupSource.s3 alone and takes
only the backup name from backupSource.destination, so a restore without
the prefix would look for the backup at the bucket root.
The restore now takes the bucket and prefix from the same helpers that
fill the backup storage, so it reads from where the backups are written.
Both places quote the two values, so a numeric bucket or prefix stays a
string instead of making the resource invalid. The backup storage still
leaves out an empty prefix, so running releases keep the same object. A
destinationPath that names no bucket fails at render time instead of
reaching the operator with an empty bucket.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit a6cf0e5ce4)
The resource map is the only object that names the connection ConfigMap
for a client such as the dashboard. It was rendered only with monitoring
enabled, and the tenant role could not read it, so a tenant session had
no way to find the ConfigMap it is allowed to read.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit e3ae8033e6)
With users configured and jetstream.enabled true, the chart enables
JetStream on the server but generates account A without it, so every
JetStream call from those users fails with error 10039, "JetStream not
enabled for account". The generated account now gets jetstream: true
unless it already sets JetStream itself. nats-server matches config keys
case-insensitively and walks the account map in no fixed order, so a
JetStream key in any casing counts as set. Otherwise the account would
carry both keys and the server would apply either one, changing from
restart to restart.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 15b7092f3a)
The render now fails when whitelistHTTP is set with an empty whitelist,
so the values description tells users that before they hit the error.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit eee0596661)
The accept and reject assertions matched anywhere in haproxy.cfg, so a
frontend that lost its reject rule stayed green as long as another
frontend still carried one. Without reject a non-matching connection
falls through to HAProxy's default accept, so that frontend is open to
everyone. Each assertion now anchors the acl, accept and reject lines to
the frontend they belong to.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 6855bfb465)
haproxy:latest now resolves to 3.3 and newer, where a backend named like
a frontend is a fatal config error, so every tcp-balancer pod fails at
config parse. Backends get a _backend suffix and the image follows the
chart appVersion on the 3.4 LTS line instead of a floating tag.
The whitelist rules render inside one guard. On the kubernetes and talos
frontends an empty whitelist means no filtering; whitelistHTTP with an
empty whitelist fails the render, because a security opt-in that quietly
became a no-op would not be noticed the way a crash is.
The unittest suite pins the rendered config for the default, whitelist
and failure cases and the image pin, so a template refactor cannot bring
the collision or the floating tag back.
Fixes#1966
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 12cc48d465)
The talos-reconcile Job runs under a 256Mi limit and applies its
TalosConfigTemplate client-side. Building the merge patch means pulling
the cluster's full OpenAPI schema down, which on a management cluster
carrying a few hundred CRDs costs around 290Mi of RSS, so the kernel
OOMKills kubectl before the Job can finish.
Only the merge path pays that cost. The first run creates the object and
stays cheap, so a fresh install succeeds and the Job then
CrashLoopBackOffs on every reconcile afterwards, which means on every
chart upgrade. Measured against a 438-CRD cluster with the Job's own
kubectl (alpine/k8s:1.33.4), cold discovery cache:
get -o jsonpath 52Mi
apply, object absent (create) 52Mi
apply, object present (merge) 292Mi <- OOMKilled at 256Mi
apply --server-side 56Mi
--validate=false changes nothing, so the cost is the merge strategy
lookup rather than validation.
Server-side apply leaves the merge to the apiserver and the client never
needs the schema. --force-conflicts keeps the Job authoritative over an
object an earlier client-side apply created.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
(cherry picked from commit 1728b49e0c)
`kubectl api-resources --api-group=<absent group>` exits 0 and prints an
empty list, so guarding on the exit code always fell through. The delete
then failed with `the server doesn't have a resource type "keycloakclient"`
— `--ignore-not-found` covers a missing object, not an unknown type.
Because the oidc-bootstrap Job is a post-install/post-upgrade Helm hook,
that failure blocked the Helm upgrade of every `kubernetes-<cluster>`
release on any cluster without the EDP Keycloak operator CRDs, which is
the default configuration (`oidc.mode: None`).
Capture the discovery output once and gate each delete on its own
resource type, matching the probe idiom already used by the platform
migrations and the ouroboros cleanup hook.
Fixes#3516
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
(cherry picked from commit 06400ac033)
Backport of #3887 to release-1.6. The fix reached main only, so every 1.6 release still ships the Job as a post-install/post-upgrade hook and hits the deadlock #3875 describes: the chart upgrade waits for MachineDeployment and the child HelmReleases, workers cannot register without the bootstrap-token Secret, and the hook that creates that Secret only runs once the wait has already succeeded. The wait cannot succeed without nodes, so the upgrade times out and Flux retries the same cycle indefinitely.
This is deterministic rather than flaky, and it takes out every tenant Kubernetes cluster on the platform at once. Anyone reading the tracker finds #3875 closed, upgrades to 1.6.4 expecting the fix, and lands in the same state, which is why this backport matters more than the usual one.
(cherry picked from commit adc9c568fd)
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The policies are right and nothing here changes them. What is missing is
the reason, and this file has now been changed in both directions inside
a week by people reasoning from the annotations alone, so the reason is
the part worth keeping.
The Job carries no hook-succeeded because the script exits 0 on the
degraded path as well as the clean one, and its log is the only record
that a finalizer outlived the wait; ttlSecondsAfterFinished reaps it on a
timescale a human can act within. The ServiceAccount, Role and
RoleBinding have no such clock, and before-hook-creation fires on a next
activation that after the last uninstall never comes, so without a
terminal policy they sit in the tenant namespace for the life of the
cluster still granting delete over every KafkaTopic in it.
Reaping them cannot strand a running pod, which is the objection the
symmetrical shape was written against: helm applies hook-succeeded only
once the hook is terminal, and activeDeadlineSeconds ends the Job inside
the uninstall hook timeout. hook-failed would decide nothing, because a
failing hook already reaps the earlier successful ones under
hook-succeeded.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 3a43bcf391)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A non-zero --request-timeout makes kubectl skip its in-cluster credentials
altogether. client-go consults them only while the assembled config still
equals the default, and the flag lands in that config, so the branch that
would load the ServiceAccount token never runs. kubectl falls back to its
built-in default cluster, localhost:8080, is refused, and the script reads
the refusal as a failed delete and exits 1. A non-zero pre-delete hook
returns from Uninstall.Run before any resource is removed, so no Kafka
release could finish deleting and helm-controller retried forever.
Measured on the pinned image rather than argued from the docs: with the
token and ca.crt mounted and KUBERNETES_SERVICE_HOST set, the same delete
reaches the cluster without the flag, goes to localhost:8080 with
--request-timeout=20s, goes to localhost:8080 with 1m, and reaches the
cluster again with 0.
The three in-tree hooks that work do not pass it. apps/kubernetes names it
in a comment only, apps/harbor and apps/mariadb never mention it, and
apps/tenant records that its DELETE calls "carry kubectl's default
--request-timeout of 0". The watch stays bounded by --timeout, and the Job
by activeDeadlineSeconds, which is that same shape.
The suite now rejects the flag. It needs the "=" because the explanation
in the script names the flag too, and matching the bare name would fail on
the comment rather than on a use.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit e63b8595d3)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Job carries no hook-succeeded, so it outlives the hook run and
ttlSecondsAfterFinished reclaims it, which is what leaves a readable
record when the finalizer outlives the wait. Its ServiceAccount, Role and
RoleBinding still carried hook-succeeded and hook-failed, so Helm reaped
them at the end of the run either way and left the Job behind them.
A Job whose ServiceAccount is gone still starts a pod. kubelet mounts the
projected volume without a token, client-go finds no in-cluster config,
kubectl falls back to localhost:8080, and the script takes its catch-all
branch and exits 1. A non-zero pre-delete hook returns from Uninstall.Run
before any resource goes, so the release never finishes deleting and
helm-controller retries it: the e2e run on the merge commit shows six
such pods in five minutes, in two namespaces, every one of them dead the
same way. The hook meant to prevent one stuck Kafka release was instead
stopping all of them.
Give all four objects the same before-hook-creation policy, so the
credentials live exactly as long as the consumer and the next run
replaces them. Both other in-tree hooks that reach the apiserver,
apps/kubernetes and apps/harbor, are symmetric this way already.
The suite now pins the policy on each of the three, because nothing
compared them: the asymmetry rendered cleanly, passed 37 helm-unittest
cases and 5 bats cases, and only a cluster showed it.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 64d5689434)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Both suites matched "strimzi.io/cluster=kafka-test" as a bare substring,
and that is a prefix of the sibling release kafka-test2 and of
kafka-test-topics. Rendering the hook with either selector left both
suites green, so neither pinned the scoping they exist to pin. Anchor on
the leading "-l " and on the trailing delimiter; both variants now red in
both suites, and the six earlier hook mutations still red exactly where
they did.
Drop the notMatchRegex on a leading slash. It cannot fail on its own: the
anchored image pattern above rejects a leading slash first, and an empty
_cluster.images-registry renders no slash to begin with. Its rationale
moves into that pattern's comment.
Record why activeDeadlineSeconds and the 180s wait have to move together.
A deadline that fires mid-wait kills the Job, and with backoffLimit: 0 the
hook then returns non-zero, which is the wedged uninstall the deliberate
exit 0 on the timeout branch exists to avoid.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 637098f222)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The suite used bats' run helper. make bats-unit-tests does not invoke the bats
binary: it runs hack/cozytest.sh over every hack/*.bats, and that runner
implements @test and little else, so the file died at 'run: command not found'
and took make unit-tests with it. Under bats it was 5/5, which is where the 5/5
in the previous commit came from. Save $? by hand instead, the shape
app-cleanup-hooks_test.bats already uses; verified under both runners.
The fake kubectl no longer defaults DELETE_RC to 0. An unset value made every
failure-path assertion vacuous; now it exits 64 and says so. Cover both
RESTMapper wordings for an absent CRD rather than just the first.
Widen the image pattern to v[0-9]+\.[0-9]+(\.[0-9]+)? like harbor and mariadb:
pinning v1\.32 reintroduced the breakage the digest was made pattern-based to
avoid, one Renovate bump later. Assert the _cluster.images-registry override
and the no-leading-slash default, neither of which was covered: the hook is the
one image in this chart whose pull failure surfaces at uninstall.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 415aa82409)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The hook's exit code is the whole contract: a non-zero one returns from
Uninstall.Run before any release resource goes, keeping the topic operator
alive; a zero one lets Helm remove it. delete_test.yaml matches manifest text,
so it stayed green when the timeout branch was put back to exit 1, when the
CRD-absent branch was, and when the selector was widened past this release.
Add a bats suite in the shape hack/app-cleanup-hooks_test.bats already uses for
qdrant and harbor: render the script, run it against a fake kubectl, pin the
exit code of each branch. Each of the three mutations above now turns it red.
Drop hook-succeeded from the delete policy and reclaim the Job with
ttlSecondsAfterFinished instead. The script exits 0 on the degraded path too,
where topics still hold the finalizer, so deleting on success removed the only
record of that seconds after it was written. Assert both in delete_test.yaml.
Pin the hook image by shape rather than by literal digest: Renovate rewrites
digests under packages/*/templates/hooks/ and nothing rewrites tests/, so the
literal turns the next bump red for no defect.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 3326e0fb9e)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Manual backport of #3936 to `release-1.6`.
The automated backport, #4372, pushed conflict markers in
`packages/apps/rabbitmq/README.md` and
`packages/system/rabbitmq-rd/cozyrds/rabbitmq.yaml` as its only commit,
so it would ship them if merged. Both are generated files, and the
conflict is the `tls` block that landed on main after 1.6 was cut.
Everything else applies unchanged, so two differences follow:
- the README parameter table and the cozyrd schema are regenerated from
the 1.6 values with cozyvalues-gen v1.6.0, so they carry the new default
and descriptions without `tls`;
- the 1.6 rabbitmq `Makefile` has no `test` target, so the helm-unittest
sweep would never enter the package and the new preset suite would not
run. The second commit adds the same two lines main already has.
Verified on this branch: the rabbitmq suite passes 4/4, `make -n test`
answers the sweep probe, and a second `make generate` produces no drift.
Change summary, unchanged from #3936: the default `resourcesPreset`
moves from `t1.nano` (128Mi) to `s1.nano` (250m CPU, 512Mi), because a
default-sized RabbitMQ 4.2 broker was OOMKilled on `t1.nano` and never
became Ready. The `resources` and `resourcesPreset` descriptions say the
preset fills every resource `resources` leaves unset.
Close#4372 in favour of this one, or tell me to take a different route
and I will.
### Screenshots
Not a UI change.
### Downstream repositories
Walked the trigger map against the diff: one app chart with a changed
default. The provider follow-up is
cozystack/terraform-provider-cozystack#34, labelled `target/1.7`.
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#34
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(rabbitmq): raise the default resources preset from t1.nano to s1.nano so a default-sized broker is not OOMKilled. Existing brokers that never set resourcesPreset move to s1.nano on upgrade and roll their pods.
```
With watch granted, kubectl delete waits for the claims to be gone, and
its default for that wait is a week. A claim held by a pod that will not
stop, or a stuck detach, would hold the post-delete hook until
helm-controller's 300s uninstall budget ran out and leave the
HelmRelease stuck uninstalling.
The PVC deletes in the clickhouse, harbor, mariadb and qdrant hooks now
wait 120s at most and report the claims still terminating by name, and
each Job carries activeDeadlineSeconds 270 with a 10s termination grace
period, the numbers packages/apps/tenant/templates/cleanup-job.yaml
documents against that budget, so a kubectl that never returns cannot
hold the hook either.
The mariadb Role no longer grants watch on the named Secrets: they carry
no finalizer, so the delete never reaches the watch, and a resourceNames
rule does not authorize a collection watch anyway.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
(cherry picked from commit f886f26ad4)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
kubectl delete waits for the object to disappear unless told otherwise,
and that wait is a watch. The post-delete hook Roles of harbor,
clickhouse, mariadb and qdrant granted get, list and delete only, so
every uninstall logged a forbidden watch, the wait was skipped and the
hook reported success before the PVCs were actually gone.
The rules now carry watch for every resource the hooks delete, including
the named Secrets the mariadb hook removes.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 711fcc168c)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
The 1.6 rabbitmq Makefile has no test target, so the helm-unittest sweep
never enters the package and the preset suite would sit there unrun.
main gained the target earlier; this brings the same two lines.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
A default-sized RabbitMQ 4.2 broker was OOMKilled on t1.nano (128Mi)
and never became Ready; its working set peaked at 125.8Mi. s1.nano keeps
the 250m CPU and gives 512Mi. The resources and resourcesPreset
descriptions now say the preset fills every resource that resources
leaves unset. The parameter table and the cozyrd schema are regenerated
from the 1.6 values, which carry no tls block.
(cherry picked from commit 12e64a99cd)
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
kubectl delete waits for the claims to be gone unless told otherwise,
and OpenSearch data pods stop slowly, so the post-delete hook could
outlive helm-controller's 300s uninstall budget on a claim that would
not release and leave the HelmRelease stuck uninstalling.
The delete now runs with --wait=false, the shape the monitoring, etcd
and seaweedfs hooks use: the deletion is requested and the claims go
once their pods have stopped, which the log now says instead of
"complete". The Job carries activeDeadlineSeconds 120 as the backstop
for a kubectl that never returns and keeps only before-hook-creation,
so a failed run and its log survive. The Role drops watch, which
nothing needs once the wait is gone.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
(cherry picked from commit 54ac8ed135)
The OpenSearch operator creates its data volumes through a StatefulSet,
so the PVCs carry no ownerReference, and the chart had no cleanup hook.
Removing the application left every data volume Bound and counting
against the tenant quota, unlike clickhouse, kafka and harbor, whose
post-delete hooks reclaim the operator-owned storage.
A post-delete hook in the same shape as the clickhouse one now deletes
the PVCs labelled opster.io/opensearch-cluster=<release>, with a Role
limited to persistentvolumeclaims and the kubectl image pinned by digest.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 13be303fbb)
cozystack-ci[bot]