admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-09 22:13:18 +09:00
Code Issues Packages Projects Releases Wiki Activity
7,834 Commits 571 Branches 294 Tags
4e5b76f2820cf805ced8ced4010fe1883ccd7d70
Commit Graph
179 Commits
Author SHA1 Message Date
Aleksei Sviridkin 9e8e10c307 fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230) 
## What this PR does

Refs #5205.

The S3 endpoint the platform backups use is copied twice, and neither
copy is refreshed after `publishing.host` changes. The COSI sidecar
writes `BucketInfo` once per `BucketAccess`, so the credentials Secret
`bucket-cozy-backups-system-credentials` rendered from it keeps the old
endpoint. The Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of `backupstrategy-controller` saw. The default
objects gate only acted on missing objects, so backups kept going to the
old host and nothing reported it.

The gate now also forces a render when a copy is stale:

- the bucket's `<bucket>-system` release, when the credentials Secret's
endpoint differs from the COSI Secret it is rendered from;
- `backupstrategy-controller`, when the endpoint an object was rendered
with differs from the credentials Secret's. The chart stamps the
rendered endpoint on the BSL and on every default Strategy as
`backups.cozystack.io/endpoint`, because the endpoint itself sits at a
different path in every kind. Objects without the annotation count as
stale until a render stamps it; `MinForceInterval` bounds the forces.

Stale credentials are forced before the BackupClass and the routed
objects are read, so an absent BackupClass or an unreadable Strategy
still fails the check but does not keep the old endpoint in the
credentials Secret. A `BucketInfo` without an endpoint is not compared,
since rendering from it would write an empty endpoint or fail the bucket
release. Repairing missing objects keeps working while the credentials
comparison is stale or fails, and a `systemSecretName` that is not
derived from a COSI Secret (no `-credentials` suffix, or the raw COSI
Secret itself) is not compared at all. RBAC is unchanged: the controller
already holds cluster-wide `get` on Secrets. No new metric: nothing
would consume it.

What this does not do: the COSI Secret still holds the endpoint the
`BucketAccess` was granted with, so after a host change the endpoint
follows only once an operator deletes the `BucketAccess` and forces
`bucket-cozy-backups` to recreate it (this issues new S3 keys). From
there the gate replaces the remaining manual steps of the recovery
described in the issue. Automatic re-grant is left for a separate issue,
and the operator runbook goes to the website, hence `Refs` rather than
`Fixes`.

Known limitation: the credentials comparison runs only when
`backupStorage.systemSecretName` is
`bucket-<bucketName>-system-credentials`, the Secret the forced
`bucket-<bucketName>-system` release renders. If an operator sets
`bucketName` and `systemSecretName` inconsistently, the comparison is
switched off rather than forcing a release that never rewrites the
compared Secret, so a stale endpoint in that Secret is not repaired.
Repairing a missing credentials Secret still forces the release derived
from `bucketName`, as before this PR.

Tested with unit tests for the gate (including the comparison
normalisation shared with the chart helpers), helm-unittest for the
annotation and the wiring, and a bats guard that every rendered Strategy
CR and the BSL carry the annotation. On a live cluster, a server-side
render of the chart produced exactly the annotation the gate derives
from the real credentials Secret, so the steady state does not force
anything.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: an operator runbook for changing `publishing.host` with
platform backups enabled (re-grant the `BucketAccess`, then let the gate
converge; restart ClickHouse pods that read the credentials from
environment variables). Not filed yet, pending approval of the text.
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup strategies and storage locations now record their storage
endpoint, enabling detection and repair of outdated endpoint settings
after bucket access changes.
* Bucket credentials and backup objects are refreshed independently when
their endpoints are missing or out of date.
* **Documentation**
* Clarified when endpoint changes trigger updates and how suspended
releases are handled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-11-09 12:02:42 +03:00
myasnikovdaniil 7097cc5cda feat(release): make backports automatic with backport-batch (#5106) 
## What this PR does

Backports become automatic. Fix merged to `main` with `fix` title, or
`security` for fix that comes through security advisory, reaches release
lines through batch that runs on Monday and Thursday, newest line first,
nobody requests, labels or opens backports by hand. Batch runs for two
newest lines with stable release (`released-lines` in policy block) and
every line branched after them that has no stable release yet: with 1.6
and 1.7 released, `release-1.8` created at `v1.8.0-rc.1` joins them, and
`v1.8.0` drops 1.6. Line counts as released once `vX.Y.Z` tag without
pre-release suffix exists on origin. Rules are in
`docs/release/backports.md`, `cmd/backport-batch` applies them,
`.github/workflows/backport-batch.yaml` runs it. Design is in
cozystack/community#114, merge shape of each original in
cozystack/community#119.

- `backport-batch run --line release-X.Y` takes PRs merged to `main`
since line branched (title and body read from merge commit) and skips
what rules exclude: other types and `deps`, breaking changes, platform
migrations, `targetVersion`, changed CRD documents, API changes that API
review gate calls sizeable (batch runs `internal/apigate` itself
whenever PR changes a file the gate parses:
`pkg/apiserver/apiserver.go`, an ApplicationDefinition, or YAML under
`packages/` or `internal/` naming `CustomResourceDefinition` or
`APIService`), and on older line what next newer line doesn't carry yet.
The rest goes onto `backport-batch/<line>-<date>` in merge order. PR
merged to `main` with merge commit goes as its own commits picked with
`cherry-pick -x --signoff`, so each keeps its author, message and
sign-offs and gets sign-off of whoever runs the batch (with `-m 1`
author was whoever merged on `main`), merges of `main` into PR branch
are left out, and merge commit joins them to the batch with original's
merge message and `Backported-from:` trailer. Squash-merged PR goes as
one pick. Conflict on any commit takes branch back to where it was
before that original, it's listed with the commit that failed, run goes
on. `--push` opens or updates draft PR, `--continue` carries on open
batch PR as fast-forward only, PR body it didn't write is never
rewritten. Same-day run over batch branch whose PR never got opened
refuses and names `run --line <line> --onto <head> --push`, which
continues it. Run or `--defer` that resets local batch branch over
commits only it has prints `git branch` command that restores them.
- `run --defer <N> --reason <why>` takes original out of open batch
(usually the one failing CI is pinned on). It rebuilds branch replaying
batch unit by unit without it, so merges and pushed conflict resolutions
keep their shape, and only this rebuild pushes with
`--force-with-lease`. Deferred original comes back in next batch.
- Line carries something only if its own history shows it: picks whose
`-x` line, the one git appends, names PR's merge commit or every commit
of its own, or pick that changes nothing. Picked commits keep their
messages and batch merge carries PR body, so text in them (`-x` lines,
`Backported-from:`, `Backport of #N`) counts only on commits from before
`batch-only` day in policy block, where it records old hand backports
and batches. Otherwise text in a merged fix could mark another fix as
already backported, and it would never reach the line. PR against line
never counts. Open batch PR counts only from this repo and from author
with write access or the batch's own bot (`BACKPORT_BATCH_BOT`, exact
login, `cozystack-ci[bot]` when unset, so maintainer's run by hand
continues scheduled batch instead of opening second one), and is read
the same way. Actions bot isn't trusted anymore, any workflow token
writes as it.
- Any failed GitHub API or git call stops the run with message naming
the call, no retries. Failed permission lookup used to count batch PR
out as look-alike, so `--continue` opened second batch PR and `notify`
posted second comment; now `run`, `check`, `verify` and `notify` exit 2
and write nothing.
- Policy applies fully from `release-1.7`. `release-1.6` takes only
fixes merged from 2026-10-26, older ones need `backport/include`. Start
days are in `backport-policy` block on the policy page, test keeps page
and code in sync.
- `backport-batch check --cut` replaces `backport-audit` as gate before
patch cut and before every rc after the first, it also fails while a fix
is only on open batch PR. Plain `check`, which scheduled run uses, lets
those through. `backport-batch fold` checks conflict resolution in the
same shape: merge on batch head whose second parent is straight line of
picks of PR's own commits in order, each with its `-x`, only PR's files
(one pick for squash). `backport-batch verify` holds every original on
batch to that shape, clean picks included, and `pr-labeler.yaml` runs it
on every push to batch PR: `pull_request_target` so it reaches
`release-*`, tool built from `main`, batch head fetched as data,
read-only token. Batch PR body says per original how many commits it
brought and which picks differ from their commits. `backport-batch
notify` reports batch that is still open from previous run and deferred
conflicts in one comment on batch PR, edited in place, with
`BACKPORT_ALERT_TEAM` mention.
- `hack/backport-batch-scheduled.sh` is what the workflow runs and
maintainer can run by hand (`--dry-run` changes nothing on GitHub): for
each line newest first `run --continue --push`, `check`, `notify`. It
pushes with `BACKPORT_BATCH_TOKEN` only and fails without it, never
`GITHUB_TOKEN`, because its pushes don't start CI. Workflow runs it as
cozystack-ci App: it mints token for this repo only with Contents and
Pull requests write, passes it as `BACKPORT_BATCH_TOKEN`, commits as
`cozystack-ci[bot]`, and sets `BACKPORT_BATCH_BOT` so the tool trusts
that bot's batch PR and comment. Job timeout is 30 minutes, under the
token's hour.
- App has no workflows permission, so workflow runs the script with
`--no-workflow-files`: original that changes `.github/workflows/` isn't
picked and waits on batch PR for maintainer, listed with its workflow
files in body and in notify comment. It's never a conflict. `check`
warns about it while it waits and `check --cut` fails on it. Maintainer
brings it over with own token by `run --continue --push` without the
flag, or takes it out with `--defer`.
- Deferred conflicts and failed batch CI wait for maintainer. `notify`
keeps one comment on batch PR listing stalled batch, conflicts and
workflow changes, mentioning `BACKPORT_ALERT_TEAM` (team or login) when
it's set. Failed batch CI isn't in the comment, it shows on batch PR
checks, and `check --cut` fails while batch is unmerged.
- Two steps that need judgement (resolve conflict, pin failed CI on
original and defer it) are written as runbooks in
`docs/release/backport-tasks/`: inputs, the only allowed output, check
it must pass before push, what never to do.
- `backport/include` and `backport/skip` let maintainer override rules
for single PR, they are new `backport/` category in `labels.yml`. Skip
wins over include, if open batch already picked it `--defer` takes it
out.
- Text written by other people (titles, release notes, reasons, file
names) goes to batch PR body only inside code spans or release-note
block, control, bidi and zero-width characters are dropped, file names
in deferred block included, git args go after `--end-of-options`, no
shell.
- Removed: `cmd/backport-audit`, `backport.yaml`, `kind/backport` and
`kind/backport-previous` labels with aliases and stale exemption,
trailer-check exemption for bot branches. `Commit trailers` job exempts
only `backport-batch/*` branches of this repo, changelog instructions
read originals from batch commits. Exemption lives in main's
`pre-commit.yml`, and batch PR runs its line's copy: `release-1.7` has
its own copy that exempts only old `backport-<pr>-to-<branch>` branches,
`release-1.6` has no such job. So on `release-1.7` picked commit with
bad trailer fails batch CI until line gets main's copy.
- `hack/close-manual-backport.sh` runs from `pr-labeler.yaml` on
`opened` and closes PR against `release-*` from outside contributor
(association contributor, first-time contributor or none, permission
read or none). Owners, members, collaborators, anyone with triage or
more and bots are left alone. When GitHub's answer on author's
permission can't be read, it closes nothing and fails the job naming the
failed read. Maintainer keeps release-only fix by reopening it.
Release-only PRs are occasional and opened by maintainers, for fix with
no counterpart on `main`, or one that on `main` came inside another PR
or was written for `main`; if its counterpart on `main` is a fix the
batch would take, the batch skips it once picking it changes nothing on
the line, otherwise maintainer labels it `backport/skip`.
- `docs/release.md` is split into pages under `docs/release/`, PR
template doesn't ask for `kind/backport` anymore. `security` type is
added to commit format in `docs/agents/contributing.md`, PR template,
Gemini review guide and labeler's list (labeler maps no `kind/*` to it,
`labels.yml` has no fitting one).
- `SECURITY.md`, `docs/security/incident-response.md` and
`GENERAL_TECHNICAL_REVIEW.md` now say two latest minor releases are
supported, same window the batch runs for, and link the backport policy
for which fixes go there.

Dry runs without push, on 2026-11-06, with `--no-workflow-files` as the
workflow runs it: `lines` gives `release-1.7` and `release-1.6`. On
`release-1.7` batch picks 12 fixes, defers 3 conflicts (#4756, #4867,
#5082), leaves #4888 and #4971 to a maintainer because they change
workflows, and skips 49. On `release-1.6` it picks #5088, defers 3
conflicts (#5071, #5086, #5087), leaves #5142 to a maintainer, and skips
590. `verify` passes both batches.

Setup after merge: no new secret. Batch runs as cozystack-ci App from
existing `COZYSTACK_CI_APP_ID` and `COZYSTACK_CI_PRIVATE_KEY`, job fails
when either is missing. App's installation already has Contents and Pull
requests write on all org repos, and no Workflows is needed: PRs that
change `.github/workflows/` wait for maintainer, who brings them over
with own token as "Changes to workflows" in `docs/release/backports.md`
says. Today that's #4888 and #4971 on `release-1.7` and #5142 on
`release-1.6`. Set `BACKPORT_ALERT_TEAM` to a team or a login (for
example `cozystack/maintainers`): unset, batch comment mentions nobody
and nobody is notified. Once this PR is merged, `Backport batch shape`
can be required check in the release lines ruleset (`release-*`
excluding `release-*.*.*`), with "Do not require status checks on
creation" on, because `cut-prerelease.yaml` creates `release-X.Y` by
push. `PR Auto-Label` with both new jobs runs on `pull_request_target`,
so it needs #5135 sorted before 2026-12-02. `kind/backport*` labels have
to be deleted by hand, label sync doesn't delete.

### Screenshots

Not a UI change.

### Downstream repositories

cozystack/community `contributors/README.md` lists commit types, so it
needs `security` too, it has no backport label text. Website has no
backport label instructions, and its links to `docs/release.md` carry no
anchor, the page stays as index. Trigger map lists commit convention
change for cozystack/ccp: this adds `security` type, but not message
format or trailer, and `package-bump` writes only `chore(...)` commits
and restates the trailer, so nothing changes there.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up: add `security` to commit types in `contributors/README.md`

### Release note

```release-note
feat(release): backports are automatic. Fixes merged to main with fix or security title reach release lines through a batch built by cmd/backport-batch every Monday and Thursday, by the rules in docs/release/backports.md. The batch runs for the two newest lines with a stable release and for a line in its rc window: the release-X.Y branch the first rc creates joins at once, and its vX.Y.0 drops the oldest line. Features, chores, docs, dependency bumps, breaking changes, migrations, CRDs and API changes that the API review gate calls sizeable wait for the next minor. The policy applies fully from release-1.7, release-1.6 takes only fixes merged from 2026-10-26. Each original reaches the line as its own commits under a merge commit, keeping their authors and sign-offs. Maintainers override the rules for single PR with backport/include and backport/skip labels and take an original out of a batch with backport-batch run --defer. backport-batch check replaces backport-audit as the gate before a patch release and before a later rc. The label-driven backport workflow and kind/backport labels are removed. Backport PR opened by hand against a release branch by author with read access or less is closed automatically. The scheduled batch runs as the cozystack-ci App and needs no secret of its own; a fix that changes .github/workflows waits on the batch pull request for a maintainer, since the App may not push workflow changes. Release documentation now lives in one page per topic under docs/release/.
```
 2026-11-09 09:42:38 +05:00
Kirill Ilin dfd7cbed36 fix(operator): reject a Git cert secret before installing CRDs and Flux 
CRDs and Flux are installed before the platform source, so refusing a
cert secret on a Git source only when the source is written exits the
operator after a partial install. The check now runs right after flag
parsing, and the source install reuses it.

The guard runs on every start, so a Git source without a cert secret is
now covered over https and ssh: a guard that misfired there would
crash-loop the operator for every Git platform source. The update test
also renames the cert Secret, as a CA rotation does.

Assisted-by: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
 2026-11-06 18:28:45 +05:00
Myasnikov Daniil eaa0169ff3 chore(release): retire the backport bot, its labels and backport-audit 
The batch replaces the label-driven backport workflow and the audit
that gated a cut against it. The kind/backport labels, the stale
exemption for them, their contract tests and the audit's make target
go with them. labels.yml gains a backport/ category instead, with
backport/include and backport/skip, the overrides the batch reads off
a merged pull request.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-11-06 12:33:37 +00:00
Myasnikov Daniil 9f8c88b4f1 feat(release): build the backport batch with backport-batch 
The rules need a tool that applies them the same way on every run and
shows what it did. backport-batch reads main's first-parent history,
judges each pull request by the policy block it reads from the page on
main, and picks what qualifies onto one batch branch per line. Each
original lands as its own commits under a merge that names it, so
authors and sign-offs survive and the line's own history shows what it
carries.

check is the gate before a cut. fold holds a hand resolution of a
conflict to the batch's shape, verify holds a whole batch to it, and
notify reports what waits on the batch for a maintainer.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-11-06 12:33:36 +00:00
Kirill Ilin c15a0d01b8 feat(operator): let the platform source trust a private registry CA 
The operator generates the cozystack-platform OCIRepository from its own
arguments and rewrites it on every start, and cozystack-packages copies
that spec. A registry behind a private CA therefore had no durable way
to be trusted: a certSecretRef patched onto the live object is dropped
on the next operator restart, after which source-controller fails TLS
and platform updates stop arriving.

--platform-source-cert-secret sets spec.certSecretRef from the operator
itself, so the CA (or a client certificate) survives the rewrite. Git
sources are refused instead of ignored, because GitRepository has no
certSecretRef and Flux reads a Git CA from the source secret instead.

Assisted-by: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
 2026-11-05 20:19:29 +05:00
Alexey Artamonov cd91138f33 fix(backups): re-render the default backup objects when the S3 endpoint changes 
The S3 endpoint the platform backups use is copied twice and neither
copy is refreshed. The COSI sidecar writes BucketInfo once per
BucketAccess, so the credentials Secret rendered from it keeps its
endpoint, and the Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of backupstrategy-controller saw. The default
objects gate only acted on objects that were missing, so after a
publishing.host change every backup kept going to the old host with
nothing reporting it.

The gate now also forces a render when a copy is stale: the bucket's
<bucket>-system release when the credentials Secret's endpoint differs
from the COSI Secret it is rendered from, and backupstrategy-controller
when the endpoint an object was rendered with differs from the
credentials Secret's. That endpoint sits at a different path in every
kind, in some only as a bare host, so the chart stamps the rendered
value on each object as backups.cozystack.io/endpoint and the gate
compares that. Objects rendered before the annotation existed count as
stale until one forced render has gone through, which stamps it.

The COSI Secret itself still holds the endpoint the BucketAccess was
granted with; it follows a host change only once the BucketAccess is
re-granted.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-11-04 19:13:55 +03:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034) 
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
 2026-11-03 23:09:46 +05:00
IvanHunters a444ac4498 Merge branch 'main' into fix/tap-pin-digest 
Reconcile the tap digest-pin change with #5071 (no-install-on-connect), now
merged into main. Keep this branch's resetTapPinState, the verify-before-pull
Trust text, and the index-README trust model, while adopting #5071's wording
that installing (not tapping) runs charts and dropping the stale
"re-registration" phrasing from the shared tap.go and rest.go comments.

Conflicts resolved in cmd/cozypkg/cmd/tap.go, pkg/registry/core/tap/rest.go,
and hack/packages-index/README.md.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-30 14:02:24 +03:00
IvanHunters 09b0688abd fix(marketplace): fail closed on an unparseable verified digest, and cover the generation gate 
A verified cosign signature whose manifest digest could not be parsed fell back
to an unpinned tag pull, which reopens the tag-mutation window the verification
was meant to close. Return an error instead.

Add a test that isolates the status.observedGeneration == metadata.generation
conjunct of artifactReady; the existing tests exercised only the tag-prefix
conjunct, so a mutation of the generation check went uncaught.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-26 17:11:52 +03:00
IvanHunters 0195e73d38 fix(marketplace): verify the signature before pulling, and anchor the identity match 
With --certificate-identity and --certificate-oidc-issuer, `cozypkg tap` pulled by
tag without verifying and applied the artifact's PackageSources with
ForceOwnership; only the Flux source carried the verify policy. A tag moved by
someone without the signing identity could land unsigned content in the cluster.
Verify the signature at tap time and pin the verified digest, so the content
pulled and applied is what a trusted identity signed.

Flux evaluates a cosign matchOIDCIdentity issuer and subject as Go regexps, and
they were written unanchored. An exact identity such as alice@example.com would
also accept malice@example.com, and the dots would match any character. Anchor
and quote both values at the API and the CLI.

Reject --tag together with a digest reference instead of silently ignoring the
tag.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-26 16:55:47 +03:00
IvanHunters 96758b9e3d fix(marketplace): drop the index-entry digest from the tap pin change 
The index-entry `digest` field, and the gate and README claims around it, assume
a published index that does not exist yet: decision 0002 in the marketplace
proposal defers it, and #4981 tracks that work. The gate also never compared a
recorded digest against the artifact it verified, so the field added surface
without the check that would make it trustworthy.

Drop it from this PR. Digest pinning still works from an explicit
oci://host/org/repo@sha256:... reference and from the digest a tag resolves to;
only the index-entry path is removed. The tag-based pinning and the verify-flag
wording stay in the index README.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-26 16:55:36 +03:00
IvanHunters 2f39e304f0 fix(marketplace): reconcile tap texts with no-install-on-connect 
Removing on-connect auto-registration made several user-facing and doc texts
wrong: they promised that tapping registers a repository's apps in the catalog
and runs its charts. Tapping now installs nothing, so the texts told operators to
wait for apps that never appear.

Reword the `cozypkg tap`/`untap` help, the tap success output (now printing a
`cozypkg add <name>` line per materialized PackageSource), the `cozypkg init`
scaffold README, the packages-index trust model, the dashboard Repositories
banner and per-repository empty state, and the stale re-registration code
comments. No behaviour changes beyond the added `cozypkg add` hint in the tap
output.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-26 15:18:14 +03:00
IvanHunters 0fd970304c fix(marketplace): preserve verify and pull secret on CLI re-tap 
A re-tap applies the OCIRepository under field owner cozypkg via server-side
apply; a field this owner set earlier and now omits is removed. Re-tapping to
move the pin without repeating --certificate-identity/--certificate-oidc-issuer
therefore silently dropped the signature-verification policy, and re-tapping a
private repo without --secret dropped the pull credential. The CLI now carries
an existing source's spec.verify and secretRef onto the applied object when the
invocation does not set them, mirroring the API connect path, so moving the pin
never silently downgrades trust or breaks a private pull.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-22 22:35:21 +03:00
IvanHunters 5747f94781 docs(marketplace): document digest pinning and verify on tap 
Update the tap command help and the community-index README for the new
behaviour: digest references, the automatic tag-to-digest pin, the
--certificate-identity / --certificate-oidc-issuer pull-time verify flags, and
the optional index entry digest. Extend the generated openapi definitions and
model-name helper for the new TapVerify type (code generation is otherwise
run in CI).

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-22 22:20:34 +03:00
IvanHunters c59ced04cc feat(marketplace): record and pin a digest in community index entries 
An index entry can now carry a digest (sha256:<64 hex>) alongside its version.
Resolving a short name pins that digest, so 'cozypkg tap <name>' consumes the
exact content the CI gate verified rather than following a mutable tag; the
version is kept informational. A malformed digest in an entry is a hard load
error, consistent with the rest of the strict index decoding.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-22 22:16:39 +03:00
IvanHunters c56b9876f4 feat(marketplace): accept a signature-verification policy at tap connect 
A tap can now carry a keyless cosign verification policy (spec.verify), so a
reference is only pulled when signed by the recorded identity. The CLI takes
--certificate-identity and --certificate-oidc-issuer (both required together,
mirroring validate); the API takes spec.verify.{issuer,subject} on the Tap
connect object. Both map to a Flux OCIRepository verify block with provider
cosign and a matchOIDCIdentity entry. A half-configured policy (only one of
the identity pair) is refused rather than set, since a policy that matched any
signer is worse than none. A key-based policy an admin sets on the source by
hand is preserved across repeat connects.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-22 22:14:32 +03:00
IvanHunters cbd116b40d fix(marketplace): reset the operator digest pin on re-tap (CLI) 
A re-tap applies the OCIRepository under field owner cozypkg via server-side
apply, which cannot clear spec.ref.digest once the operator owns it. A
tag-based re-tap would therefore leave the source pinned to the old digest and
never follow the newly requested tag. After applying, the CLI now clears the
operator's digest pin and the pinned-digest annotation for a tag-based re-tap
(an explicit-digest re-tap keeps the digest it just set), alongside the
materialized-revision annotation, so the operator re-pins the new reference.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-22 22:07:46 +03:00
IvanHunters 840a600543 feat(marketplace): pin an explicit digest at tap connect (CLI) 
buildTapOCIRepository now writes spec.ref.digest when the reference carries
one, and the pull resolves the artifact by digest so it fetches the exact
content regardless of where the tag points. A tag-only reference keeps being
pinned to a digest later by the tap materializer; an explicit @sha256
reference is pinned here at connect time, with any tag left informational.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-22 21:58:07 +03:00
IvanHunters b8a129e356 feat(marketplace): accept @sha256 digest references in tap connect 
parseOCIRef (CLI) and parseConnectURL (API) now accept a pinned digest
reference (oci://host/org/repo@sha256:...), with an optional informational
tag before it. The digest must be a full sha256:<64 hex>, the same contract
the platform source's digest= reference enforces. A reference carrying only
a digest no longer defaults its tag to latest, since the digest is the
authoritative reference Flux resolves before any tag.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-10-22 21:55:45 +03:00
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website 
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-21 17:50:46 +05:00
Alexey Artamonov 8bcb709ce8 fix(fluxcd): refuse a flux-plunger drain deadline that leaves no time 
A drain deadline of zero or less is already spent when the drain
starts, so flux-plunger would exit holding every uninstall fence
without listing one. The chart could produce one: its 40-second floor
was a literal, so raising the manager's stop bound or the margin before
SIGKILL, which the drain subtraction is already tied to, would leave the
floor rendering a non-positive deadline. The binary now refuses such a
flag at startup, before it reaches for a cluster, and the chart's floor
and the timings the README and the values comment state are held to
the same constants.

A whole number of a million seconds or more printed in exponent form
and was refused as a fraction; the range is now checked first, so it
gets the range message.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-10-13 15:09:00 +03:00
Alexey Artamonov 42a27567ce fix(fluxcd): keep Helm uninstall working during flux-plunger recovery 
flux-plunger recovers a "has no deployed releases" HelmRelease by
briefly suspending it, deleting the stale storage Secret and
unsuspending. helm-controller v1.5.x skips the uninstall and drops its
finalizer when it handles a delete on a suspended HelmRelease, and it
wakes on the same delete event flux-plunger does, so a delete inside
that window orphaned the release's DataVolume, PVCs, RBAC and storage
Secret. Lifting the suspension on the delete event cannot win that race
reliably.

Hold the release with a flux-plunger finalizer while it is suspended.
The finalizer is applied together with the suspension in one server-side
apply. On a delete flux-plunger lifts its suspension and keeps the
finalizer, helm-controller reconciles the unsuspended generation and
runs the uninstall, and flux-plunger removes the finalizer once
helm-controller's own is gone and that generation is observed. This
holds whichever controller handles the delete first. A completed
recovery lifts the suspension and the finalizer together, and a
terminating release never begins recovery.

Decide ownership of a suspension by which field manager owns
spec.suspend in metadata.managedFields rather than by an annotation. The
apiserver transfers the field away the moment an operator, cozyhr resume
or a backup controller writes a different value to it, so a later
suspension is never misread as flux-plunger's, and one set through the
shared flux-client-side-apply manager (platform migrations, the tenant
delete hook, a manual kubectl apply) is never claimed. A release deleted
under another manager's suspension is left to that manager's decision.

Acquire applies without forcing first and, on a conflict, re-fetches and
forces only if spec.suspend is still false: a backup controller owns the
field at false through a dynamic-client Update, which a non-forced apply
can never take, while true is an active suspension to leave alone. Every
apply is preconditioned on the resourceVersion and uid of the read that
decided on it. Without the resourceVersion, the unforced apply of true
over a foreign true is accepted as shared ownership, a window the
optimistic-locked merge patch this replaces had closed, and lifting the
suspension and the finalizer from a release read as live would delete it
without an uninstall if helm-controller had meanwhile handled a delete.
Without the uid, an apply to a release that has gone would create an
empty one. The apply carries only spec.suspend and the finalizer, so it
cannot leak the required spec.interval.

Read the release, every re-check and the storage Secret list from the
apiserver rather than the informer cache. A lagging cache would let the
processed-version guard delete the revision below the one just
reclaimed, and the ClusterRole grants secrets get and list but not the
watch an informer needs.

Emit a Warning when flux-plunger holds a suspension it failed to clear,
and Normal events when it leaves an external suspension alone or defers
a recovery. An external suspension on a release flux-plunger has
recovered before is a Warning instead: it is most likely one an earlier
flux-plunger left under the shared manager, which nothing lifts unless
an operator does. A release deleted under another manager's suspension
is a Normal event when that was a migration's choice, and a Warning when
the release carries flux-plunger's processed-version annotation, since
then the skipped uninstall is an accident.

Keep a single writer with leader election. A second process able to act
under the same field manager could clear a suspension the first is still
recovering, and no rollout strategy prevents one: the next pod starts as
soon as the old one is terminating, and an eviction or a node drain
replaces a pod the same way. The old pod keeps its Lease, unreleased,
through its stop and drain, and the Lease outlasts both, so the next pod
acts only after the old one is done; the price is that a new pod waits
for the Lease to run out. The strategy sets maxSurge 0 with
maxUnavailable 1 and stays RollingUpdate because existing installs carry
the apiserver-defaulted rollingUpdate block and helm-controller applies
server-side, where type Recreate over it is rejected on upgrade.

Give the finalizer a way out when flux-plunger itself goes. Once the
manager has stopped, within a bounded time, and with it every reconcile,
flux-plunger drains its finalizers under a deadline that, with the stop,
fits inside the pod's termination grace period. A stop that overran its
bound is drained too and still reported, and only a pod that led drains.
The drain writes its events straight to the apiserver, since the
manager's recorder stops with the manager. A recovery in progress is
aborted, the suspension and the finalizer lifted together for the next
flux-plunger to start again, and a release being deleted has the
suspension lifted and keeps the finalizer until helm-controller has run
the uninstall. A release helm-controller does not handle in time keeps
it. The waiting lives in the controller, where the envtest suites hold
it against a real apiserver's generations and finalizers, rather than in
a shell script against a model of them.

A pre-delete hook on the chart is the backstop for what the drain could
not finish, so uninstalling the component never leaves a release that
tenant teardown waits on forever. It stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and finalizer without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, with a JSON patch preconditioned on the resourceVersion and
uid it read: kubectl apply --server-side on a release last written by a
client-side apply moves that apply's fields to the applying manager and
strips every one the body leaves out. The Job's deadline sits under the
300s helm-controller gives a pre-delete hook by default. A Deployment
that cannot be stopped or a listing that never succeeds ends the hook
before any finalizer is touched, and a finalizer it cannot remove ends
it after the sweep; either way it exits non-zero: a failed pre-delete
hook leaves the release untouched and helm-controller retries the
uninstall, hook included, from a state it can still converge from, where
exiting 0 would remove the component and strand the finalizer. A
flux-plunger pod still present after its grace period, typically one
stuck Terminating on a NotReady node, is only reported, since failing on
it would fail every retry until the node returns; because it may still
be setting finalizers, they are listed again after the sweep and any
found fails the hook. The Job runs once, since helm-controller's retry
is the one that starts afresh, and is kept after success so that its
report stays readable; helm.sh/hook-output-log-policy would make
helm-controller call a nil output writer (fluxcd/helm-controller#1562).
A delete still held after ten minutes raises a Warning, and the held
state is re-checked on a timer instead of only on helm-controller's next
status write. The chart README documents the finalizer and how to clear
it.

Run the envtest suites on every pull request through a new make target
that fetches a kube-apiserver and etcd with a pinned setup-envtest and
takes the HelmRelease CRD from the in-tree Flux manifests: the unit
tests drive a fake client, and everything the finalizer rests on (apply
preconditions, the finalizer set-merge, real managedFields) only holds
against a real apiserver. CI caches those binaries, and the target uses
them without asking the release index, so a pull request does not depend
on their sources being reachable; the control plane gets a longer start
timeout because it starts beside the other test goals.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-15 23:34:52 +03:00
Aleksei Sviridkin 8c1a78dc9b feat(operator): default system namespace memory via LimitRange (#3494) 
## What this PR does

System components now get a default container memory limit, so their pod
cgroups carry a `memory.max` and drop out of the Talos userspace OOM
handler's victim set. `cozystack-operator` maintains a LimitRange named
`cozystack-system-defaults` in every system namespace it reconciles.

The Talos handler (v1.12+) scores cgroups with `memory_max.hasValue() ?
0.0 : {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current`
and drops everything that scores zero. Only a memory limit sets
`memory.max`, so only a limit takes a pod out of the candidate set. A
request just moves a pod from BestEffort to Burstable, which under the
default `strictCgroupClassOrdering: true` still means killed second.
`PriorityClass` does nothing here, because the handler reads cgroupfs
and never talks to the apiserver. Victim selection is also unrelated to
the trigger, so a tenant workload thrashing against its own limit can
drive node memory pressure while a system component with no limit is
what gets killed.

An audit of a running cluster found around 70 workloads across ~30
packages with at least one container missing a memory limit. Per-chart
resources would not cover that set, and would not keep covering it as
packages are added. So the LimitRange is written by
`reconcileNamespaces`, which already creates these namespaces and
already tells system from tenant. It also covers init containers, which
#3495 can't reach: one of them without a limit leaves the pod cgroup
without `memory.max`, whatever the main containers declare.

Tenant namespaces are skipped. Where the tenant chart's own
`tenant-range-limits` exists (only when the operator configured
`resourceQuotas`), a second LimitRange next to it would leave the result
to the order LimitRanger iterates them. Where it doesn't exist, a tenant
namespace is not the place for the platform to impose a default.

### Why the default is 32Gi

The handler grants immunity for `memory.max` existing at all, whatever
its value. So the ceiling doesn't need to be fitted, and 32Gi is high
enough that nothing plausibly reaches it. A ceiling low enough to bind
would turn a rare pressure-driven kill into a deterministic one for any
component whose real usage nobody measured. This repo already has one
such case, recorded in the `vertical-pod-autoscaler` values, where a
110Mi limit OOM-killed the updater and failed HelmReleases with it.
Scheduling is unaffected, since pods are scheduled against requests and
the paired default request is 32Mi. An operator who wants the ceiling to
bind can lower it, per component in that component's chart or fleet-wide
with the knob, once real usage is measured.

### When the default is withheld

A default below a container's own request makes that pod impossible to
admit, so the operator checks before it applies. It scans Deployments,
StatefulSets, DaemonSets, CronJobs, Jobs and Pods in the namespace, init
containers included, for a container that requests more than the
configured limit and declares no limit of its own. If it finds one, it
withdraws the managed LimitRange and records a hold in the
`operator.cozystack.io/system-memory-state` annotation on the Namespace.

The hold does not clear by itself when the blocker disappears, because
an empty scan looks the same as a pod between deletion and recreation.
Lowering a previously applied limit also creates a hold, even with no
visible blocker, because LimitRanger writes the defaulted limit onto
admitted pods and the scan can no longer tell which containers declare
one. A cluster administrator releases the hold by copying the
`<hold>:<target>` token from the state into the
`operator.cozystack.io/system-memory-ack` annotation. The token is used
once and only for that hold and target. If a blocker is still visible,
the token is rejected and the hold stays. Package reconciles are
scheduled at least every five minutes, so an acknowledgement takes
effect on a quiet cluster too. The reason and the token also appear in
the operator log under `withholding the default container memory limit`.

### Namespace another LimitRange already governs

If a system namespace already has a LimitRange that says anything about
memory, at the Container or the Pod scope, the operator writes nothing
there and withdraws what it wrote before. A second default leaves the
effective value to iteration order. A foreign `max` below this default,
a `min` above the default request or a `maxLimitRequestRatio` rejects
pods outright, which takes out a whole namespace. At the Pod scope the
collision can't be decided in advance, because it depends on how many
containers a future pod has. A LimitRange that bounds only cpu or
storage coexists normally. A same-named LimitRange without the
operator's managed-by label is treated as foreign and left alone.

This matters most in `kube-system`, which is in scope because
`cozystack-scheduler` installs there, and where a distribution's or an
administrator's own LimitRange is likeliest to sit.

### Disabling

Setting `cozystackOperator.systemNamespaceMemoryLimit` to `0` disables
the feature. On the next reconcile the operator removes every LimitRange
it owns cluster-wide, found by the managed-by label. While the feature
is on, it also removes its LimitRange from namespaces no active Package
targets any more. The safety state on the Namespace stays in both cases.

### Docs

`docs/operations/system-memory-limits.md` landed in #3496 ahead of this
code, so main describes a `4Gi` default and installer values that do
nothing yet. The page is corrected to the 32Gi default and now describes
withholding, the acknowledgement, and the foreign LimitRange rule.

### Not in this PR

linstor policy. #3824 extends the controller's existing exemption from
limits to the satellite, and a linstor limit here would contradict it,
so linstor stays with that PR.

The scan reads six workload kinds through `APIReader`, uncached, on
every reconcile of every system namespace. That cost is tracked in
#3977, since the system namespace set isn't known when the cache is
built and the right fix depends on a reconcile-rate measurement nobody
has taken.

### Verification

`go vet` and `go test` pass for `cmd/cozystack-operator` and
`internal/operator`, and `helm unittest` passes on the installer chart
for the knobs unset, set, and the limit at `0`. The reconciler tests
cover withholding on an oversized request, the hold surviving the
blocker's removal, the hold on a lowered limit, acknowledgement and
token rejection, every memory field of a foreign LimitRange at both
scopes, ownership by label, the disabled sweep, and a deleting Package
not creating policy.

The withhold design has not been run on a live cluster yet. An earlier
design of this PR was, and that run does not carry over.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff.
`packages/core/installer/values.yaml` gains two keys, but the
`ansible-cozystack` trigger is for a required key or a rename of one the
role passes, and these are new, optional and default to `""`, so the
role is unaffected. Nothing else in the map is reached by the Go
changes, the operator template or the page.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(operator): system namespaces now get a default container memory limit through a LimitRange maintained by cozystack-operator, which keeps system components out of the Talos userspace OOM handler's victim set. Tune it with cozystackOperator.systemNamespaceMemoryLimit (default 32Gi) and cozystackOperator.systemNamespaceMemoryRequest (default 32Mi); set the limit to 0 to disable it. The default is far above any real working set on purpose, because the handler grants immunity for a memory.max existing at all, not for its value. A namespace with a container that requests more than the limit and declares no limit of its own gets no default, and stays without one until a cluster administrator acknowledges the hold recorded on the Namespace (see docs/operations/system-memory-limits.md); lowering a previously applied limit needs the same acknowledgement. A namespace where another LimitRange already sets anything about memory is left alone. A container that declares its own memory limit is never changed.
```
 2026-10-07 11:19:26 +03:00
Aleksei Sviridkin 798094e725 feat(release): add backport-audit, a pre-release backport triage tool (#3527) 
## What this PR does

This adds `cmd/backport-audit`, a maintainer tool that checks, before a
release or an rc is cut, whether everything labelled for backport
actually landed on the release branch. It prints the URLs of the PRs
that did not and exits non-zero, so the cut can gate on it.
`docs/release.md` and `docs/agents/releasing.md` now send the patch-cut
check through it.

The docs it replaces inventoried backports with `gh pr list --search
"is:merged label:kind/backport"`. That lists every PR ever labelled, not
what reached the branch. Answering the real question by hand means
reading the branch history per PR, and that is how backports go missing
quietly.

```console
$ go build ./cmd/backport-audit
$ ./backport-audit release-1.4

=== release-1.4 === 30 candidate PRs: MISSING=5 dropped=4 backported=21

  MISSING -- labelled for this line, no trace of it here (5):
    https://github.com/cozystack/cozystack/pull/2704
      #2704 fix(info): use root-host for Keycloak OIDC issuer URL in tenant kubeconfig
      label=kind/backport author=myasnikovdaniil merged=2026-05-26 -- no backport PR, nothing on branch
```

A `kind/backport` label does not name a branch. `backport.yaml` resolves
the target when the PR merges, so the same label means `release-1.5` on
a PR merged in June and `release-1.6` on one merged in August. The audit
reproduces that rule instead of trusting the label text. The rule
changed once, on 2026-08-03, when 53a7fc4dc made cutting an rc freeze
the line and 6ff5b98d5 pointed the bot at the branch list. Before that
date a line exists from its first stable release, after it from the
moment its `release-X.Y` branch is pushed. The audit estimates that
moment from the earliest `vX.Y.0-rc.N` tagged commit. This is an
estimate, because the commit timestamp is not the branch-push timestamp.
The README has the full reasoning.

Landing is checked per commit, because a commit is what a backport
loses: the bot's conflict drafts stop at the first commit that does not
apply. Every non-empty commit of the original needs its own evidence on
the branch. That is reachability, an `-x` reference, or a commit with
its subject among the branch commits that are not on main. Some but not
all is PARTIAL. A multi-commit original with a merged backport PR and no
commit evidence is UNVERIFIED. Both fail the gate. A maintainer can
confirm a squashed backport with a comment on the merged backport PR
that is exactly `backport-audit: complete`.

Two more sections start from the backport PRs on the branch instead of
from labels. DUPLICATE lists an original with more than one live
backport PR and fails the gate. UNLABELLED lists backports of originals
that are not labelled for this line. It is informational and does not
change the exit code.

Exit code 2 means the audit could not give an answer. That happens when
a `gh` listing reaches its cap, since `gh` truncates silently, or when a
git read that a verdict depends on fails. The label listings are checked
against the lower of `--limit` and 1000, because GitHub search never
returns more than 1000 results.

### Verification

`cmd/backport-audit/main_test.go` covers candidate selection, line
resolution and the evidence rules. `make unit-tests` runs it through a
new `test-backport-audit` target, since `./cmd/...` is not part of
`go-unit-tests`. `hack/release-freeze-contract.bats` pins the audit's
label set equal to the one `backport.yaml` acts on, in both directions.

### Limits

A MISSING verdict is a prompt to check, not proof of absence. A hand
backport that was squashed, reworded and never linked to its original is
invisible to every evidence layer. A label added after merge is read
against the line that was current at merge time, while the bot targets
the line current when the label is added. Such a backport shows as
MISSING on the old line and under UNLABELLED on the new one. The audit
reports that something is missing, never why. `docs/release.md` now
notes that before #3155 the bot left only a "Backport failed" comment on
a conflicting cherry-pick, which is the first thing to check behind a
MISSING.

`make generate` does not apply. This is a standalone `main` package with
no API types, no CRDs and no package `values.yaml`.

### Screenshots

Not a UI change. Sample CLI output is above.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff file by file. The diff adds
`cmd/backport-audit/{main.go,main_test.go,README.md}` and changes
`Makefile`, `hack/release-freeze-contract.bats`, `docs/release.md`,
`docs/agents/releasing.md` and one `.gitignore` line for the root
binary.

Nothing in it reaches a downstream repository. No package under
`packages/apps/` or `packages/extra/` is added, renamed or removed, and
no `values.schema.json`, `values.yaml` default or version enum changes,
so the website generator lists and the terraform provider schemas are
untouched. No `ApplicationDefinition` semantics, `release.prefix`,
output Secret or Service name, namespace, platform variant or bundle
changes. The paths that reach ccp and external-apps-example
(`hack/package.mk`, `hack/common-envs.mk`, `hack/update-crd.sh`) are not
touched. The Makefile change adds a `test-backport-audit` target to
`unit-tests` and changes no existing target. No node prerequisite
changes, so talm and the ansible playbooks need nothing. The website
"developer tooling" trigger covers tools contributors use on packages.
This is maintainer release tooling, and its process doc lives in this
repository only.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(release): add `backport-audit`, a maintainer tool that reports which `kind/backport` / `kind/backport-previous` PRs have not landed on a release branch (missing outright, partially backported, waiting on an open backport PR, or deliberately dropped) and exits non-zero while anything is outstanding, so a patch cut can gate on it
```
 2026-10-07 08:59:38 +03:00
Myasnikov Daniil 5b585d6eda fix(operator): persist holds for unsafe memory defaults 
An admitted Pod loses the distinction between explicit and defaulted
limits. Its disappearance cannot prove a lower default is safe for
its replacement. Keep that decision on the Namespace until an
administrator acknowledges the particular target, without waiving
current blockers or foreign policy ownership.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 06:06:51 +03:00
Myasnikov Daniil e83b4eb492 fix(operator): align memory policy ownership messages 
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
 2026-10-07 06:06:50 +03:00
Myasnikov Daniil b3168a3824 test(operator): pin system memory defaults 
Keep the parser inputs tied to the constants registered as flag defaults,
but compare the results with literal 32Gi and 32Mi expectations. A shipped
default can no longer move while the defaults case stays green.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 06:06:50 +03:00
Myasnikov Daniil 6a521f0c68 fix(operator): sweep memory defaults when disabled 
List LimitRanges by the ownership label once per Package reconcile when
the feature is disabled, and remove managed objects even in namespaces
outside the current Package target set. Skip the per-namespace path while
disabled and keep cleanup failures non-fatal to namespace reconciliation.

Update the flag help, installer guidance, and operations documentation to
describe the global cleanup boundary.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 06:06:49 +03:00
Myasnikov Daniil f6dcb69b61 refactor(operator): withhold instead of raising, and drop the grace period 
The raise-and-grace machinery cannot work: LimitRanger writes the
raised limit onto the very container that justified the raise, so at
that container's next admission it carries a limit, the scan skips
containers that have one, and the raise has erased its own evidence.
The ceiling then drops on schedule and the workload is rejected
permanently. Any such machinery approximates admission logic ahead of
admission, and that is incomplete by construction: a
LimitRange default is one number for a whole namespace and cannot mean
"the configured limit, or this container's request, whichever is
larger". Only per-pod conditional defaulting expresses that, which is a
mutating webhook and not this.

The justification for the machinery had also expired. It was designed
when the default was 4Gi, where withholding was self-perpetuating for
VPA-managed namespaces. At 32Gi nothing in the tree approaches the
ceiling -- the largest static memory request is 2Gi -- and VPA sets
requests and limits together, so those containers carry their own limit
and were never blockers.

So a namespace holding a container that requests more than the
configured limit with no limit of its own is now left with no default at
all, and anything written there earlier is withdrawn. It fails in the
safe direction, which is the whole argument: a namespace with no
memory.max is a namespace behaving as it did in every release before
this, while a default below a request is a namespace whose workload
cannot be admitted. It needs no stored state and no timer, because
withholding is not a decision that has to survive between reconciles --
once the oversized request is gone the default is applied again on the
next pass.

Gone with it: the grace period, the clear-since annotation and its
clamp, the injected clock, containerMemoryDefault, raisedCeiling, and
fourteen tests. The scan stays, because deciding whether to apply or
withhold is what it was always for. The foreign-LimitRange ownership
guard, the disable path and the 32Gi default are untouched.

The residual sharp edge is stated rather than machinery: a workload
whose only evidence is a live Pod, built from a custom resource with no
template this scan can read, can still be missed if the scan runs while
that Pod is absent, and applying the default then rejects its
recreation. At 32Gi that is unreachable in this tree. It becomes
reachable if an operator lowers the knob below such a workload's
request, which the flag help, values.yaml and the operations page now
say plainly.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 06:06:48 +03:00
Myasnikov Daniil c6342bf885 fix(operator): bring the flag help in line with the guard and removal scope 
The --help text still carried two claims that values.yaml and the
operations page had already narrowed, and it is the one surface an
operator reads without opening the tree.

Both were wrong in the direction that costs the reader something. "A
namespace where another LimitRange already defaults or caps container
memory is left alone" described the guard before it was widened, so an
administrator holding a container memory min, a maxLimitRequestRatio or
any Pod-scoped bound would read it, expect the LimitRange to be applied
there anyway, and get a namespace with no memory.max for a reason the
help says does not apply. "Removes any previously created one" is the
universal removal claim, and removal is scoped to namespaces an active
Package still targets.

Two comments corrected while here. constrainsMemory said it switched on
the item's own maps "rather than a hand-picked few", which contradicts
the hand-picked list immediately below it; what is actually true is that
a field added upstream has to be added there too, which is the reason
it has its own table test. And memoryDefaultSettledAt claimed the count
of raised namespaces falls to none once the oversized requests are
gone, which stopped being true when the scan started keeping Failed
pods: a carcass Kubernetes never collects holds both the raise and that
namespace's per-reconcile pod List open.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 06:06:46 +03:00
Myasnikov Daniil 81c12bc953 fix(operator): withhold on any foreign memory policy, not two fields 
The coexistence guard treated a namespace as already-governed only when
another LimitRange carried a container memory default or max, and said
so in its comment. That is not the set of fields that collide with what
this reconciler writes. A container min above the default request
rejects the pod, and so does a maxLimitRequestRatio, since a 32Gi
default against a 32Mi default request is a ratio of 1024:1 and no cap
allows it. So the namespace-wide admission failure the guard exists to
prevent stayed reachable through fields the guard did not read, and
kube-system is where it is likeliest, this feature having deliberately
brought it in scope.

Enumerating the colliding fields does not fix it, because at the Pod
scope the collision is not decidable here. Whether a per-container
default of 32Gi breaches a pod-wide max depends on how many containers
the pod has, and the pods in question do not exist yet: any arithmetic
about them is a guess, and a wrong guess costs a namespace its entire
pod admission. So the rule is ownership rather than arithmetic. If
anything else in the namespace has an opinion about memory, at either
scope and in any field, the namespace is not this reconciler's to
default. Narrow still on the resource, because nothing here writes cpu
or storage and a LimitRange bounding those must keep coexisting. This
is the rule the tenant-namespace exclusion already rests on.

The pod scan no longer skips Failed pods, only Succeeded. A pod that
ran to completion is never recreated from its spec, but an eviction or
a node crash leaves a Failed carcass holding the same oversized request
that its controller will recreate, and that carcass is frequently the
only surviving evidence the namespace needs a raised ceiling, since the
shape the scan looks for cannot be admitted once the namespace has
settled. Dropping it lowered the ceiling under a workload that was
coming back, which is what the grace period exists to prevent.

A grace stamp dated ahead of the clock is clamped to now. Unclamped it
held the ceiling for the skew plus the grace period, silently, so an
hour of skew bought ninety minutes instead of thirty. Clamping never
shortens the wait, it only stops it being extended.

The two flag defaults are named constants the flag registration and its
test both read. The test previously duplicated the literal and still
asserted 4Gi after the default moved to 32Gi, so it misinformed a
reader about the shipped value and pinned nothing.

The disable claim is narrowed to what the code guarantees. "Removes any
previously created one" was universal, but the reconciler only runs for
namespaces an active Package still targets, so a namespace whose
Package was removed keeps its LimitRange. The flag help, values.yaml
and the operations page now say that, and name the managed-by label for
finding leftovers by hand.

The test asserting a Pod-scoped memory max should be ignored is gone,
since under the ownership rule a Pod-scoped memory bound withholds.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 06:06:45 +03:00
Myasnikov Daniil e131a49514 feat(operator): default the system namespace memory ceiling to 32Gi 
The guard in front of this default reads declared memory requests. It
cannot see a container's real working set, so a system component that
requests little but legitimately uses more than the ceiling, and
declares no limit of its own, was handed a deterministic kernel OOM on
its next recreation in exchange for leaving the Talos handler's victim
set. Since the feature ships enabled, that trade fell on the whole
fleet on upgrade.

It is not a trade this feature has to make. The handler discards a
cgroup for carrying a memory.max at all, whatever its value, so
immunity does not require a fitted ceiling -- only a real one. A
default far above any plausible working set buys the immunity and
leaves nothing for the ceiling to bind on, which also keeps it clear of
the failure mode this repository already carries a scar from, where an
undersized limit converted a rare pressure-driven kill into a reliable
one.

Nothing about scheduling changes: only requests are scheduled against,
and the paired default request stays 32Mi. An operator who wants the
ceiling to bind lowers it, per component in a chart or fleet-wide with
this knob, once real usage is measured -- which is the direction that
needs evidence, and now the direction that needs it rather than the
default.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 05:36:53 +03:00
Myasnikov Daniil b6cb0def78 docs(operator): say that kube-system is in scope for the memory default 
The scope of the default LimitRange is every namespace a Package targets
whose name does not start with `tenant-`, and cozystack-scheduler installs
into kube-system, so a namespace owned by the underlying Kubernetes
distribution is defaulted like any other. Neither the flag help nor the
installer value said so, and both are what somebody setting the knob
reads.

docs/operations/system-memory-limits.md already covers this, including the
part that surprises: containers the kubelet builds from an on-disk
manifest never pass through admission, so a Talos control-plane static pod
gets no memory.max from this feature in the first place and the mirror pod
is the only thing the LimitRange reaches. Both notes now carry the same
statement, so neither reading of the knob leaves it out.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 05:36:51 +03:00
Myasnikov Daniil 9866113da0 test(operator): pin the system namespace memory flag parsing 
parseQuantity and the request-above-limit check were closures inside
main(), so no branch of either was reachable from a test: not empty, not
malformed, not negative, not a request of 0 against a real limit, and not
the pair check itself. That last one is the one worth having. A limit and
a request that each parse but contradict each other produce a LimitRange
the API server rejects, once per system namespace on every Package
reconcile, and nothing in that error points back at a flag.

Lift both into parseSystemNamespaceMemory, which returns an error instead
of calling os.Exit, and pin all of it: the shipped defaults, empty as the
off switch, an explicit 0 limit beside a set request, a 0 request against
a real limit (supported on purpose), malformed and negative values on
either flag, request above limit, and request equal to limit as the legal
boundary. main() keeps the same exit-on-error behaviour, now in one place.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 05:36:47 +03:00
Myasnikov Daniil dd36eeddbc docs(operator): describe the raise, not the retract, in the shipped help 
The flag help and the installer values comment both still said the
operator skips a namespace holding a request above the configured limit
and retracts any LimitRange it applied there. It stopped doing that in
3127c0fad: it raises that namespace's ceiling to clear the request and
drops back once the request is gone. An operator reading the shipped help
and then finding an 8Gi LimitRange in cozy-monitoring had no way to
connect the two.

Both texts now describe the raise, the drop-back and the log line to grep
for. The values comment also loses its VPA warning, which this branch's
own comment in findRequestAboveDefaultLimit refutes — VPA rescales the
limit in proportion to the request under the default controlledValues, so
it cannot be what puts a namespace over — and gains a pointer to
declaring a limit in the component's own chart, which is how
packages/system/linstor now stays out of the default's reach.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 05:36:47 +03:00
Myasnikov Daniil 7eecbe9b4c docs(operator): document a zero default memory request 
--system-namespace-memory-request=0 passes the startup check, since 0 is
not greater than the limit, and produces a LimitRange with
defaultRequest: 0. Whether that is a footgun to reject or a supported
setting to explain was open, so I asked a real API server: envtest with
kube-apiserver 1.36.0 and LimitRanger enabled, with a second namespace
using 32Mi as the positive control that the plugin was actually running.

It is supported. The API server accepts the LimitRange and stores
defaultRequest as 0 rather than folding it back to the default -- the
Default-to-DefaultRequest copy only fills keys that are absent, and this
one is present. A container declaring no memory of its own is then
admitted with an explicit request of 0 and a limit of 4Gi, QoS Burstable.
So LimitRanger does not read zero as absent, and the container still gets
memory.max on its cgroup, which is the whole reason this feature exists.

What it costs is the scheduling signal: nothing is reserved for that
container, which is exactly where system components stood before this
feature. Say so in the flag help text, where an operator setting the knob
will see it, and note at the startup check that 0 passing is deliberate so
nobody hardens it into a rejection later.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 05:36:45 +03:00
Myasnikov Daniil 546d8e6a54 feat(operator): default system namespace memory via LimitRange 
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.

Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.

An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.

The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.

Two things keep the feature from being able to hurt more than it helps.

A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.

And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-10-07 05:36:41 +03:00
Myasnikov Daniil d3df698c43 docs(release): gate maintenance cuts on backport delivery 
Listing labelled pull requests cannot establish whether their changes
landed on a maintenance branch.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
 2026-10-07 05:19:16 +03:00
Myasnikov Daniil a0b1dfb45b feat(release): add per-commit backport audit 
Backport labels record requests, and a merged conflict draft may still
lack commits. Release cuts need a check of what reached the branch.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
 2026-10-07 05:19:15 +03:00
Aleksei Sviridkin 520af57905 chore: drop private cluster, host and personal names from the tree 
Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-07 04:38:06 +03:00
Aleksei Sviridkin de570a3806 feat(platform): add slim installation variants (#4595) 
## What this PR does

This adds three minimal platform variants for small installs, such as
arm64 labs, where isp-full is too heavy. They are `isp-slim` for Talos,
`isp-slim-generic` for k3s, kubeadm or RKE2, and `isp-hosted-slim` for
clusters where the host provides CNI and storage.

A slim variant installs only the base platform: the engine, API and
dashboard, tenants, ingress and Gateway API. The two non-hosted variants
also get LINSTOR. objectstorage-controller stays on too, because
cozystack-controller watches BucketClaim unconditionally and crash-loops
without that CRD. Everything else is opt-in through
`bundles.enabledPackages`, including every paas and naas application and
its operator, monitoring, backups, etcd, SeaweedFS, metrics-server, VPA,
Multus and MetalLB. The iaas bundle is refused, so there are no VMs and
no managed Kubernetes. With the stock preset a slim variant emits 20
Packages (15 for hosted), where isp-full emits 77.

On the two non-hosted slim variants, networking is Cilium alone. Without
VMs nothing needs Kube-OVN, and Cilium L2 announcements take the place
of MetalLB. The admin creates a `CiliumLoadBalancerIPPool` and a
`CiliumL2AnnouncementPolicy`, or uses `publishing.externalIPs`. Pod
ranges come from `node.spec.podCIDR`, which Talos and k3s allocate by
default. `networking.encryption` only drives Kube-OVN IPsec, so the slim
variants refuse it instead of silently ignoring it.

The tenant application needs its own slim variant. The default one waits
on the monitoring, etcd and SeaweedFS applications, so on slim
cozystack-basics and tenant-root would never become ready. tenant-rd and
cozystack-basics still reference the default variant's artifacts by
name. Those artifacts exist anyway, because the PackageSource builds
them for every variant.

The full variants don't change. I rendered isp-full, isp-full-generic
and isp-hosted against main and got byte-identical Packages. The only
new output there is the slim variant in the tenant-application
PackageSource.

Things to know before using it:

- An opt-in package does not pull in its dependencies. The presets and
the docs list the chains. A package enabled without its chain stays
`DependenciesNotReady`.
- Slim is for new installs. Switching a live isp-full cluster to
isp-slim moves the networking Package from `kubeovn-cilium` to `cilium`,
the operator removes the Kube-OVN release, and running pods lose
networking. The other Packages are kept by `helm.sh/resource-policy:
keep`, so the switch doesn't make the cluster smaller either.
- No e2e suite runs a slim variant yet. Helm unit tests cover the
presets, opt-in, OIDC, the networking values and both refusals, and a
dependency check over every emitted Package passes for all three
presets.

I installed isp-slim from a build of this branch on a fresh three-node
Talos 1.13 cluster (amd64). All Packages and HelmReleases went Ready,
with no Kube-OVN, MetalLB, monitoring, backup or KubeVirt pods. From
outside, the dashboard and the API answered by name with valid
certificates. A LoadBalancer Service got an address from a Cilium pool
and answered through L2 announcements. Pod-to-pod traffic across nodes,
cluster DNS and a replicated LINSTOR volume worked. Opting in
postgres-operator and postgres-application gave a Ready Postgres. That
run found the BucketClaim crash-loop above, which is fixed here. The
base platform used about 215m CPU and 3.2 GiB of memory, not counting
the Kubernetes control plane.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#718
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [x]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up: cozystack/ansible-cozystack#79
- [x] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
cozystack/ccp#23
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(platform): add the isp-slim, isp-slim-generic and isp-hosted-slim variants. They install only the base platform (engine, dashboard, tenants, ingress, gateway, and LINSTOR outside hosted) with Cilium-only networking and no MetalLB; applications, operators, monitoring, backups, etcd and SeaweedFS are opt-in through bundles.enabledPackages, and the iaas bundle is not available.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `isp-slim`, `isp-slim-generic`, and `isp-hosted-slim`
platform variants, with support for PaaS and NaaS bundles.
* Slim variants use Cilium networking and include a reduced set of
packages by default. Additional packages and their dependencies can be
enabled as needed.
* **Limitations**
* IaaS and the platform encryption toggle are not supported on slim
variants. Cilium-native encryption is not configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-01 14:03:22 +03:00
Andrey Kolkov f6a615f693 fix(backups): scrub legacy plaintext passwords from CNPG Backup snapshots (#4558) 
## What this PR does

A CNPG `Backup` taken while the Postgres app still accepted
`users[].password` copied that password into
`status.underlyingResources`, where anyone with `get backups` in the
namespace can read it. #4078 stopped new snapshots from carrying the
field but left every existing `Backup` as it was (#4179).

This adds `LegacyPasswordScrubber`, a leader-elected runnable in the
backup strategy controller that, on start, lists `Backup` objects and
removes `users.*.password` from every snapshot whose `kind`/`apiVersion`
is the CNPG snapshot's.

- **Why on every leader start and not a platform migration.** The
migration hook runs pre-upgrade, while the old controller is still
leader, so a `Backup` the old replica writes during the rollout would
keep its password and the migration never re-runs. With `--leader-elect`
on, the old replicas stop writing before the new leader's scrub runs,
and a `Backup` brought back from an object-level export is cleaned on
the next start too. On a clean cluster the whole pass is one List from
the manager cache, which `BackupReconciler` already populates.
- **Write shape.** The patch is a merge patch computed from the diff, so
the request body only nulls the `password` keys and cannot overwrite
status another writer changed since the List. The snapshot is handled as
generic JSON (numbers kept as `json.Number`), so fields the current
snapshot type does not know survive.
- **Failure handling.** A `Backup` that cannot be decoded or patched is
logged and skipped, and a failed List is logged too; `Start` always
returns nil, because a returned error would stop the manager and take
every backup down over one object.
- **Restore compatibility.** Restore already ignored the key; a test
decodes a scrubbed snapshot through `unmarshalCNPGBackupSnapshot` to pin
that.
- RBAC: `list` and `patch` on `backups.cozystack.io/backups` are already
granted to the controller, so the chart is unchanged.
- `docs/operations/backup-classes.md` no longer tells operators to
delete pre-upgrade `Backup` objects, and says what the scrub does not
reach (audit logs, etcd backups, external exports of the objects, Helm
history of the init-script Secret), so a password once set through
`users[].password` is still treated as exposed until rotated.

The runnable carries a `TODO(#4179)` to remove it once upgrading from a
release that accepted `users[].password` is no longer supported.

Verification: `go test ./internal/backupcontroller/...` and `go build
./cmd/backupstrategy-controller/` are green. The new tests cover
scrubbing with everything else kept (including an unknown field with a
large integer), leaving non-CNPG, other-apiVersion, clean and
snapshot-less Backups unpatched, and continuing past a Backup that fails
to decode or patch. Disabling the delete, or returning on the first
patch error, makes the tests fail. Not exercised on a live cluster yet.

Fixes #4179

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map against the diff: it touches only
`internal/backupcontroller`, `cmd/backupstrategy-controller` and the
in-repo `docs/operations/backup-classes.md`, with no package, values,
schema, CRD, variant or asset change. The website carries no "delete
pre-upgrade Backups" workaround this would make obsolete; its stale
`users[name].password` rows are the #4078 follow-up already noted there.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the backup strategy controller now removes plaintext `users[].password` values left in `status.underlyingResources` of CNPG `Backup` objects taken before managed Postgres stopped accepting user passwords. It runs on every controller start and needs no action. It does not reach copies outside the cluster (audit logs, etcd backups, exported `Backup` objects), so a password once set through `users[].password` should still be treated as exposed until it is rotated.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Security**
* At controller startup, plaintext user passwords are removed from
matching CNPG backup snapshots. Other snapshot data is preserved.
* This cleanup does not affect external copies or remove older
init-script manifests from Helm history. Treat passwords previously set
through `users[].password` as exposed until rotated.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-30 11:58:06 +04:00
Aleksei Sviridkin 465195c671 feat(operator): register slim platform variants 
Add isp-slim, isp-slim-generic and isp-hosted-slim to the platform
PackageSource so a Package can select the minimal presets shipped by
the platform chart.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-30 03:12:00 +03:00
Mattia Eleuteri e510d49b3d feat(monitoring): re-render node agents when a monitoring label changes 
The node fluent-bit builds its tenant routes from a lookup of namespace
labels, and helm-controller upgrades a release only when its chart or
values digest changes. A tenant turning monitoring on therefore got no
route, and a removed one kept its route, until an unrelated change
upgraded monitoring-agents.

The reconciler digests the label across all namespaces and requests a
forced upgrade of the release when the digest moves. It digests every
labelled namespace rather than repeating the chart's selection, so it
cannot drift from the chart; the cost is an occasional upgrade that
renders the same config.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
 2026-09-29 17:10:55 +02:00
Andrey Kolkov 19764c4bba fix(backups): scrub legacy plaintext passwords from CNPG Backup snapshots 
A CNPG Backup taken while the Postgres app still accepted users[].password
copied that password into status.underlyingResources, where anyone with
`get backups` in the namespace can read it. Dropping the field stopped new
snapshots from carrying it but left every existing Backup exposed.

The backup strategy controller now removes users.*.password from CNPG
snapshots when its leader starts. It runs on every start rather than once,
so a Backup written by a pre-upgrade replica during the rollout, or brought
back from an object-level export, is still cleaned; on a clean cluster it
is a single cached List. The patch is a diff-computed merge patch that
only nulls the password keys, and a Backup that cannot be decoded or
patched is logged and skipped instead of stopping the manager.

Fixes #4179

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-09-29 11:21:08 +04:00
Aleksei Sviridkin 2c2c0301f7 chore(deps): roll up pending Renovate updates and move to Go 1.27.1 (#4447) 
## What this PR does

This replaces the open Renovate PRs with one rollup. Go modules move to
their newest releases that keep the Kubernetes libraries and
controller-runtime where they are, the toolchain and builder images move
to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers.
Tests for the code the bumps touch come first, in their own commits, and
pass on both the old and the new dependencies.

Before the bumps:

- The e2e sandbox image builds again: dl.min.io answers 410 for mc since
MinIO archived it, so mc now comes from its last GitHub release with a
pinned checksum.
- Renovate stops offering tags of the apimachinery fork. They mirror
upstream, so #2999 is a downgrade that drops our patch.
- `make unit-tests` now runs the tests of the nested Go modules
(kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1).
They never ran in CI before.
- New tests: JWT verification in the dashboard token proxy, the kube-ovn
plunger metrics, cert-manager object idempotency in the tenant gateway,
values-file layering in the operator, and the kubevirt CSI driver
wrapper.
- One real fix. The lineage webhook decoded admitted objects with
encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5
compares numbers exactly and would emit a replace that rewrites such a
field in the pod. The webhook now decodes with apimachinery's json.

Held back on purpose, with the reasons in the commit messages:

- Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork
rebased onto release-1.37 first.
- cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is
built and tested with.
- grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch
release carries the fix yet.
- kubevirt.io/api and the CDI API stay on what upstream csi-driver
builds against. From CDI v1.60 it no longer compiles.
- api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types
are not forced onto Go 1.27.
- go fix runs without embedlit (Go 1.27-only syntax, breaks backports to
release branches on Go 1.26) and omitzero (would make metadata, spec and
status required in the published OpenAPI schemas), and one errorsastype
rewrite that did not compile is left out.

govulncheck reports nothing reachable in any of the five modules,
standard library included.

Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805,
#3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open.

Not checked locally: the images that build upstream projects (kube-ovn,
kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The
image build jobs of this PR cover them.

Closes #4487.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added a restore resource policy option for Velero strategies.
  * Added a test target to run Go tests across tracked modules.
* **Bug Fixes**
* Updated the end-to-end test environment to download and verify the
MinIO client from its current release location.
* Added coverage for token validation, webhook patching, controller
behavior, and reconciliation.
* **Maintenance**
* Updated the Go toolchain, dependencies, and container image versions.
  * Refreshed pinned workflow actions and base image digests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-26 10:17:29 +04:00
IvanHunters 0bae19fccc feat(lineage-webhook): refuse a second ApplicationDefinition for a kind 
A definition declaring an application kind that another definition already
declares now fails when it is applied, instead of sitting next to the first
one where it has no effect. An update that keeps its kind is allowed, so a
duplicate that predates the check keeps upgrading, and so is a second
definition from the same Helm release, since a release that renames its
definition object creates the new one before it deletes the old one.

The webhook is registered with failurePolicy Ignore: ApplicationDefinitions
ship with the platform itself, so an outage of the webhook must never block
a bootstrap or an upgrade, and a failure to read the existing definitions
lets the request through with a warning rather than answering with an error,
which Ignore would not cover. It is the feedback layer; which definition
manages a kind is decided by the ownership rule whatever it lets through.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-09-25 09:39:35 +03:00
IvanHunters 0b7dd23b99 fix(appdef): let one ApplicationDefinition own each application kind 
Nothing enforced that spec.application.kind is unique, and each consumer that
maps a kind back to a definition resolved a duplicate its own way: the
chartRef reconciler let every definition rewrite every release of its kind,
the lineage webhook and the CA projection reconciler took whichever came
first in an unordered list, and cozystack-api registered one resource per
definition. A second definition declaring a kind already in use could
therefore repoint that kind's tenant releases at its own artifact.

Add one ownership rule, the definition created first owns the kind with ties
broken by name, and route every consumer through it. A definition that
arrives later with a kind already in use no longer manages that kind's
releases whatever it is named: the chartRef reconciler skips it with a
Warning event and re-reconciles the next owner when the current one goes,
cozystack-api registers the owner only and logs the rest, its restart hash
moves when ownership moves, and the lineage and CA projection lookups
resolve the owner.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-09-25 09:39:35 +03:00
Aleksei Sviridkin 7a0aab2bdc refactor: apply the Go 1.27 modernizers 
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.

Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-24 17:23:18 +03:00