The store names and the short-term retention mirror metricsStorages in
the monitoring package, which an operator can rename or resize; they
are flags so the chart can follow such a change without a rebuild. The
cost limit defaults are the ones the metrics proposal settles on: 30d
range, 11000 points per series, 30s timeout.
Store names become part of a Service host name, so they are checked
as DNS labels at startup.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The flag now also bounds the body read and the token and access
reviews, and 0 lifts all of those at once; an operator raising it for
slow store queries needs to know that. The store comment named a
design document the repository does not have.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The package and serve comments described a delivery plan rather than
the code: the command named metrics it does not serve, and serve gave a
plan as the reason to refuse unknown roots before authenticating.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The default of 11000 buckets was justified by a metrics proxy that
does not exist on this branch. The number is Prometheus's cap on
points per series in a range query, which the metrics root reuses.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
An absent namespace resolves to the root store, and before the
informer's first list every namespace is absent, so a request served
in that window read a labeled namespace's logs from the wrong store.
Readiness keeps the Service from routing there, but a client that
addresses the pod directly is not stopped by it.
The resolver now fails while the cache is unsynced, which answers
503 like any other resolution error.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vlselect widens the window to step boundaries and returns a
zero-filled bucket for each one, so a start off a boundary yields one
bucket more than the span divided by the step. Sizing the step from
the unaligned span let hits and stats_query_range return MaxBuckets+1
buckets.
The minimum step now spans MaxBuckets-1 steps, and the smallest
accepted --logs-max-buckets rises to 4 so that this step still fits
the third-of-range cap that keeps alignment inside the window.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The flag parser stops at the first argument that is not a flag, so
a stray word in the container args made every flag after it fall back
to its default without an error, the root tenant included.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
NaN fails the negative check, as every comparison does, and +Inf
refills a bucket instantly, so either value switched the per-user rate
limit off while the flag looked set. Both now stop the proxy at
startup.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Startup refuses --logs-max-buckets below 3 and --logs-max-range below
3s, but only the refusals were tested, so moving either bound up by
one went unnoticed. Pin the smallest accepted pair.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The step is capped at a third of the maximum range so that vlselect's
alignment to whole steps cannot widen the window past that range. With
fewer than three buckets, or a range under three seconds, the minimum
step exceeds that cap and the bound no longer holds, so such settings
are refused at startup.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Zero is the documented way to disable a limit; a negative value
disabled it too, silently. Refuse it so a typo in the chart cannot turn
off a per-user or total cap unnoticed.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
A negative duration silently disables the bound it configures, so a
typo in the chart would turn off the request deadline or the tail idle
close without any sign. Refuse it when the flags are parsed.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The deploy unit builds an image and chart around this binary, so its
flags follow the agreed contract: listen and probe addresses, cluster
domain, root tenant and auth cache TTL, plus the logs limits. Defaults
are chosen so a stock install is safe without tuning; readiness waits
for the namespace cache because store resolution reads it on every
request. On shutdown open tails are cut after a grace period so their
clients reconnect with since to another replica.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
## What this PR does
Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.
```
docs/operations/backup-classes.md -> operations/services/backup-classes
docs/operations/resource-presets.md -> guides/resource-management
docs/storage-immutability.md -> operations/storage-class-immutability
docs/operations/system-memory-limits.md -> operations/configuration/system-memory-limits
docs/gateway-classes.md -> networking/gateway-classes
docs/observability/distributed-tracing.md -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md -> operations/services/object-storage/capacity-for-backups
```
`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.
References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.
Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.
### Downstream repositories
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763
Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.
Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.
### Release note
```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
Reconcile the tap digest-pin change with #5071 (no-install-on-connect), now
merged into main. Keep this branch's resetTapPinState, the verify-before-pull
Trust text, and the index-README trust model, while adopting #5071's wording
that installing (not tapping) runs charts and dropping the stale
"re-registration" phrasing from the shared tap.go and rest.go comments.
Conflicts resolved in cmd/cozypkg/cmd/tap.go, pkg/registry/core/tap/rest.go,
and hack/packages-index/README.md.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
A verified cosign signature whose manifest digest could not be parsed fell back
to an unpinned tag pull, which reopens the tag-mutation window the verification
was meant to close. Return an error instead.
Add a test that isolates the status.observedGeneration == metadata.generation
conjunct of artifactReady; the existing tests exercised only the tag-prefix
conjunct, so a mutation of the generation check went uncaught.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
With --certificate-identity and --certificate-oidc-issuer, `cozypkg tap` pulled by
tag without verifying and applied the artifact's PackageSources with
ForceOwnership; only the Flux source carried the verify policy. A tag moved by
someone without the signing identity could land unsigned content in the cluster.
Verify the signature at tap time and pin the verified digest, so the content
pulled and applied is what a trusted identity signed.
Flux evaluates a cosign matchOIDCIdentity issuer and subject as Go regexps, and
they were written unanchored. An exact identity such as alice@example.com would
also accept malice@example.com, and the dots would match any character. Anchor
and quote both values at the API and the CLI.
Reject --tag together with a digest reference instead of silently ignoring the
tag.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The index-entry `digest` field, and the gate and README claims around it, assume
a published index that does not exist yet: decision 0002 in the marketplace
proposal defers it, and #4981 tracks that work. The gate also never compared a
recorded digest against the artifact it verified, so the field added surface
without the check that would make it trustworthy.
Drop it from this PR. Digest pinning still works from an explicit
oci://host/org/repo@sha256:... reference and from the digest a tag resolves to;
only the index-entry path is removed. The tag-based pinning and the verify-flag
wording stay in the index README.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Removing on-connect auto-registration made several user-facing and doc texts
wrong: they promised that tapping registers a repository's apps in the catalog
and runs its charts. Tapping now installs nothing, so the texts told operators to
wait for apps that never appear.
Reword the `cozypkg tap`/`untap` help, the tap success output (now printing a
`cozypkg add <name>` line per materialized PackageSource), the `cozypkg init`
scaffold README, the packages-index trust model, the dashboard Repositories
banner and per-repository empty state, and the stale re-registration code
comments. No behaviour changes beyond the added `cozypkg add` hint in the tap
output.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
A re-tap applies the OCIRepository under field owner cozypkg via server-side
apply; a field this owner set earlier and now omits is removed. Re-tapping to
move the pin without repeating --certificate-identity/--certificate-oidc-issuer
therefore silently dropped the signature-verification policy, and re-tapping a
private repo without --secret dropped the pull credential. The CLI now carries
an existing source's spec.verify and secretRef onto the applied object when the
invocation does not set them, mirroring the API connect path, so moving the pin
never silently downgrades trust or breaks a private pull.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Update the tap command help and the community-index README for the new
behaviour: digest references, the automatic tag-to-digest pin, the
--certificate-identity / --certificate-oidc-issuer pull-time verify flags, and
the optional index entry digest. Extend the generated openapi definitions and
model-name helper for the new TapVerify type (code generation is otherwise
run in CI).
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
An index entry can now carry a digest (sha256:<64 hex>) alongside its version.
Resolving a short name pins that digest, so 'cozypkg tap <name>' consumes the
exact content the CI gate verified rather than following a mutable tag; the
version is kept informational. A malformed digest in an entry is a hard load
error, consistent with the rest of the strict index decoding.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
A tap can now carry a keyless cosign verification policy (spec.verify), so a
reference is only pulled when signed by the recorded identity. The CLI takes
--certificate-identity and --certificate-oidc-issuer (both required together,
mirroring validate); the API takes spec.verify.{issuer,subject} on the Tap
connect object. Both map to a Flux OCIRepository verify block with provider
cosign and a matchOIDCIdentity entry. A half-configured policy (only one of
the identity pair) is refused rather than set, since a policy that matched any
signer is worse than none. A key-based policy an admin sets on the source by
hand is preserved across repeat connects.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
A re-tap applies the OCIRepository under field owner cozypkg via server-side
apply, which cannot clear spec.ref.digest once the operator owns it. A
tag-based re-tap would therefore leave the source pinned to the old digest and
never follow the newly requested tag. After applying, the CLI now clears the
operator's digest pin and the pinned-digest annotation for a tag-based re-tap
(an explicit-digest re-tap keeps the digest it just set), alongside the
materialized-revision annotation, so the operator re-pins the new reference.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
buildTapOCIRepository now writes spec.ref.digest when the reference carries
one, and the pull resolves the artifact by digest so it fetches the exact
content regardless of where the tag points. A tag-only reference keeps being
pinned to a digest later by the tap materializer; an explicit @sha256
reference is pinned here at connect time, with any tag left informational.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
parseOCIRef (CLI) and parseConnectURL (API) now accept a pinned digest
reference (oci://host/org/repo@sha256:...), with an optional informational
tag before it. The digest must be a full sha256:<64 hex>, the same contract
the platform source's digest= reference enforces. A reference carrying only
a digest no longer defaults its tag to latest, since the digest is the
authoritative reference Flux resolves before any tag.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.
docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A drain deadline of zero or less is already spent when the drain
starts, so flux-plunger would exit holding every uninstall fence
without listing one. The chart could produce one: its 40-second floor
was a literal, so raising the manager's stop bound or the margin before
SIGKILL, which the drain subtraction is already tied to, would leave the
floor rendering a non-positive deadline. The binary now refuses such a
flag at startup, before it reaches for a cluster, and the chart's floor
and the timings the README and the values comment state are held to
the same constants.
A whole number of a million seconds or more printed in exponent form
and was refused as a fraction; the range is now checked first, so it
gets the range message.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
flux-plunger recovers a "has no deployed releases" HelmRelease by
briefly suspending it, deleting the stale storage Secret and
unsuspending. helm-controller v1.5.x skips the uninstall and drops its
finalizer when it handles a delete on a suspended HelmRelease, and it
wakes on the same delete event flux-plunger does, so a delete inside
that window orphaned the release's DataVolume, PVCs, RBAC and storage
Secret. Lifting the suspension on the delete event cannot win that race
reliably.
Hold the release with a flux-plunger finalizer while it is suspended.
The finalizer is applied together with the suspension in one server-side
apply. On a delete flux-plunger lifts its suspension and keeps the
finalizer, helm-controller reconciles the unsuspended generation and
runs the uninstall, and flux-plunger removes the finalizer once
helm-controller's own is gone and that generation is observed. This
holds whichever controller handles the delete first. A completed
recovery lifts the suspension and the finalizer together, and a
terminating release never begins recovery.
Decide ownership of a suspension by which field manager owns
spec.suspend in metadata.managedFields rather than by an annotation. The
apiserver transfers the field away the moment an operator, cozyhr resume
or a backup controller writes a different value to it, so a later
suspension is never misread as flux-plunger's, and one set through the
shared flux-client-side-apply manager (platform migrations, the tenant
delete hook, a manual kubectl apply) is never claimed. A release deleted
under another manager's suspension is left to that manager's decision.
Acquire applies without forcing first and, on a conflict, re-fetches and
forces only if spec.suspend is still false: a backup controller owns the
field at false through a dynamic-client Update, which a non-forced apply
can never take, while true is an active suspension to leave alone. Every
apply is preconditioned on the resourceVersion and uid of the read that
decided on it. Without the resourceVersion, the unforced apply of true
over a foreign true is accepted as shared ownership, a window the
optimistic-locked merge patch this replaces had closed, and lifting the
suspension and the finalizer from a release read as live would delete it
without an uninstall if helm-controller had meanwhile handled a delete.
Without the uid, an apply to a release that has gone would create an
empty one. The apply carries only spec.suspend and the finalizer, so it
cannot leak the required spec.interval.
Read the release, every re-check and the storage Secret list from the
apiserver rather than the informer cache. A lagging cache would let the
processed-version guard delete the revision below the one just
reclaimed, and the ClusterRole grants secrets get and list but not the
watch an informer needs.
Emit a Warning when flux-plunger holds a suspension it failed to clear,
and Normal events when it leaves an external suspension alone or defers
a recovery. An external suspension on a release flux-plunger has
recovered before is a Warning instead: it is most likely one an earlier
flux-plunger left under the shared manager, which nothing lifts unless
an operator does. A release deleted under another manager's suspension
is a Normal event when that was a migration's choice, and a Warning when
the release carries flux-plunger's processed-version annotation, since
then the skipped uninstall is an accident.
Keep a single writer with leader election. A second process able to act
under the same field manager could clear a suspension the first is still
recovering, and no rollout strategy prevents one: the next pod starts as
soon as the old one is terminating, and an eviction or a node drain
replaces a pod the same way. The old pod keeps its Lease, unreleased,
through its stop and drain, and the Lease outlasts both, so the next pod
acts only after the old one is done; the price is that a new pod waits
for the Lease to run out. The strategy sets maxSurge 0 with
maxUnavailable 1 and stays RollingUpdate because existing installs carry
the apiserver-defaulted rollingUpdate block and helm-controller applies
server-side, where type Recreate over it is rejected on upgrade.
Give the finalizer a way out when flux-plunger itself goes. Once the
manager has stopped, within a bounded time, and with it every reconcile,
flux-plunger drains its finalizers under a deadline that, with the stop,
fits inside the pod's termination grace period. A stop that overran its
bound is drained too and still reported, and only a pod that led drains.
The drain writes its events straight to the apiserver, since the
manager's recorder stops with the manager. A recovery in progress is
aborted, the suspension and the finalizer lifted together for the next
flux-plunger to start again, and a release being deleted has the
suspension lifted and keeps the finalizer until helm-controller has run
the uninstall. A release helm-controller does not handle in time keeps
it. The waiting lives in the controller, where the envtest suites hold
it against a real apiserver's generations and finalizers, rather than in
a shell script against a model of them.
A pre-delete hook on the chart is the backstop for what the drain could
not finish, so uninstalling the component never leaves a release that
tenant teardown waits on forever. It stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and finalizer without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, with a JSON patch preconditioned on the resourceVersion and
uid it read: kubectl apply --server-side on a release last written by a
client-side apply moves that apply's fields to the applying manager and
strips every one the body leaves out. The Job's deadline sits under the
300s helm-controller gives a pre-delete hook by default. A Deployment
that cannot be stopped or a listing that never succeeds ends the hook
before any finalizer is touched, and a finalizer it cannot remove ends
it after the sweep; either way it exits non-zero: a failed pre-delete
hook leaves the release untouched and helm-controller retries the
uninstall, hook included, from a state it can still converge from, where
exiting 0 would remove the component and strand the finalizer. A
flux-plunger pod still present after its grace period, typically one
stuck Terminating on a NotReady node, is only reported, since failing on
it would fail every retry until the node returns; because it may still
be setting finalizers, they are listed again after the sweep and any
found fails the hook. The Job runs once, since helm-controller's retry
is the one that starts afresh, and is kept after success so that its
report stays readable; helm.sh/hook-output-log-policy would make
helm-controller call a nil output writer (fluxcd/helm-controller#1562).
A delete still held after ten minutes raises a Warning, and the held
state is re-checked on a timer instead of only on helm-controller's next
status write. The chart README documents the finalizer and how to clear
it.
Run the envtest suites on every pull request through a new make target
that fetches a kube-apiserver and etcd with a pinned setup-envtest and
takes the HelmRelease CRD from the in-tree Flux manifests: the unit
tests drive a fake client, and everything the finalizer rests on (apply
preconditions, the finalizer set-merge, real managedFields) only holds
against a real apiserver. CI caches those binaries, and the target uses
them without asking the release index, so a pull request does not depend
on their sources being reachable; the control plane gets a longer start
timeout because it starts beside the other test goals.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
## What this PR does
System components now get a default container memory limit, so their pod
cgroups carry a `memory.max` and drop out of the Talos userspace OOM
handler's victim set. `cozystack-operator` maintains a LimitRange named
`cozystack-system-defaults` in every system namespace it reconciles.
The Talos handler (v1.12+) scores cgroups with `memory_max.hasValue() ?
0.0 : {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current`
and drops everything that scores zero. Only a memory limit sets
`memory.max`, so only a limit takes a pod out of the candidate set. A
request just moves a pod from BestEffort to Burstable, which under the
default `strictCgroupClassOrdering: true` still means killed second.
`PriorityClass` does nothing here, because the handler reads cgroupfs
and never talks to the apiserver. Victim selection is also unrelated to
the trigger, so a tenant workload thrashing against its own limit can
drive node memory pressure while a system component with no limit is
what gets killed.
An audit of a running cluster found around 70 workloads across ~30
packages with at least one container missing a memory limit. Per-chart
resources would not cover that set, and would not keep covering it as
packages are added. So the LimitRange is written by
`reconcileNamespaces`, which already creates these namespaces and
already tells system from tenant. It also covers init containers, which
#3495 can't reach: one of them without a limit leaves the pod cgroup
without `memory.max`, whatever the main containers declare.
Tenant namespaces are skipped. Where the tenant chart's own
`tenant-range-limits` exists (only when the operator configured
`resourceQuotas`), a second LimitRange next to it would leave the result
to the order LimitRanger iterates them. Where it doesn't exist, a tenant
namespace is not the place for the platform to impose a default.
### Why the default is 32Gi
The handler grants immunity for `memory.max` existing at all, whatever
its value. So the ceiling doesn't need to be fitted, and 32Gi is high
enough that nothing plausibly reaches it. A ceiling low enough to bind
would turn a rare pressure-driven kill into a deterministic one for any
component whose real usage nobody measured. This repo already has one
such case, recorded in the `vertical-pod-autoscaler` values, where a
110Mi limit OOM-killed the updater and failed HelmReleases with it.
Scheduling is unaffected, since pods are scheduled against requests and
the paired default request is 32Mi. An operator who wants the ceiling to
bind can lower it, per component in that component's chart or fleet-wide
with the knob, once real usage is measured.
### When the default is withheld
A default below a container's own request makes that pod impossible to
admit, so the operator checks before it applies. It scans Deployments,
StatefulSets, DaemonSets, CronJobs, Jobs and Pods in the namespace, init
containers included, for a container that requests more than the
configured limit and declares no limit of its own. If it finds one, it
withdraws the managed LimitRange and records a hold in the
`operator.cozystack.io/system-memory-state` annotation on the Namespace.
The hold does not clear by itself when the blocker disappears, because
an empty scan looks the same as a pod between deletion and recreation.
Lowering a previously applied limit also creates a hold, even with no
visible blocker, because LimitRanger writes the defaulted limit onto
admitted pods and the scan can no longer tell which containers declare
one. A cluster administrator releases the hold by copying the
`<hold>:<target>` token from the state into the
`operator.cozystack.io/system-memory-ack` annotation. The token is used
once and only for that hold and target. If a blocker is still visible,
the token is rejected and the hold stays. Package reconciles are
scheduled at least every five minutes, so an acknowledgement takes
effect on a quiet cluster too. The reason and the token also appear in
the operator log under `withholding the default container memory limit`.
### Namespace another LimitRange already governs
If a system namespace already has a LimitRange that says anything about
memory, at the Container or the Pod scope, the operator writes nothing
there and withdraws what it wrote before. A second default leaves the
effective value to iteration order. A foreign `max` below this default,
a `min` above the default request or a `maxLimitRequestRatio` rejects
pods outright, which takes out a whole namespace. At the Pod scope the
collision can't be decided in advance, because it depends on how many
containers a future pod has. A LimitRange that bounds only cpu or
storage coexists normally. A same-named LimitRange without the
operator's managed-by label is treated as foreign and left alone.
This matters most in `kube-system`, which is in scope because
`cozystack-scheduler` installs there, and where a distribution's or an
administrator's own LimitRange is likeliest to sit.
### Disabling
Setting `cozystackOperator.systemNamespaceMemoryLimit` to `0` disables
the feature. On the next reconcile the operator removes every LimitRange
it owns cluster-wide, found by the managed-by label. While the feature
is on, it also removes its LimitRange from namespaces no active Package
targets any more. The safety state on the Namespace stays in both cases.
### Docs
`docs/operations/system-memory-limits.md` landed in #3496 ahead of this
code, so main describes a `4Gi` default and installer values that do
nothing yet. The page is corrected to the 32Gi default and now describes
withholding, the acknowledgement, and the foreign LimitRange rule.
### Not in this PR
linstor policy. #3824 extends the controller's existing exemption from
limits to the satellite, and a linstor limit here would contradict it,
so linstor stays with that PR.
The scan reads six workload kinds through `APIReader`, uncached, on
every reconcile of every system namespace. That cost is tracked in
#3977, since the system namespace set isn't known when the cache is
built and the right fix depends on a reconcile-rate measurement nobody
has taken.
### Verification
`go vet` and `go test` pass for `cmd/cozystack-operator` and
`internal/operator`, and `helm unittest` passes on the installer chart
for the knobs unset, set, and the limit at `0`. The reconciler tests
cover withholding on an oversized request, the hold surviving the
blocker's removal, the hold on a lowered limit, acknowledgement and
token rejection, every memory field of a foreign LimitRange at both
scopes, ownership by label, the disabled sweep, and a deleting Package
not creating policy.
The withhold design has not been run on a live cluster yet. An earlier
design of this PR was, and that run does not carry over.
### Screenshots
Not a UI change.
### Downstream repositories
I walked the trigger map against the diff.
`packages/core/installer/values.yaml` gains two keys, but the
`ansible-cozystack` trigger is for a required key or a rename of one the
role passes, and these are new, optional and default to `""`, so the
role is unaffected. Nothing else in the map is reached by the Go
changes, the operator template or the page.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(operator): system namespaces now get a default container memory limit through a LimitRange maintained by cozystack-operator, which keeps system components out of the Talos userspace OOM handler's victim set. Tune it with cozystackOperator.systemNamespaceMemoryLimit (default 32Gi) and cozystackOperator.systemNamespaceMemoryRequest (default 32Mi); set the limit to 0 to disable it. The default is far above any real working set on purpose, because the handler grants immunity for a memory.max existing at all, not for its value. A namespace with a container that requests more than the limit and declares no limit of its own gets no default, and stays without one until a cluster administrator acknowledges the hold recorded on the Namespace (see docs/operations/system-memory-limits.md); lowering a previously applied limit needs the same acknowledgement. A namespace where another LimitRange already sets anything about memory is left alone. A container that declares its own memory limit is never changed.
```
## What this PR does
This adds `cmd/backport-audit`, a maintainer tool that checks, before a
release or an rc is cut, whether everything labelled for backport
actually landed on the release branch. It prints the URLs of the PRs
that did not and exits non-zero, so the cut can gate on it.
`docs/release.md` and `docs/agents/releasing.md` now send the patch-cut
check through it.
The docs it replaces inventoried backports with `gh pr list --search
"is:merged label:kind/backport"`. That lists every PR ever labelled, not
what reached the branch. Answering the real question by hand means
reading the branch history per PR, and that is how backports go missing
quietly.
```console
$ go build ./cmd/backport-audit
$ ./backport-audit release-1.4
=== release-1.4 === 30 candidate PRs: MISSING=5 dropped=4 backported=21
MISSING -- labelled for this line, no trace of it here (5):
https://github.com/cozystack/cozystack/pull/2704#2704 fix(info): use root-host for Keycloak OIDC issuer URL in tenant kubeconfig
label=kind/backport author=myasnikovdaniil merged=2026-05-26 -- no backport PR, nothing on branch
```
A `kind/backport` label does not name a branch. `backport.yaml` resolves
the target when the PR merges, so the same label means `release-1.5` on
a PR merged in June and `release-1.6` on one merged in August. The audit
reproduces that rule instead of trusting the label text. The rule
changed once, on 2026-08-03, when 53a7fc4dc made cutting an rc freeze
the line and 6ff5b98d5 pointed the bot at the branch list. Before that
date a line exists from its first stable release, after it from the
moment its `release-X.Y` branch is pushed. The audit estimates that
moment from the earliest `vX.Y.0-rc.N` tagged commit. This is an
estimate, because the commit timestamp is not the branch-push timestamp.
The README has the full reasoning.
Landing is checked per commit, because a commit is what a backport
loses: the bot's conflict drafts stop at the first commit that does not
apply. Every non-empty commit of the original needs its own evidence on
the branch. That is reachability, an `-x` reference, or a commit with
its subject among the branch commits that are not on main. Some but not
all is PARTIAL. A multi-commit original with a merged backport PR and no
commit evidence is UNVERIFIED. Both fail the gate. A maintainer can
confirm a squashed backport with a comment on the merged backport PR
that is exactly `backport-audit: complete`.
Two more sections start from the backport PRs on the branch instead of
from labels. DUPLICATE lists an original with more than one live
backport PR and fails the gate. UNLABELLED lists backports of originals
that are not labelled for this line. It is informational and does not
change the exit code.
Exit code 2 means the audit could not give an answer. That happens when
a `gh` listing reaches its cap, since `gh` truncates silently, or when a
git read that a verdict depends on fails. The label listings are checked
against the lower of `--limit` and 1000, because GitHub search never
returns more than 1000 results.
### Verification
`cmd/backport-audit/main_test.go` covers candidate selection, line
resolution and the evidence rules. `make unit-tests` runs it through a
new `test-backport-audit` target, since `./cmd/...` is not part of
`go-unit-tests`. `hack/release-freeze-contract.bats` pins the audit's
label set equal to the one `backport.yaml` acts on, in both directions.
### Limits
A MISSING verdict is a prompt to check, not proof of absence. A hand
backport that was squashed, reworded and never linked to its original is
invisible to every evidence layer. A label added after merge is read
against the line that was current at merge time, while the bot targets
the line current when the label is added. Such a backport shows as
MISSING on the old line and under UNLABELLED on the new one. The audit
reports that something is missing, never why. `docs/release.md` now
notes that before #3155 the bot left only a "Backport failed" comment on
a conflicting cherry-pick, which is the first thing to check behind a
MISSING.
`make generate` does not apply. This is a standalone `main` package with
no API types, no CRDs and no package `values.yaml`.
### Screenshots
Not a UI change. Sample CLI output is above.
### Downstream repositories
I walked the trigger map in `docs/agents/contributing.md` against the
diff file by file. The diff adds
`cmd/backport-audit/{main.go,main_test.go,README.md}` and changes
`Makefile`, `hack/release-freeze-contract.bats`, `docs/release.md`,
`docs/agents/releasing.md` and one `.gitignore` line for the root
binary.
Nothing in it reaches a downstream repository. No package under
`packages/apps/` or `packages/extra/` is added, renamed or removed, and
no `values.schema.json`, `values.yaml` default or version enum changes,
so the website generator lists and the terraform provider schemas are
untouched. No `ApplicationDefinition` semantics, `release.prefix`,
output Secret or Service name, namespace, platform variant or bundle
changes. The paths that reach ccp and external-apps-example
(`hack/package.mk`, `hack/common-envs.mk`, `hack/update-crd.sh`) are not
touched. The Makefile change adds a `test-backport-audit` target to
`unit-tests` and changes no existing target. No node prerequisite
changes, so talm and the ansible playbooks need nothing. The website
"developer tooling" trigger covers tools contributors use on packages.
This is maintainer release tooling, and its process doc lives in this
repository only.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(release): add `backport-audit`, a maintainer tool that reports which `kind/backport` / `kind/backport-previous` PRs have not landed on a release branch (missing outright, partially backported, waiting on an open backport PR, or deliberately dropped) and exits non-zero while anything is outstanding, so a patch cut can gate on it
```
An admitted Pod loses the distinction between explicit and defaulted
limits. Its disappearance cannot prove a lower default is safe for
its replacement. Keep that decision on the Namespace until an
administrator acknowledges the particular target, without waiving
current blockers or foreign policy ownership.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Keep the parser inputs tied to the constants registered as flag defaults,
but compare the results with literal 32Gi and 32Mi expectations. A shipped
default can no longer move while the defaults case stays green.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
List LimitRanges by the ownership label once per Package reconcile when
the feature is disabled, and remove managed objects even in namespaces
outside the current Package target set. Skip the per-namespace path while
disabled and keep cleanup failures non-fatal to namespace reconciliation.
Update the flag help, installer guidance, and operations documentation to
describe the global cleanup boundary.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The raise-and-grace machinery cannot work: LimitRanger writes the
raised limit onto the very container that justified the raise, so at
that container's next admission it carries a limit, the scan skips
containers that have one, and the raise has erased its own evidence.
The ceiling then drops on schedule and the workload is rejected
permanently. Any such machinery approximates admission logic ahead of
admission, and that is incomplete by construction: a
LimitRange default is one number for a whole namespace and cannot mean
"the configured limit, or this container's request, whichever is
larger". Only per-pod conditional defaulting expresses that, which is a
mutating webhook and not this.
The justification for the machinery had also expired. It was designed
when the default was 4Gi, where withholding was self-perpetuating for
VPA-managed namespaces. At 32Gi nothing in the tree approaches the
ceiling -- the largest static memory request is 2Gi -- and VPA sets
requests and limits together, so those containers carry their own limit
and were never blockers.
So a namespace holding a container that requests more than the
configured limit with no limit of its own is now left with no default at
all, and anything written there earlier is withdrawn. It fails in the
safe direction, which is the whole argument: a namespace with no
memory.max is a namespace behaving as it did in every release before
this, while a default below a request is a namespace whose workload
cannot be admitted. It needs no stored state and no timer, because
withholding is not a decision that has to survive between reconciles --
once the oversized request is gone the default is applied again on the
next pass.
Gone with it: the grace period, the clear-since annotation and its
clamp, the injected clock, containerMemoryDefault, raisedCeiling, and
fourteen tests. The scan stays, because deciding whether to apply or
withhold is what it was always for. The foreign-LimitRange ownership
guard, the disable path and the 32Gi default are untouched.
The residual sharp edge is stated rather than machinery: a workload
whose only evidence is a live Pod, built from a custom resource with no
template this scan can read, can still be missed if the scan runs while
that Pod is absent, and applying the default then rejects its
recreation. At 32Gi that is unreachable in this tree. It becomes
reachable if an operator lowers the knob below such a workload's
request, which the flag help, values.yaml and the operations page now
say plainly.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The --help text still carried two claims that values.yaml and the
operations page had already narrowed, and it is the one surface an
operator reads without opening the tree.
Both were wrong in the direction that costs the reader something. "A
namespace where another LimitRange already defaults or caps container
memory is left alone" described the guard before it was widened, so an
administrator holding a container memory min, a maxLimitRequestRatio or
any Pod-scoped bound would read it, expect the LimitRange to be applied
there anyway, and get a namespace with no memory.max for a reason the
help says does not apply. "Removes any previously created one" is the
universal removal claim, and removal is scoped to namespaces an active
Package still targets.
Two comments corrected while here. constrainsMemory said it switched on
the item's own maps "rather than a hand-picked few", which contradicts
the hand-picked list immediately below it; what is actually true is that
a field added upstream has to be added there too, which is the reason
it has its own table test. And memoryDefaultSettledAt claimed the count
of raised namespaces falls to none once the oversized requests are
gone, which stopped being true when the scan started keeping Failed
pods: a carcass Kubernetes never collects holds both the raise and that
namespace's per-reconcile pod List open.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The coexistence guard treated a namespace as already-governed only when
another LimitRange carried a container memory default or max, and said
so in its comment. That is not the set of fields that collide with what
this reconciler writes. A container min above the default request
rejects the pod, and so does a maxLimitRequestRatio, since a 32Gi
default against a 32Mi default request is a ratio of 1024:1 and no cap
allows it. So the namespace-wide admission failure the guard exists to
prevent stayed reachable through fields the guard did not read, and
kube-system is where it is likeliest, this feature having deliberately
brought it in scope.
Enumerating the colliding fields does not fix it, because at the Pod
scope the collision is not decidable here. Whether a per-container
default of 32Gi breaches a pod-wide max depends on how many containers
the pod has, and the pods in question do not exist yet: any arithmetic
about them is a guess, and a wrong guess costs a namespace its entire
pod admission. So the rule is ownership rather than arithmetic. If
anything else in the namespace has an opinion about memory, at either
scope and in any field, the namespace is not this reconciler's to
default. Narrow still on the resource, because nothing here writes cpu
or storage and a LimitRange bounding those must keep coexisting. This
is the rule the tenant-namespace exclusion already rests on.
The pod scan no longer skips Failed pods, only Succeeded. A pod that
ran to completion is never recreated from its spec, but an eviction or
a node crash leaves a Failed carcass holding the same oversized request
that its controller will recreate, and that carcass is frequently the
only surviving evidence the namespace needs a raised ceiling, since the
shape the scan looks for cannot be admitted once the namespace has
settled. Dropping it lowered the ceiling under a workload that was
coming back, which is what the grace period exists to prevent.
A grace stamp dated ahead of the clock is clamped to now. Unclamped it
held the ceiling for the skew plus the grace period, silently, so an
hour of skew bought ninety minutes instead of thirty. Clamping never
shortens the wait, it only stops it being extended.
The two flag defaults are named constants the flag registration and its
test both read. The test previously duplicated the literal and still
asserted 4Gi after the default moved to 32Gi, so it misinformed a
reader about the shipped value and pinned nothing.
The disable claim is narrowed to what the code guarantees. "Removes any
previously created one" was universal, but the reconciler only runs for
namespaces an active Package still targets, so a namespace whose
Package was removed keeps its LimitRange. The flag help, values.yaml
and the operations page now say that, and name the managed-by label for
finding leftovers by hand.
The test asserting a Pod-scoped memory max should be ignored is gone,
since under the ownership rule a Pod-scoped memory bound withholds.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The guard in front of this default reads declared memory requests. It
cannot see a container's real working set, so a system component that
requests little but legitimately uses more than the ceiling, and
declares no limit of its own, was handed a deterministic kernel OOM on
its next recreation in exchange for leaving the Talos handler's victim
set. Since the feature ships enabled, that trade fell on the whole
fleet on upgrade.
It is not a trade this feature has to make. The handler discards a
cgroup for carrying a memory.max at all, whatever its value, so
immunity does not require a fitted ceiling -- only a real one. A
default far above any plausible working set buys the immunity and
leaves nothing for the ceiling to bind on, which also keeps it clear of
the failure mode this repository already carries a scar from, where an
undersized limit converted a rare pressure-driven kill into a reliable
one.
Nothing about scheduling changes: only requests are scheduled against,
and the paired default request stays 32Mi. An operator who wants the
ceiling to bind lowers it, per component in a chart or fleet-wide with
this knob, once real usage is measured -- which is the direction that
needs evidence, and now the direction that needs it rather than the
default.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The scope of the default LimitRange is every namespace a Package targets
whose name does not start with `tenant-`, and cozystack-scheduler installs
into kube-system, so a namespace owned by the underlying Kubernetes
distribution is defaulted like any other. Neither the flag help nor the
installer value said so, and both are what somebody setting the knob
reads.
docs/operations/system-memory-limits.md already covers this, including the
part that surprises: containers the kubelet builds from an on-disk
manifest never pass through admission, so a Talos control-plane static pod
gets no memory.max from this feature in the first place and the mirror pod
is the only thing the LimitRange reaches. Both notes now carry the same
statement, so neither reading of the knob leaves it out.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
parseQuantity and the request-above-limit check were closures inside
main(), so no branch of either was reachable from a test: not empty, not
malformed, not negative, not a request of 0 against a real limit, and not
the pair check itself. That last one is the one worth having. A limit and
a request that each parse but contradict each other produce a LimitRange
the API server rejects, once per system namespace on every Package
reconcile, and nothing in that error points back at a flag.
Lift both into parseSystemNamespaceMemory, which returns an error instead
of calling os.Exit, and pin all of it: the shipped defaults, empty as the
off switch, an explicit 0 limit beside a set request, a 0 request against
a real limit (supported on purpose), malformed and negative values on
either flag, request above limit, and request equal to limit as the legal
boundary. main() keeps the same exit-on-error behaviour, now in one place.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The flag help and the installer values comment both still said the
operator skips a namespace holding a request above the configured limit
and retracts any LimitRange it applied there. It stopped doing that in
3127c0fad: it raises that namespace's ceiling to clear the request and
drops back once the request is gone. An operator reading the shipped help
and then finding an 8Gi LimitRange in cozy-monitoring had no way to
connect the two.
Both texts now describe the raise, the drop-back and the log line to grep
for. The values comment also loses its VPA warning, which this branch's
own comment in findRequestAboveDefaultLimit refutes — VPA rescales the
limit in proportion to the request under the default controlledValues, so
it cannot be what puts a namespace over — and gains a pointer to
declaring a limit in the component's own chart, which is how
packages/system/linstor now stays out of the default's reach.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
--system-namespace-memory-request=0 passes the startup check, since 0 is
not greater than the limit, and produces a LimitRange with
defaultRequest: 0. Whether that is a footgun to reject or a supported
setting to explain was open, so I asked a real API server: envtest with
kube-apiserver 1.36.0 and LimitRanger enabled, with a second namespace
using 32Mi as the positive control that the plugin was actually running.
It is supported. The API server accepts the LimitRange and stores
defaultRequest as 0 rather than folding it back to the default -- the
Default-to-DefaultRequest copy only fills keys that are absent, and this
one is present. A container declaring no memory of its own is then
admitted with an explicit request of 0 and a limit of 4Gi, QoS Burstable.
So LimitRanger does not read zero as absent, and the container still gets
memory.max on its cgroup, which is the whole reason this feature exists.
What it costs is the scheduling signal: nothing is reserved for that
container, which is exactly where system components stood before this
feature. Say so in the flag help text, where an operator setting the knob
will see it, and note at the startup check that 0 passing is deliberate so
nobody hardens it into a rejection later.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.
Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.
An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.
The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.
Two things keep the feature from being able to hurt more than it helps.
A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.
And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Backport labels record requests, and a merged conflict draft may still
lack commits. Release cuts need a check of what reached the branch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
This adds three minimal platform variants for small installs, such as
arm64 labs, where isp-full is too heavy. They are `isp-slim` for Talos,
`isp-slim-generic` for k3s, kubeadm or RKE2, and `isp-hosted-slim` for
clusters where the host provides CNI and storage.
A slim variant installs only the base platform: the engine, API and
dashboard, tenants, ingress and Gateway API. The two non-hosted variants
also get LINSTOR. objectstorage-controller stays on too, because
cozystack-controller watches BucketClaim unconditionally and crash-loops
without that CRD. Everything else is opt-in through
`bundles.enabledPackages`, including every paas and naas application and
its operator, monitoring, backups, etcd, SeaweedFS, metrics-server, VPA,
Multus and MetalLB. The iaas bundle is refused, so there are no VMs and
no managed Kubernetes. With the stock preset a slim variant emits 20
Packages (15 for hosted), where isp-full emits 77.
On the two non-hosted slim variants, networking is Cilium alone. Without
VMs nothing needs Kube-OVN, and Cilium L2 announcements take the place
of MetalLB. The admin creates a `CiliumLoadBalancerIPPool` and a
`CiliumL2AnnouncementPolicy`, or uses `publishing.externalIPs`. Pod
ranges come from `node.spec.podCIDR`, which Talos and k3s allocate by
default. `networking.encryption` only drives Kube-OVN IPsec, so the slim
variants refuse it instead of silently ignoring it.
The tenant application needs its own slim variant. The default one waits
on the monitoring, etcd and SeaweedFS applications, so on slim
cozystack-basics and tenant-root would never become ready. tenant-rd and
cozystack-basics still reference the default variant's artifacts by
name. Those artifacts exist anyway, because the PackageSource builds
them for every variant.
The full variants don't change. I rendered isp-full, isp-full-generic
and isp-hosted against main and got byte-identical Packages. The only
new output there is the slim variant in the tenant-application
PackageSource.
Things to know before using it:
- An opt-in package does not pull in its dependencies. The presets and
the docs list the chains. A package enabled without its chain stays
`DependenciesNotReady`.
- Slim is for new installs. Switching a live isp-full cluster to
isp-slim moves the networking Package from `kubeovn-cilium` to `cilium`,
the operator removes the Kube-OVN release, and running pods lose
networking. The other Packages are kept by `helm.sh/resource-policy:
keep`, so the switch doesn't make the cluster smaller either.
- No e2e suite runs a slim variant yet. Helm unit tests cover the
presets, opt-in, OIDC, the networking values and both refusals, and a
dependency check over every emitted Package passes for all three
presets.
I installed isp-slim from a build of this branch on a fresh three-node
Talos 1.13 cluster (amd64). All Packages and HelmReleases went Ready,
with no Kube-OVN, MetalLB, monitoring, backup or KubeVirt pods. From
outside, the dashboard and the API answered by name with valid
certificates. A LoadBalancer Service got an address from a Cilium pool
and answered through L2 announcements. Pod-to-pod traffic across nodes,
cluster DNS and a replicated LINSTOR volume worked. Opting in
postgres-operator and postgres-application gave a Ready Postgres. That
run found the BucketClaim crash-loop above, which is fixed here. The
base platform used about 215m CPU and 3.2 GiB of memory, not counting
the Kubernetes control plane.
### Screenshots
Not a UI change.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#718
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [x]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up: cozystack/ansible-cozystack#79
- [x] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
cozystack/ccp#23
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(platform): add the isp-slim, isp-slim-generic and isp-hosted-slim variants. They install only the base platform (engine, dashboard, tenants, ingress, gateway, and LINSTOR outside hosted) with Cilium-only networking and no MetalLB; applications, operators, monitoring, backups, etcd and SeaweedFS are opt-in through bundles.enabledPackages, and the iaas bundle is not available.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added the `isp-slim`, `isp-slim-generic`, and `isp-hosted-slim`
platform variants, with support for PaaS and NaaS bundles.
* Slim variants use Cilium networking and include a reduced set of
packages by default. Additional packages and their dependencies can be
enabled as needed.
* **Limitations**
* IaaS and the platform encryption toggle are not supported on slim
variants. Cilium-native encryption is not configured.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
A CNPG `Backup` taken while the Postgres app still accepted
`users[].password` copied that password into
`status.underlyingResources`, where anyone with `get backups` in the
namespace can read it. #4078 stopped new snapshots from carrying the
field but left every existing `Backup` as it was (#4179).
This adds `LegacyPasswordScrubber`, a leader-elected runnable in the
backup strategy controller that, on start, lists `Backup` objects and
removes `users.*.password` from every snapshot whose `kind`/`apiVersion`
is the CNPG snapshot's.
- **Why on every leader start and not a platform migration.** The
migration hook runs pre-upgrade, while the old controller is still
leader, so a `Backup` the old replica writes during the rollout would
keep its password and the migration never re-runs. With `--leader-elect`
on, the old replicas stop writing before the new leader's scrub runs,
and a `Backup` brought back from an object-level export is cleaned on
the next start too. On a clean cluster the whole pass is one List from
the manager cache, which `BackupReconciler` already populates.
- **Write shape.** The patch is a merge patch computed from the diff, so
the request body only nulls the `password` keys and cannot overwrite
status another writer changed since the List. The snapshot is handled as
generic JSON (numbers kept as `json.Number`), so fields the current
snapshot type does not know survive.
- **Failure handling.** A `Backup` that cannot be decoded or patched is
logged and skipped, and a failed List is logged too; `Start` always
returns nil, because a returned error would stop the manager and take
every backup down over one object.
- **Restore compatibility.** Restore already ignored the key; a test
decodes a scrubbed snapshot through `unmarshalCNPGBackupSnapshot` to pin
that.
- RBAC: `list` and `patch` on `backups.cozystack.io/backups` are already
granted to the controller, so the chart is unchanged.
- `docs/operations/backup-classes.md` no longer tells operators to
delete pre-upgrade `Backup` objects, and says what the scrub does not
reach (audit logs, etcd backups, external exports of the objects, Helm
history of the init-script Secret), so a password once set through
`users[].password` is still treated as exposed until rotated.
The runnable carries a `TODO(#4179)` to remove it once upgrading from a
release that accepted `users[].password` is no longer supported.
Verification: `go test ./internal/backupcontroller/...` and `go build
./cmd/backupstrategy-controller/` are green. The new tests cover
scrubbing with everything else kept (including an unknown field with a
large integer), leaving non-CNPG, other-apiVersion, clean and
snapshot-less Backups unpatched, and continuing past a Backup that fails
to decode or patch. Disabling the delete, or returning on the first
patch error, makes the tests fail. Not exercised on a live cluster yet.
Fixes#4179
### Screenshots
N/A — no UI changes.
### Downstream repositories
Walked the trigger map against the diff: it touches only
`internal/backupcontroller`, `cmd/backupstrategy-controller` and the
in-repo `docs/operations/backup-classes.md`, with no package, values,
schema, CRD, variant or asset change. The website carries no "delete
pre-upgrade Backups" workaround this would make obsolete; its stale
`users[name].password` rows are the #4078 follow-up already noted there.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(backups): the backup strategy controller now removes plaintext `users[].password` values left in `status.underlyingResources` of CNPG `Backup` objects taken before managed Postgres stopped accepting user passwords. It runs on every controller start and needs no action. It does not reach copies outside the cluster (audit logs, etcd backups, exported `Backup` objects), so a password once set through `users[].password` should still be treated as exposed until it is rotated.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Security**
* At controller startup, plaintext user passwords are removed from
matching CNPG backup snapshots. Other snapshot data is preserved.
* This cleanup does not affect external copies or remove older
init-script manifests from Helm history. Treat passwords previously set
through `users[].password` as exposed until rotated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add isp-slim, isp-slim-generic and isp-hosted-slim to the platform
PackageSource so a Package can select the minimal presets shipped by
the platform chart.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>