Commit Graph
146 Commits
Author SHA1 Message Date
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Myasnikov Daniil 5b585d6eda fix(operator): persist holds for unsafe memory defaults
An admitted Pod loses the distinction between explicit and defaulted
limits. Its disappearance cannot prove a lower default is safe for
its replacement. Keep that decision on the Namespace until an
administrator acknowledges the particular target, without waiving
current blockers or foreign policy ownership.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:51 +03:00
Myasnikov Daniil 30ecc41070 fix(operator): close memory policy safety gaps
Preserve foreign same-name objects and keep create-time SSA non-force.

Only ignore failed pods controlled by terminal Jobs.

Make the failure-containment test inject Apply errors for real.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:51 +03:00
Myasnikov Daniil 6a521f0c68 fix(operator): sweep memory defaults when disabled
List LimitRanges by the ownership label once per Package reconcile when
the feature is disabled, and remove managed objects even in namespaces
outside the current Package target set. Skip the per-namespace path while
disabled and keep cleanup failures non-fatal to namespace reconciliation.

Update the flag help, installer guidance, and operations documentation to
describe the global cleanup boundary.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:49 +03:00
Myasnikov Daniil f6dcb69b61 refactor(operator): withhold instead of raising, and drop the grace period
The raise-and-grace machinery cannot work: LimitRanger writes the
raised limit onto the very container that justified the raise, so at
that container's next admission it carries a limit, the scan skips
containers that have one, and the raise has erased its own evidence.
The ceiling then drops on schedule and the workload is rejected
permanently. Any such machinery approximates admission logic ahead of
admission, and that is incomplete by construction: a
LimitRange default is one number for a whole namespace and cannot mean
"the configured limit, or this container's request, whichever is
larger". Only per-pod conditional defaulting expresses that, which is a
mutating webhook and not this.

The justification for the machinery had also expired. It was designed
when the default was 4Gi, where withholding was self-perpetuating for
VPA-managed namespaces. At 32Gi nothing in the tree approaches the
ceiling -- the largest static memory request is 2Gi -- and VPA sets
requests and limits together, so those containers carry their own limit
and were never blockers.

So a namespace holding a container that requests more than the
configured limit with no limit of its own is now left with no default at
all, and anything written there earlier is withdrawn. It fails in the
safe direction, which is the whole argument: a namespace with no
memory.max is a namespace behaving as it did in every release before
this, while a default below a request is a namespace whose workload
cannot be admitted. It needs no stored state and no timer, because
withholding is not a decision that has to survive between reconciles --
once the oversized request is gone the default is applied again on the
next pass.

Gone with it: the grace period, the clear-since annotation and its
clamp, the injected clock, containerMemoryDefault, raisedCeiling, and
fourteen tests. The scan stays, because deciding whether to apply or
withhold is what it was always for. The foreign-LimitRange ownership
guard, the disable path and the 32Gi default are untouched.

The residual sharp edge is stated rather than machinery: a workload
whose only evidence is a live Pod, built from a custom resource with no
template this scan can read, can still be missed if the scan runs while
that Pod is absent, and applying the default then rejects its
recreation. At 32Gi that is unreachable in this tree. It becomes
reachable if an operator lowers the knob below such a workload's
request, which the flag help, values.yaml and the operations page now
say plainly.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:48 +03:00
Myasnikov Daniil 81c12bc953 fix(operator): withhold on any foreign memory policy, not two fields
The coexistence guard treated a namespace as already-governed only when
another LimitRange carried a container memory default or max, and said
so in its comment. That is not the set of fields that collide with what
this reconciler writes. A container min above the default request
rejects the pod, and so does a maxLimitRequestRatio, since a 32Gi
default against a 32Mi default request is a ratio of 1024:1 and no cap
allows it. So the namespace-wide admission failure the guard exists to
prevent stayed reachable through fields the guard did not read, and
kube-system is where it is likeliest, this feature having deliberately
brought it in scope.

Enumerating the colliding fields does not fix it, because at the Pod
scope the collision is not decidable here. Whether a per-container
default of 32Gi breaches a pod-wide max depends on how many containers
the pod has, and the pods in question do not exist yet: any arithmetic
about them is a guess, and a wrong guess costs a namespace its entire
pod admission. So the rule is ownership rather than arithmetic. If
anything else in the namespace has an opinion about memory, at either
scope and in any field, the namespace is not this reconciler's to
default. Narrow still on the resource, because nothing here writes cpu
or storage and a LimitRange bounding those must keep coexisting. This
is the rule the tenant-namespace exclusion already rests on.

The pod scan no longer skips Failed pods, only Succeeded. A pod that
ran to completion is never recreated from its spec, but an eviction or
a node crash leaves a Failed carcass holding the same oversized request
that its controller will recreate, and that carcass is frequently the
only surviving evidence the namespace needs a raised ceiling, since the
shape the scan looks for cannot be admitted once the namespace has
settled. Dropping it lowered the ceiling under a workload that was
coming back, which is what the grace period exists to prevent.

A grace stamp dated ahead of the clock is clamped to now. Unclamped it
held the ceiling for the skew plus the grace period, silently, so an
hour of skew bought ninety minutes instead of thirty. Clamping never
shortens the wait, it only stops it being extended.

The two flag defaults are named constants the flag registration and its
test both read. The test previously duplicated the literal and still
asserted 4Gi after the default moved to 32Gi, so it misinformed a
reader about the shipped value and pinned nothing.

The disable claim is narrowed to what the code guarantees. "Removes any
previously created one" was universal, but the reconciler only runs for
namespaces an active Package still targets, so a namespace whose
Package was removed keeps its LimitRange. The flag help, values.yaml
and the operations page now say that, and name the managed-by label for
finding leftovers by hand.

The test asserting a Pod-scoped memory max should be ignored is gone,
since under the ownership rule a Pod-scoped memory bound withholds.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:45 +03:00
Myasnikov Daniil 9fe93a3d7e docs(operator): fix three claims that overstated what the code does
The static-pod note said containers built from an on-disk manifest are
unaffected "in both directions". True of the container, which the
kubelet runs from disk whatever the API says, but the mirror pod is an
ordinary API object that passes admission like any other, so a static
pod declaring a memory request above the ceiling with no limit of its
own has its mirror pod rejected -- leaving the container running with no
representation in the API. Worth naming, since kube-system is in scope
by design. The scan does cover it: a mirror pod is a live Pod.

The ReplicaSet exclusion was justified on live ones being copies of
their Deployment's template. That does not reach an ownerless
ReplicaSet, which at replicas 0 has neither pods to be seen through nor
a Deployment template standing in for it. It stays out of scope --
nothing here ships a bare ReplicaSet -- but as a gap left open rather
than one the stated reason closes.

The delete guard's comment claimed the enabled apply always adopts and
stamps a same-named foreign object. It does not when that object's spec
already matches, because then the apply is skipped entirely. The
outcome is the same one the guard gives on purpose, but it is reached by
the apply not running.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:54 +03:00
Myasnikov Daniil e131a49514 feat(operator): default the system namespace memory ceiling to 32Gi
The guard in front of this default reads declared memory requests. It
cannot see a container's real working set, so a system component that
requests little but legitimately uses more than the ceiling, and
declares no limit of its own, was handed a deterministic kernel OOM on
its next recreation in exchange for leaving the Talos handler's victim
set. Since the feature ships enabled, that trade fell on the whole
fleet on upgrade.

It is not a trade this feature has to make. The handler discards a
cgroup for carrying a memory.max at all, whatever its value, so
immunity does not require a fitted ceiling -- only a real one. A
default far above any plausible working set buys the immunity and
leaves nothing for the ceiling to bind on, which also keeps it clear of
the failure mode this repository already carries a scar from, where an
undersized limit converted a rare pressure-driven kill into a reliable
one.

Nothing about scheduling changes: only requests are scheduled against,
and the paired default request stays 32Mi. An operator who wants the
ceiling to bind lowers it, per component in a chart or fleet-wide with
this knob, once real usage is measured -- which is the direction that
needs evidence, and now the direction that needs it rather than the
default.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:53 +03:00
Myasnikov Daniil b6cb0def78 docs(operator): say that kube-system is in scope for the memory default
The scope of the default LimitRange is every namespace a Package targets
whose name does not start with `tenant-`, and cozystack-scheduler installs
into kube-system, so a namespace owned by the underlying Kubernetes
distribution is defaulted like any other. Neither the flag help nor the
installer value said so, and both are what somebody setting the knob
reads.

docs/operations/system-memory-limits.md already covers this, including the
part that surprises: containers the kubelet builds from an on-disk
manifest never pass through admission, so a Talos control-plane static pod
gets no memory.max from this feature in the first place and the mirror pod
is the only thing the LimitRange reaches. Both notes now carry the same
statement, so neither reading of the knob leaves it out.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:51 +03:00
Myasnikov Daniil dd36eeddbc docs(operator): describe the raise, not the retract, in the shipped help
The flag help and the installer values comment both still said the
operator skips a namespace holding a request above the configured limit
and retracts any LimitRange it applied there. It stopped doing that in
3127c0fad: it raises that namespace's ceiling to clear the request and
drops back once the request is gone. An operator reading the shipped help
and then finding an 8Gi LimitRange in cozy-monitoring had no way to
connect the two.

Both texts now describe the raise, the drop-back and the log line to grep
for. The values comment also loses its VPA warning, which this branch's
own comment in findRequestAboveDefaultLimit refutes — VPA rescales the
limit in proportion to the request under the default controlledValues, so
it cannot be what puts a namespace over — and gains a pointer to
declaring a limit in the component's own chart, which is how
packages/system/linstor now stays out of the default's reach.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:47 +03:00
Myasnikov Daniil 546d8e6a54 feat(operator): default system namespace memory via LimitRange
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.

Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.

An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.

The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.

Two things keep the feature from being able to hurt more than it helps.

A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.

And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:41 +03:00
Andrei Kvapil c3cead8f64 fix(telemetry): make the documented opt-out cover both collectors
Telemetry is reported by two components, cozystack-operator (cluster facts)
and cozystack-controller (application counts), but only the operator had a
reachable switch. The documented opt-out sets
cozystackOperator.disableTelemetry on the cozy-installer chart, which cannot
reach a package the platform deploys, so cozystack-controller kept reporting
after an operator had followed the procedure — silently, with nothing to
indicate the opt-out was partial.

The platform now exposes telemetry.disabled and threads it into the
cozystack-controller component of the cozystack-engine Package, the same way
the bundle already threads cilium, multus and linstor values. The installer
key is also declared in values.yaml: it was only ever referenced from the
template, so the documented --set target did not appear in the chart's own
values.

Both keys stay false by default; this changes no behaviour for anyone who has
not opted out.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-08-12 19:48:37 +02:00
cozystack-ci[bot] 235d38dae1 Prepare release v1.6.0 (promoted from v1.6.0-rc.4)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-07-22 13:00:10 +00:00
cozystack-ci[bot] d95685c427 Prepare release v1.6.0-rc.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-07-21 10:31:23 +00:00
Myasnikov DaniilandClaude Opus 4.8 fd7fcbda4e refactor(operator): read platform version from env (decouple from build)
The operator's platform version was baked into the image at build time
(`-ldflags -X pkg/version.Version` from `git describe`), so a
`vX.Y.Z-rc.N` image literally carried the rc string. That blocks promoting
a release-candidate to stable by retag: the bits would still self-report
the rc version.

Resolve the version from the COZYSTACK_VERSION env var at runtime, falling
back to the build-time value when unset (local builds still work). The
installer chart threads `cozystackOperator.platformVersion` into the
operator Deployment as that env var, and `make manifests` stamps it into
the kubectl-apply release artifacts. The same image bits can now report any
release name, so promotion needs no rebuild.

The only behavioral reader is the telemetry metric
`cozy_cluster_info{cozystack_version=...}`; cozypkg/check-readiness keep
their own build-time Version and are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-06 10:03:46 +05:00
Ivan Okhotnikov fff9e827d1 fix(operator): address review feedback on private platform source
- Run `go mod tidy` to move github.com/fluxcd/pkg/apis/meta from
  indirect to direct require: it became a direct import after the
  SecretRef type fix.
- Refine the --platform-source-secret flag docstring and the matching
  values.yaml comment: drop identity.pub from the SSH required-keys
  list (Flux source-controller only consumes identity + known_hosts)
  and mention bearerToken as an alternative to username+password for
  Git over HTTPS.
- Note in values.yaml that the Secret must exist in cozy-system before
  the operator reconciles; otherwise source-controller reports a
  failed condition until the Secret is created.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-06-24 23:22:54 +03:00
IvanHunters 0427b184fa feat(operator): support private OCI registry credentials for platform source
Added --platform-source-secret flag to cozystack-operator that allows
specifying a Secret name for registry authentication. This enables using
private OCI registries and Git repositories as platform sources.

Changes:
- cmd/cozystack-operator/main.go:
  - Add --platform-source-secret flag
  - Pass secretName to generateOCIRepository() and generateGitRepository()
  - Set spec.secretRef in generated resources when secret is specified

- packages/core/installer/values.yaml:
  - Add cozystackOperator.platformSourceSecret parameter

- packages/core/installer/templates/cozystack-operator.yaml:
  - Pass --platform-source-secret flag to operator container

Usage:
1. Create registry credentials secret in cozy-system namespace:
   kubectl create secret docker-registry my-registry-creds \
     --namespace cozy-system \
     --docker-server=registry.example.com \
     --docker-username=user \
     --docker-password=token

2. Configure Helm values:
   cozystackOperator:
     platformSourceUrl: 'oci://registry.example.com/repo/packages'
     platformSourceSecret: 'my-registry-creds'

Closes #2455

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-06-24 22:54:06 +03:00
cozystack-ci[bot] e1aaa90ed3 Prepare release v1.5.0
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-06-22 10:55:37 +00:00
cozystack-ci[bot] 9f655e29ff Prepare release v1.4.0-rc.2
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-05-13 21:48:39 +00:00
cozystack-ci[bot] 252c18f3e8 Prepare release v1.4.0-rc.1
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-05-13 12:29:01 +00:00
myasnikovdaniil 7520967d8d feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) (#2509)
## Why

The `cozystack-operator` Package reconciler builds every `HelmRelease`
it generates with a handful of hardcoded values. Three of those have
caused recurring pain across the project:

- **`Spec.Interval = 5m`** (line 220 before this PR) — too long for E2E
install where dependent HRs reconcile out of order; previously addressed
by a manual `flux reconcile --force` workaround in
`hack/e2e-install-cozystack.bats`.
- **`Install.Remediation{Retries: -1}` + `Upgrade.Remediation{Retries:
-1}`** — semantically "retry forever, never remediate", but the
retry-on-failure timing was tied to `Spec.Interval` (5m). Audit of the
last 30 successful PR runs found that **every** run hit `kubectl wait
hr/seaweedfs-system` timeouts at the 2-min mark, recovering only after
the inline `flux reconcile --force` workaround.
- **`Install.Timeout = 10m` / `Upgrade.Timeout = 10m`** — directly
contradicts the per-Application timeout work in `pkg/config/config.go` +
`pkg/registry/apps/application/rest.go` (commit `7b146cbe`).
`packages/apps/tenant/templates/etcd.yaml` hand-rolls its HR YAML to
bypass this for cert rotation; harbor has had multiple commits chasing
the same problem.
- **`Spec.MaxHistory`** unset → Helm default 5. Test clusters bounce HRs
frequently and accumulate release-history Secrets.

This PR exposes all four as operator flags + helm chart values, with
defaults that preserve current behavior.

## What

### New operator flags

| Flag | Default | HR field |
|---|---|---|
| `--helmrelease-interval` | `5m` | `Spec.Interval` |
| `--helmrelease-retry-interval` | `30s` |
`Spec.{Install,Upgrade}.Strategy.RetryInterval` |
| `--helmrelease-install-timeout` | `10m` | `Spec.Install.Timeout` |
| `--helmrelease-upgrade-timeout` | `10m` | `Spec.Upgrade.Timeout` |
| `--helmrelease-max-history` | `5` | `Spec.MaxHistory` |

### Strategy switch

`Install.Strategy.Name` and `Upgrade.Strategy.Name` are now set to
`RetryOnFailure` (instead of leaving the default `RemediateOnFailure`),
with `RetryInterval` set from the new flag. Functionally equivalent to
the previous `Remediation{Retries:-1}` ("retry forever") since
remediation never fired anyway, but **decouples retry-on-failure timing
from `Spec.Interval`** so failed releases recover at 30s without polling
healthy releases at the same fast cadence.

### Chart exposure

`packages/core/installer/values.yaml` gets five new optional values
(`cozystackOperator.helmRelease{Interval,RetryInterval,InstallTimeout,UpgradeTimeout,MaxHistory}`),
each defaulting to empty so the operator's own default applies. The
chart conditionally renders the corresponding flag only when the value
is set.

## Production safety

- All static defaults match the previous hardcoded values (5m / 10m /
10m / 5). No new code path runs until an admin explicitly sets a flag.
- **One intentional behavior change: failed-release retry cadence**
drops from ~5m (previously coupled to `Spec.Interval`) to 30s
(`RetryInterval` default). This is the whole motivation for the strategy
switch — the recurring `kubectl wait hr/seaweedfs-system` flakes
documented above were caused by the previous coupling. Healthy-release
reconcile cadence is unchanged at 5m in production; CI overrides this to
30s via `--set cozystackOperator.helmReleaseInterval=30s` in
`hack/e2e-install-cozystack.bats` for faster convergence on E2E
clusters.
- `RetryOnFailure` is functionally equivalent to `RemediateOnFailure +
Retries:-1` since neither ever triggers remediation.
- Empty value in the chart → flag not rendered → operator falls back to
its built-in default. (`helmReleaseMaxHistory: 0` — unlimited history
per Helm semantics — is also honored; the chart guards on `ne (toString
…) ""` rather than truthiness so int `0` is not silently dropped.)

## Verification

- `go build ./...` clean
- `go test ./internal/operator/...` passes (covers `Interval`,
`MaxHistory`, `Install/Upgrade.Timeout`, `Strategy.Name`,
`RetryInterval`, `ChartRef` propagation, plus a regression test for
`MaxHistory=0`)
- `helm template installer packages/core/installer` renders cleanly with
default and overridden values; `--set
cozystackOperator.helmReleaseMaxHistory=0` correctly emits
`--helmrelease-max-history=0`

## Followups (deliberately not bundled)

- **Per-component timeout overrides** on `ComponentInstall` (mirroring
the `UpgradeCRDs` pattern from #2427). Charts like etcd that need a 30m
timeout would set it on their `Component` rather than tuning a global
flag. Touches the Package CR API + codegen — bigger lift, separate PR.
- **`Remediation.Retries` as a flag.** No current pain; skip.
- **`createOrUpdateHelmRelease` field preservation.** Currently
preserves `Suspend` but not `Test`/`Rollback`/`Uninstall`. Would
silently overwrite an SRE's manual edit. Minor cleanliness, separate PR.

## Test plan

- [ ] CI passes E2E with `helmReleaseInterval=30s` set in install bats
(existing T1 test path)
- [ ] No regression on production install path (default values)
- [ ] HR objects in a healthy cluster render with
`Strategy.Name=RetryOnFailure` and `RetryInterval=30s`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Operator and Helm chart now accept CLI flags and Helm values to
configure HelmRelease reconciliation intervals, retry behavior,
install/upgrade timeouts, and history retention.

* **Bug Fixes**
* Operator now validates HelmRelease timing/history inputs on startup
and will exit if values are invalid or non-positive.

* **Tests**
* Added unit and e2e coverage to verify HelmRelease configuration and a
chart install path using a custom reconciliation interval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-05 14:35:00 +05:00
Myasnikov Daniil 730037c85d fix(installer): address review feedback on cozy-system bootstrap
- Bump alpine/k8s to 1.33.4 (was 1.32.0, EOL); matches in-tree pin used
  by cozystack-api, seaweedfs, vm-disk, and vm-instance hooks.
- Skip hostNetwork=true and not-ready/CNI tolerations on the hosted
  variant: managed clusters provide CNI before install, and kube-system
  may carry strict PSA there that would reject hostNetwork pods.
- Add packages/core/installer/templates/cozy-system-namespace.yaml gated
  on .Values.bareNamespace (default false). The top-level Makefile's
  manifests target now passes --set bareNamespace=true and renders
  Namespace + operator into _out/assets/cozystack-operator-*.yaml so
  the kubectl-apply install path stays self-contained. Helm
  install/upgrade callers keep the default and bootstrap via
  --create-namespace + the pre-install labeler hook.
- Add a bats assertion in hack/e2e-install-cozystack.bats checking
  cozy-system carries pod-security.kubernetes.io/enforce=privileged and
  cozystack.io/system=true after helm install.
- Inline comment near hostNetwork=true documenting why dnsPolicy left
  at Default (cluster DNS not up at install time;
  ClusterFirstWithHostNet would not help).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-04 22:13:39 +05:00
Myasnikov Daniil e410b02e1d feat(operator): expose Install/Upgrade timeouts and MaxHistory as flags
The HelmRelease generation in package_reconciler hardcoded three values
that have caused recurring pain:

- Install.Timeout = 10m (line 214)
- Upgrade.Timeout = 10m (line 220)
- MaxHistory = unset (Helm default 5)

The 10m timeout in particular contradicts the per-Application timeout
work done in `pkg/config/config.go` and
`pkg/registry/apps/application/rest.go` (commit 7b146cbe), which lets
individual Applications override the install/upgrade timeout via
annotation. The PackageSource side had no equivalent — etcd works around
it by hand-rolling its HR YAML in `packages/apps/tenant/templates/etcd.yaml`
to set a 30m timeout, harbor has had multiple commits chasing the same
problem from the other side. This closes that gap.

New operator flags (each maps to a chart value with empty default;
operator uses its own default when value is empty):

- `--helmrelease-install-timeout` (default 10m) → Spec.Install.Timeout
- `--helmrelease-upgrade-timeout` (default 10m) → Spec.Upgrade.Timeout
- `--helmrelease-max-history` (default 5)        → Spec.MaxHistory

Production behaviour unchanged. E2E and edge cases (cert rotation,
slow-installing charts) can override per-cluster without modifying
chart manifests.

Per-component overrides on `ComponentInstall` (mirroring the existing
`UpgradeCRDs` pattern) would be a strictly better fix for charts like
etcd that need an outlier timeout — left as a deliberate followup since
it touches the Package CR API.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 12:44:06 +05:00
Myasnikov Daniil c4c70e0e18 feat(operator): switch HelmRelease retries to Strategy=RetryOnFailure with configurable RetryInterval
The operator generated HelmReleases with `Install.Remediation{Retries: -1}`
and `Upgrade.Remediation{Retries: -1}`, which is the v2 way of saying
"retry forever, never remediate". Combined with the spec-level
`Interval: 5m`, this meant a failed install/upgrade waited a full 5
minutes before the next attempt — even when the underlying race that
caused the failure (e.g. a chart artifact not quite Ready, a HR
dependency reconciling out of order) cleared in seconds.

Audit of the last 30 successful PR runs found that every single run had
its `Install Cozystack` step's `kubectl wait hr/seaweedfs-system` time
out at the 2-minute mark and recover only after an inline
`flux reconcile --force` workaround. Same root cause: 5-minute retry
interval was longer than the test's wait window. That workaround can be
removed once this change lands.

Switch to `Strategy.Name=RetryOnFailure` with `RetryInterval` exposed via
a new `--helmrelease-retry-interval` operator flag (default 30s):

- Functionally equivalent to the previous configuration ("retry forever
  on failure") since `Retries: -1` meant remediation never fired anyway.
- Decouples retry-on-failure timing from `spec.Interval` so failed
  releases recover at 30s without polling healthy releases at the same
  cadence (which would multiply controller load by ~10x for no benefit).
- Helm chart exposes `cozystackOperator.helmReleaseRetryInterval`,
  defaulting to empty (operator uses its own 30s default).
- Same flag pattern as the existing `--helmrelease-interval` (5m default),
  also added in this PR via cherry-pick.

Verified: go build green, go test ./internal/operator/... pass, helm
template renders both flags conditionally on values.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 12:40:13 +05:00
Myasnikov Daniil e197aba625 feat(operator): make HelmRelease Interval configurable, override to 30s in E2E
Adds --helmrelease-interval (default 5m, matching today's behaviour) to the
operator and a corresponding cozystackOperator.helmReleaseInterval value
on the cozy-installer chart (default empty -> no flag rendered ->
operator default 5m applies).

E2E install sets it to 30s. Rationale: with the 5m default, dependency-
blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only
every 5 min, producing an 8-10 min dead zone in the E2E install where
the fast pack of HRs is Ready but the slow tail hasn't been retried yet.
30s collapses the gap.

Production defaults are unchanged. The override is opt-in per install.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 12:36:28 +05:00
cozystack-ci[bot] b52e2801b4 Prepare release v1.3.0
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-04-22 07:28:09 +00:00
cozystack-ci[bot] 12bf6b0e26 Prepare release v1.3.0-rc.1
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
2026-04-16 08:50:30 +00:00
Andrei Kvapil 94223327fe Release v1.2.1 (#2306)
This PR prepares the release `v1.2.1`.
2026-04-02 18:56:08 +02:00
cozystack-bot c652e1e0f9 Prepare release v1.2.0
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-03-27 14:00:46 +00:00
cozystack-bot 3c971d0e1b Prepare release v1.1.0
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-03-06 12:03:56 +00:00
cozystack-bot 66a61bd63e Prepare release v1.0.0
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-02-26 15:17:04 +00:00
cozystack-bot 65c6936e95 Prepare release v1.0.0-rc.2
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-02-25 16:09:50 +00:00
cozystack-bot 7994976052 Prepare release v1.0.0-rc.1
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-02-21 20:58:57 +00:00
cozystack-bot bff5468b52 Prepare release v1.0.0-beta.6
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-02-17 22:36:40 +00:00
cozystack-bot bae70596fc Prepare release v1.0.0-beta.5
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-02-16 16:01:30 +00:00
Aleksei SviridkinandClaude 7ca6e5ce9e feat(installer): add variant-aware templates for generic Kubernetes support
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-16 18:47:04 +03:00
cozystack-bot b6a840e873 Prepare release v1.0.0-beta.4
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-02-14 09:17:24 +00:00
Andrei KvapilandClaude ba7c729066 [installer] Merge operator templates into single variant-based template
Consolidate three separate operator deployment templates (talos, generic,
hosted) into a single template with variant selection via values.yaml
parameter. This resolves conflicts when running `make apply` from the
installer package, which previously rendered all three deployments.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-11 22:02:58 +01:00
cozystack-bot da359d558a Prepare release v1.0.0-beta.3
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-02-11 06:13:35 +00:00
cozystack-bot 8f3d686492 Prepare release v1.0.0-beta.2
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-02-02 12:34:58 +00:00
cozystack-bot 4ffe453351 Prepare release v1.0.0-alpha.2
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-01-20 07:50:59 +00:00
Andrei KvapilandClaude 987a74ae5a refactor(telemetry): split telemetry between operator and controller
Split telemetry collection between cozystack-operator and cozystack-controller:

cozystack-operator now collects cluster-level metrics:
- cozy_cluster_info (cozystack_version, kubernetes_version)
- cozy_nodes_count (os, kernel)
- cozy_cluster_capacity (cpu, memory, nvidia.com/* resources)
- cozy_loadbalancers_count
- cozy_pvs_count (driver, size)
- cozy_package_info (name, variant)

cozystack-controller now collects application-level metrics:
- cozy_application_count (kind) - counts HelmReleases per ApplicationDefinition

Other changes:
- Add pkg/version for build-time version injection via ldflags
- Remove --cozystack-version flag (version now embedded at build time)
- Remove bundle/oidc configuration from telemetry (replaced by package_info)
- Remove cozy_workloads_count metric (replaced by cozy_application_count)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-19 13:52:15 +01:00
cozystack-bot 09290b5586 Prepare release v1.0.0-alpha.1
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2026-01-15 23:32:41 +00:00
Andrei KvapilandClaude 99ac2fc710 refactor(installer): migrate installer to cozystack-operator
Remove legacy installer components (cozystack-assets-server,
installer.sh script, cozystack container image) in favor of
cozystack-operator based deployment.

Move migration scripts from scripts/migrations/ to
packages/core/platform/images/migrations/ for containerized execution.

Add grafana-dashboards image for centralized dashboard management.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-15 14:14:48 +01:00
Andrei Kvapil cf8ac03f45 Release v0.40.2 (#1858)
This PR prepares the release `v0.40.2`.
2026-01-13 17:25:18 +01:00
Andrei Kvapil dececc8587 Release v0.40.1 (#1855)
This PR prepares the release `v0.40.1`.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-13 11:13:10 +01:00
Andrei Kvapil c43db3b7cc [cozystack-operator] Add deployment files
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-12-29 14:04:10 +01:00
cozystack-bot d53506ae2a Prepare release v0.38.2
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2025-12-01 19:14:15 +00:00
Andrei Kvapil 27efd3ad5e Release v0.38.1 (#1668)
This PR prepares the release `v0.38.1`.
2025-11-27 15:53:53 +01:00
cozystack-bot 975011e04e Prepare release v0.38.0
Signed-off-by: cozystack-bot <217169706+cozystack-bot@users.noreply.github.com>
2025-11-25 15:59:42 +00:00