## What this PR does
Implements the tenant Flux sharding design from #2817: a new
`flux-shard-operator` system package that spreads tenant HelmReleases
across multiple helm-controller shards, so a noisy tenant (e.g. a
HelmRelease stuck in infinite remediation) cannot degrade the others.
- **Placement controller** owns the tenant→shard assignment: the unit of
placement is the tenant (all of its HelmReleases share one shard label),
distribution is greedy least-loaded weighted by HelmRelease count, with
pinned tenants, threshold-gated rebalance, paced moves, and never moving
deleting tenants. Watches are metadata-only, so the controller does not
decode the helm-controller status-patch firehose. Assignments are
recorded on tenant namespaces; HelmRelease labels remain the source of
truth on restarts.
- **Shard runtime** is provisioned by the operator: `shardCount`
helm-controller Deployments cloned from the flux-aio `flux` Deployment
and sanitised, so the image and feature-gates stay version-synced
automatically. flux-aio itself is unchanged and keeps owning everything
without a shard key.
- **CREATE-only mutating webhook** (`failurePolicy: Ignore`) stamps the
tenant's shard onto every HelmRelease at admission, so each one is born
on the correct shard regardless of creation path; any miss degrades
gracefully to the placement controller.
- The hand-rolled `flux-tenants` Deployment is retired: the operator
drains it (backfill `tenants` → `shard<i>`) and deletes it once empty;
migration 44 removes a leftover only under the same guard.
Ships with `shardCount: auto` (the default): the operator sizes the
shard count from the tenant HelmRelease count — `K = clamp(ceil(H/100),
1, min(16, T))` with hysteresis (scale up past 120 HR/shard, down under
60) — so small clusters stay at one shard (≈ today's behaviour) and
large fleets shard out automatically. An integer value pins the count
explicitly.
One deviation from the design doc: the webhook/controller scope is the
`sharding.fluxcd.io/key` label rather than
`internal.cozystack.io/tenantmodule` — every tenant HelmRelease is born
with `key=tenants` (stamped by ApplicationDefinition `release.labels`
and chart templates), so the webhook rewrites it and the placement
controller continuously relabels stragglers; `tenantmodule` covers only
the 6 tenant chart modules.
### Screenshots
Not a UI change.
### Release note
```release-note
feat(fluxcd): tenant HelmReleases are now reconciled by operator-managed helm-controller shards; the flux-shard-operator balances tenants across `shardCount` instances and retires the legacy flux-tenants deployment
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced **flux-shard-operator** to shard helm-controller workloads
with deterministic tenant placement, optional rebalance controls, and
autosized shard count.
* Added a **CREATE-only mutating webhook** to stamp the correct shard
label on tenant HelmReleases.
* **Bug Fixes**
* Application updates now preserve existing **flux-shard** label
assignments to prevent unwanted shard re-stamping.
* **Chores**
* Added platform packaging for the new component, including an image
build step.
* Removed the legacy tenants deployment and added a migration to retire
it safely.
* **Tests**
* Expanded unit and chart tests for placement, provisioning, webhook
behavior, and configuration parsing.
* **Documentation**
* Added operator documentation describing configuration and metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Resolve go.mod conflict: keep main's go 1.26.4 and golang.org/x/* bumps, and keep gomodules.xyz/jsonpatch/v2 v2.4.0 as a direct dependency (imported by the flux-shard-operator admission webhook) instead of main's indirect entry. go mod tidy leaves go.sum consistent.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
release.cozystack.io/helm-install-timeout sets both Install.Timeout and
Upgrade.Timeout to one value. Add release.cozystack.io/helm-upgrade-timeout
so an ApplicationDefinition can override only Upgrade.Timeout — letting a
kind keep a short install budget but a longer upgrade budget (or vice
versa). When set it wins over the value the install annotation would
otherwise apply to the upgrade side.
- config: add HelmUpgradeTimeoutAnnotation + ReleaseConfig.HelmUpgradeTimeout;
generalize ParseHelmInstallTimeoutAnnotation to ParseHelmTimeoutAnnotation
since it now parses both annotations (identical Flux unit validation).
- server: parse the new annotation at startup with the same loud-fail
behavior; update --helmrelease-upgrade-timeout help.
- rest: apply HelmUpgradeTimeout after the install-annotation copy so the
upgrade-specific value wins; restore the accurate carrier comment.
- tests: cover the upgrade-only override, the asymmetric precedence, and
the install-sets-both regression guard.
Behavior is unchanged for every existing ApplicationDefinition (none set
the new annotation today).
Address review feedback from coderabbitai on rest.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The ParsePositiveDuration table exercised all error inputs but did not
distinguish which branch fired. Add an errMatch field so the table
pins "must be > 0" for zero/negative and "invalid duration" for
malformed/empty inputs, mirroring TestParseHelmInstallTimeoutAnnotation
and keeping the two parsers' diagnostics from drifting.
Address review feedback from coderabbitai on config_test.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
HelmReleaseMaxHistory carried omitempty, so a value of 0 (which Helm
treats as "unlimited") would be dropped on marshal and become
indistinguishable from an unset field. The runtime path (start.go
always populates it from --helmrelease-max-history) is safe today, but
drop omitempty so the explicit "unlimited" intent survives if
ReleaseConfig is ever serialized.
Address review feedback from coderabbitai on config.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
gofmt -w on the two touched test files: the case-table struct fields in
rest_helmrelease_spec_test.go and main_test.go were misaligned, and
main_test.go carried a trailing blank line. CI runs no gofmt gate
(pre-commit only runs make generate), so this would otherwise land
unformatted.
Address review feedback from lexfrei.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The convertApplicationToHelmRelease comment listed only kubernetes-rd
as carrying release.cozystack.io/helm-install-timeout, dropping
tenant-rd. tenant-rd/cozyrds/tenant.yaml sets the 15m annotation for
the seaweedfs-db CNPG bootstrap whose first reconcile exceeds the flux
default. Restore the accurate two-carrier list and rationale.
Address review feedback from lexfrei on rest.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The XValidation justification was factually wrong: the deployed
helm-controller CRD has no rule rejecting a Strategy alongside a
Remediation entry (only .retryInterval is gated on Strategy.Name), so
the claimed "silently break every HR" safety net does not exist.
Reword the comment in all three test files to the real reason
Remediation stays nil: retries are driven by a single
Strategy.RetryInterval path, matching cozystack-operator's
PackageReconciler.buildHelmReleaseSpec.
Address review feedback from lexfrei on rest_helmrelease_spec_test.go,
rest_timeout_test.go and internal/operator/package_reconciler_test.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Production change:
- convertApplicationToHelmRelease now sets Install/Upgrade.Timeout
unconditionally (matching cozystack-operator's buildHelmReleaseSpec).
The previous "stay nil if zero" branch was a leaky test shim — it kept
fixtures with empty ReleaseConfig{} green at the cost of api/operator
drift.
Test changes:
- newRESTForTimeout seeds the new HelmRelease* globals with
production-shaped defaults so the unset-annotation case exercises
"global default applies" rather than "field stays nil".
- TestConvertApplicationToHelmRelease_PerAppTimeoutOverridesGlobal is
table-driven and now covers asymmetric globals (install != upgrade)
with and without per-app override — guards against a regression that
only assigns one side under override.
- New start_test.go: table-driven validator coverage for each
malformed/non-positive duration flag, the negative MaxHistory branch,
the MaxHistory=0 unlimited branch, asymmetric timeout acceptance, and
a smoke test that NewCozyServerOptions defaults validate.
Refactor:
- Extracted parseAndValidateHelmReleaseFlags from Complete() so the
flag-validation portion is unit-testable without kubernetes I/O.
Documentation:
- Reworded --helmrelease-install/upgrade-timeout help so admins reading
--help see the dual-purpose annotation explicitly (one annotation
overrides both timeouts together).
- Dropped PR-#2509 references and "preserving the pre-#2509 behaviour
for tests" comments — kept the technical rationale (Strategy.Name
vs Remediation.Retries:-1 + 5m Interval coupling) on its own merits.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
convertApplicationToHelmRelease now builds the HelmRelease spec from
ReleaseConfig instead of hardcoding values, bringing it in lockstep
with cozystack-operator's PackageReconciler.buildHelmReleaseSpec
(PR #2509). The drift between the two HelmRelease-generating paths
was the rationale for this follow-up.
Changes to the generated spec:
- Spec.Interval reads ReleaseConfig.HelmReleaseInterval (was
hardcoded 5m).
- Spec.MaxHistory set as &maxHistory pointer from
ReleaseConfig.HelmReleaseMaxHistory; 0 survives as pointer-to-0
(unlimited per Helm semantics) - mirrors the operator's
TestBuildHelmReleaseSpecZeroMaxHistory.
- Install.Strategy{Name: RetryOnFailure, RetryInterval} replaces
Install.Remediation{Retries: -1}. Same for Upgrade.Strategy.
Functionally equivalent to "retry forever, never remediate" for
the api-side path (which also never relied on remediation
firing) but decouples failed-install retry timing from
Spec.Interval.
- Install.Remediation / Upgrade.Remediation stay nil:
helm-controller's XValidation rejects RetryOnFailure +
RetryInterval alongside a Remediation entry, so reintroducing
Retries:-1 "for safety" would silently break every HR.
Install/Upgrade.Timeout precedence:
- per-Application HelmInstallTimeout (annotation-driven) wins
when non-zero (preserves the kubernetes-rd opt-in path);
- otherwise HelmReleaseInstallTimeout / HelmReleaseUpgradeTimeout
globals apply;
- when both are zero, the field stays nil and flux defaults
apply (preserves the legacy code path for tests that
hand-construct ReleaseConfig without populating the new globals).
Tests:
- rest_timeout_test.go: Remediation.Retries == -1 assertions
replaced with Strategy.Name == RetryOnFailure +
Remediation == nil, matching the operator's spec test.
- rest_helmrelease_spec_test.go: new file mirroring
TestBuildHelmReleaseSpec / TestBuildHelmReleaseSpecZeroMaxHistory
from internal/operator/package_reconciler_test.go, plus a third
case asserting per-Application HelmInstallTimeout overrides the
global Install/Upgrade timeout defaults.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Wire five flags into cozystack-api that mirror the cozystack-operator
flags introduced in PR #2509:
--helmrelease-interval (default 5m)
--helmrelease-retry-interval (default 30s)
--helmrelease-install-timeout (default 10m)
--helmrelease-upgrade-timeout (default 10m)
--helmrelease-max-history (default 5)
CozyServerOptions.Complete() validates each duration via
config.ParsePositiveDuration so a misconfigured operator restarts
loudly instead of waiting until the first Application is created,
and rejects MaxHistory < 0 (0 is valid: unlimited per Helm
semantics, matching the operator).
Validated values are written into every ReleaseConfig built from
the ApplicationDefinitionList. The per-Application
HelmInstallTimeout (annotation-driven) still wins over the new
global Install/Upgrade timeout defaults; this commit only plumbs
the values - the consumer in convertApplicationToHelmRelease lands
in the next commit.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Lift parsePositiveDuration from cmd/cozystack-operator/main.go into
pkg/config so cozystack-api can reuse the same startup validator
without duplication. Operator main.go now imports and delegates;
the table-driven test moves to pkg/config/config_test.go.
Extend ReleaseConfig with the five HelmRelease generation knobs
the cozystack-operator already exposes via flags (PR #2509):
HelmReleaseInterval, HelmReleaseRetryInterval,
HelmReleaseInstallTimeout, HelmReleaseUpgradeTimeout,
HelmReleaseMaxHistory. The existing per-Application
HelmInstallTimeout (set from the
release.cozystack.io/helm-install-timeout annotation) keeps its
override semantics - it still wins over the new global
Install/Upgrade timeout defaults when set.
No behaviour change in this commit; the new fields are wired up in
follow-up commits.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
REST.Update wraps the HelmRelease write in retry.RetryOnConflict so a stale
resourceVersion (the helm-controller racing the status subresource) is
refreshed and retried instead of surfacing a 409. Add a unit test using a fake
client whose interceptor returns one Conflict on the first Update and succeeds
on the second, asserting Update returns nil and writes exactly twice.
Addresses non-blocking follow-up 3 on #2948.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The aggregated API server translates app CR updates (Etcd, Postgres, ...)
into a read-convert-update of the backing HelmRelease. Flux's
helm-controller continuously writes the HelmRelease status, which shares
the object's resourceVersion, so a CR update that lands while a prior
reconcile is still in flight reads a resourceVersion that goes stale
before the Update and the request fails with a 409 Conflict
("the object has been modified; please apply your changes to the latest
version and try again").
The HelmRelease spec is fully derived from the Application the caller
just applied, so a stale-resourceVersion conflict is never a real spec
conflict here. Wrap the Update in retry.RetryOnConflict, refreshing the
live resourceVersion on each conflict before retrying. The no-conflict
path is unchanged, and this mirrors the existing RetryOnConflict usage in
workloadmonitor_controller.go.
Surfaced by the etcd.bats e2e test, which applies the same singleton
Etcd resource three times back-to-back and intermittently failed the
third apply with the 409.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 76410599de)
## What this PR does
`cozystack-api` published the free-form `.spec` of aggregated
`apps.cozystack.io` resources (Tenant, Application, …) as the boolean
`additionalProperties: true`. The Kubernetes ValidatingAdmissionPolicy
**status type-checker** run by `kube-controller-manager`
nil-dereferences when it recurses into that node
(`k8s.io/apiserver/pkg/cel/openapi.isExtension`). As soon as a VAP
matches one of these resources — `cozystack-tenant-host-policy` matches
`tenants` — KCM `CrashLoopBackOff`s on every control-plane node, stalls
reconciliation, and times out the install (`kubectl wait hr --all` never
completes). With the 3× install retry removed, a single occurrence fails
E2E outright.
This emits `x-kubernetes-preserve-unknown-fields: true` instead, in
`pkg/cmd/server/openapi.go::patchSpec` (both the schemaless and
custom-schema branches, covering the OpenAPI v2 and v3 post-processors).
It is semantically equivalent — "arbitrary fields allowed" — and the
type-checker handles it safely. All managed-app custom schemas already
use the safe **object-form** `additionalProperties`, so `patchSpec` was
the only source of the boolean form.
### Regression test
`TestPatchedSpecDoesNotPanicVAPTypeChecker` reproduces the exact crash
through `k8s.io/apiserver/pkg/cel/openapi.SchemaDeclType` — the entry
point KCM's `validatingadmissionpolicystatus` controller uses. It
**panics on the pre-fix code and passes after the fix**. Companion
structural tests assert the published spec carries
`x-kubernetes-preserve-unknown-fields: true` and never the boolean
`additionalProperties: true`, and that genuine object-form maps are left
untouched.
Verified against the pristine vendored `k8s.io/apiserver@v0.34.1` (the
version cozystack-api builds against), which reproduces the panic, and
previously confirmed end-to-end on `kindest/node:v1.33.1`.
### Upstream
The underlying nil-dereference is a Kubernetes KCM bug — the VAP status
type-checker should treat `additionalProperties: true` as dynamic rather
than panic. A separate upstream report is being prepared; cozystack
fixes the symptom immediately with the schema change above.
Fixes#2863
### Release note
```release-note
fix(api): publish the free-form resource `.spec` as `x-kubernetes-preserve-unknown-fields` instead of `additionalProperties: true`, fixing a cluster-wide `kube-controller-manager` CrashLoopBackOff (nil-pointer panic in the ValidatingAdmissionPolicy status type-checker) that stalled installs.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Ensured published OpenAPI schemas no longer contain boolean-form
additionalProperties anywhere; top-level and nested schemas now use
preserve-unknown-fields for free-form specs, preventing
validation/runtime issues.
* **Tests**
* Added regression tests that feed varied untrusted schemas and verify
boolean additionalProperties are removed and published schemas remain
valid for tooling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from gemini-code-assist on pkg/cmd/server/openapi.go:
markOpenObject and sanitizeForV2 initialized s.Extensions with the generic
map[string]any{}; use the named spec.Extensions{} type instead, matching the
kube-openapi types and the generated definitions in zz_generated.openapi.go.
No behavioral change — spec.Extensions is defined as map[string]interface{}.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from lexfrei and coderabbitai on
pkg/cmd/server/openapi_test.go:
TestPatchSpecSanitizesUserSuppliedBooleanAdditionalProperties only exercised the
top-level, properties, items, and allOf positions, leaving the one remaining
type-checker-reachable sanitizer branch untested: a boolean additionalProperties
nested inside an object-form additionalProperties. Deleting that recursion line
kept the whole suite green while the input panicked SchemaDeclType again (#2863).
Add the nested-under-additionalproperties case (the reachable blocker, verified
red->green by mutation) plus anyOf/oneOf/not cases (defense-in-depth: those
branches are recursed by the sanitizer but never traversed by SchemaDeclType, so
the hasBooleanAdditionalProperties assertion guards them). Rewrite the test
docstring to distinguish the type-checker-reachable cases from the
defense-in-depth ones, replacing the inaccurate "each case panics SchemaDeclType"
note.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Update handler rebuilds the HelmRelease from the Application, which
re-stamped sharding.fluxcd.io/key with the ApplicationDefinition default
and bounced the HelmRelease off its operator-assigned shard on every
update. Carry the live label over from the current HelmRelease instead.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Addresses review feedback on #2863. The previous fix only converted the open
spec wrapper cozystack-api injects, so a user-supplied ApplicationDefinition
openAPISchema carrying a boolean additionalProperties — declared explicitly at
the top level, or nested under properties/items/allOf/anyOf/oneOf — was
republished verbatim and re-triggered the kube-controller-manager VAP
type-checker crash. openAPISchema is untrusted input (external-apps and
operator-authored definitions flow through unvalidated), so close the whole
class rather than only the schemas cozystack ships.
Recursively rewrite every boolean-form additionalProperties (nil inner schema):
additionalProperties:true becomes x-kubernetes-preserve-unknown-fields:true,
additionalProperties:false is dropped. Both JSON forms carry a nil inner schema
and crash the type-checker identically; object-form maps are recursed into and
left intact. Also publish the schemaless spec from a fresh schema so no
inherited $ref or properties leak into the open object.
Regression tests cover top-level true/false and nested (properties, items,
allOf) boolean forms, asserting the published schema carries no boolean
additionalProperties node and does not panic SchemaDeclType.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
cozystack-api published the free-form ".spec" of aggregated apps.cozystack.io
resources (Tenant, Application, ...) as the boolean `additionalProperties: true`.
The Kubernetes ValidatingAdmissionPolicy status type-checker run by
kube-controller-manager nil-dereferences when it recurses into that node, so as
soon as a VAP matches one of these resources (cozystack-tenant-host-policy on
tenants) KCM CrashLoopBackOffs cluster-wide and stalls reconciliation, failing
cold installs once the 3x install retry was removed.
Emit `x-kubernetes-preserve-unknown-fields: true` instead. It is semantically
equivalent ("arbitrary fields allowed") and the type-checker handles it safely.
All app custom schemas already use the safe object-form additionalProperties,
so patchSpec was the only source of the boolean form.
Adds a regression test that reproduces the exact crash through
k8s.io/apiserver/pkg/cel/openapi.SchemaDeclType (the entry point KCM uses): it
panics on the pre-fix code and passes after the fix.
Fixes#2863
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Watch goroutine never closed the events channel it owns, and
ProxyWatcher.Stop does not close the channel it proxies. When the
backing watcher ended, consumers ranging over ResultChan blocked until
their own timeout instead of seeing end-of-stream. Closing the channel
on goroutine exit matches the watch.Interface contract and the pattern
already used by the tenantmodule and application registries.
Goroutine-termination tests now assert the result channel closes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Four goroutine-termination tests duplicated the same IsStopped polling
loop; waitForFakeWatcherStop keeps the deadline logic in one place.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Manual runtime.NewScheme setups in the List/Get error-path tests silently
discarded AddToScheme errors; the shared helper fails the test instead.
Renamed from newWatchTestScheme since it now serves both test files.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Cover the per-event filtering paths of REST.Watch directly instead of
only transitively through the bookmark test: authorization filtering via
RoleBindings, privileged groups (system:masters, cozystack-cluster-admin),
non-tenant namespace and foreign object drops, backing bookmark
forwarding, field and label selector filtering, missing user rejection,
access-check error resilience, ADDED deduplication by resourceVersion,
terminating bookmark flush on backing watcher close, and goroutine
shutdown on Stop and context cancellation.
Also close the remaining gaps on the sibling read paths sharing the same
authorization logic: List/filterAccessible (per-user filtering, privileged
groups, missing user, RoleBindings outside the tenant set, backing list
errors) and the Get error branches.
Branches the controller-runtime fake client cannot drive (backing
bookmarks, non-Namespace objects, chosen resourceVersions, backing watch
errors) are exercised through a stub client.WithWatch backed by a
test-owned FakeWatcher.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
While preparing for my CozySummit talk of last week (creating commits
from CozySummit edits with
[gitops-reverser](https://github.com/ConfigButler/gitops-reverser)) I
noticed warnings on my console:
```
Warning: event bookmark expired err="reflector.go:343: hasn't received required bookmark event marking the end of initial events stream, received last event 59.998946102s ago"
The warning fires on every reflector restart and continuously while the cache is unsynced from the WatchList perspective.
```
I've been searching a bit better and it turns out that `TenantSecret`,
`TenantModule`, and `TenantNamespace` (group `core.cozystack.io`) all
ignore `ListOptions.SendInitialEvents`. Interestingly enough
`apps.cozystack.io/Application` already does implement this in the right
way.
Under the [WatchList / streaming-list
protocol](https://kubernetes.io/docs/reference/using-api/api-concepts/#streaming-lists)
(client-go's `WatchListClient`, default-on since v1.35), clients expect
the initial `ADDED` events to be terminated by a
[`Bookmark`](https://kubernetes.io/docs/reference/using-api/api-concepts/#watch-bookmarks)
annotated with `k8s.io/initial-events-end: "true"`. These three
resources only forwarded the backing watcher's bookmarks (which lack the
annotation), so the marker was never sent and client-go reflectors never
reached `HasSynced` — looping on `reflector.go:343: hasn't received
required bookmark event marking the end of initial events stream`.
**Fix:** extracted the contract into a shared
`registry.InitialEventsBookmarker` helper (`pkg/registry/utils.go`) and
wired it into all four `Watch` implementations. It annotates the first
backing bookmark as the terminating marker, with fallbacks to emit it
before the first live event and on watcher close. The annotation reuses
the upstream `metav1.InitialEventsAnnotationKey` constant rather than a
string literal.
The full bookmark machinery in `k8s.io/apiserver/pkg/storage/cacher`
isn't reusable here: it's internal to the watch-cache layer and only
fires for cacher-backed storage, whereas these resources translate a
controller-runtime watch in custom REST storage that bypasses the
cacher.
`Application` is refactored onto the helper with its `WorkloadMonitor`
event-draining preserved. Adds unit tests for the helper and watch tests
for each core resource (`ADDED` → annotated `Bookmark` → live event).
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(api): core.cozystack.io resources (TenantSecret, TenantModule, TenantNamespace) now emit the k8s.io/initial-events-end bookmark for SendInitialEvents=true (WatchList) watches, so client-go informers reach HasSynced instead of looping on the missing-bookmark warning.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Watch endpoints reliably emit the "initial-events-end" bookmark after
initial events, improving watch/list contract compliance and client
reflector synchronization.
* **Refactor**
* Watch implementations consolidated around a centralized initial-events
bookmarker and stop-aware event forwarding for consistent bookmark
behavior.
* **New Features**
* Introduced a reusable initial-events bookmarker to coordinate bookmark
emission and resourceVersion observation across translated watch
streams.
* **Tests**
* Added unit and integration tests validating bookmark emission,
ordering, resourceVersion seeding/observation, and close-time behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The aggregated apiserver resources in group core.cozystack.io
(TenantSecret, TenantModule, TenantNamespace) ignored the
SendInitialEvents field of ListOptions. Clients using the WatchList /
streaming-list protocol (client-go's WatchListClient feature, on by
default since v1.35) never received the required
k8s.io/initial-events-end bookmark, so their reflectors stayed stuck in
the initial-stream phase and logged "hasn't received required bookmark
event marking the end of initial events stream" every ~10s, never
reaching HasSynced.
apps.cozystack.io/Application already implemented this contract inline.
Extract that logic into a shared registry.InitialEventsBookmarker helper
(pkg/registry/utils.go) and wire it into all four translating Watch
implementations: after the initial ADDED events, the first backing
bookmark is annotated as the initial-events-end marker, with fallbacks
that emit it before the first live event and on watcher close. Application
is refactored onto the shared helper with its WorkloadMonitor
pending-event draining preserved.
Adds unit tests for the bookmarker and watch tests for each of the three
core resources asserting ADDED -> annotated Bookmark -> live event.
```release-note
[apiserver] core.cozystack.io resources (TenantSecret, TenantModule,
TenantNamespace) now emit the k8s.io/initial-events-end bookmark for
SendInitialEvents=true (WatchList) watches, so client-go informers reach
HasSynced instead of looping on the missing-bookmark warning.
```
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Simon Koudijs <simonkoudijs@gmail.com>
instancetype/instanceprofile replaced the static vm-instance enum with live
dropdowns sourced from instancetype.kubevirt.io. A wrong group/version would
make those dropdowns silently empty in every cluster (List tolerates the
error) — the failure mode the appkind GVR guard already catches. Pin both
GVRs to the served API.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The one-shot option watch emitted the current state once then blocked on
ctx.Done(). If the initial List failed (e.g. every provider forbidden by a
broken RBAC config) it blocked silently, leaving the client with an empty
stream indistinguishable from "no options". Log the error so an operator can
tell the two apart.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The gpu, image, storagepool, storageclass and vmdisk providers carry the real
parsing logic (whitelist-intersect-allocatable, prefix strip, nested pool
walk, default-class marking, size label) but had no tests — and List tolerates
a provider error by skipping it, so a logic bug ships as a silently empty or
wrong dropdown rather than a failure. Add table tests against the fake dynamic
client for each, raising option-package coverage from 39% to 77%.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The core fuzzer was migrated from gofuzz to randfill, but the apps fuzzer was
left on gofuzz. TestRoundTripTypes runs both fuzzers (apps first), so
registering a gofuzz.Continue func against the randfill-based roundtrip harness
panics ("customFuncs' second argument must be a randfill.Continue").
Migrate the apps fuzzer the same way (randfill.Continue / FillNoCustom). With
no gofuzz consumers left, go mod tidy drops the dependency. TestRoundTripTypes
now passes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Option REST deliberately makes List tolerate a single failing provider
(it logs and skips it, so a partial RBAC config drops one dropdown instead
of breaking the whole list) while Get surfaces the error as a 500. Neither
path was covered. Add tests that inject a failing provider and assert List
keeps the healthy sources and omits the failing one, and that Get returns an
InternalError.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The appkind option provider queried applicationdefinitions in the
apps.cozystack.io group, but the ApplicationDefinition CRD is served from
cozystack.io. The provider listed a resource that does not exist, so the
appkind source resolved empty on every cluster (tolerated and dropped by
List, 500 on Get), and the backup form fields that reference it never
populated.
Point the provider at cozystack.io, which the existing cozystack.io read
grant already covers. Add a guard test pinning the CRD group, since the
existing fake-client test is built from the same GVR and cannot catch a
wrong group.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add three dropdown sources to the Option provider registry so the
dashboard's backup forms can use the unified x-cozystack-options
mechanism instead of injecting client-side enum arrays:
- plan / backup: namespaced, list Plans / Backups in the tenant namespace
- appkind: cluster-scoped, lists application kinds from ApplicationDefinitions
Annotate the backups.cozystack.io CRD schemas (Plan, BackupJob, Backup,
RestoreJob) with x-cozystack-options on backupClassName (backupclass),
applicationRef.kind / targetApplicationRef.kind (appkind), planRef.name
(plan) and backupRef.name (backup). This also gives the previously
unused backupclass source its first consumer.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Introduce a read-only, namespaced virtual resource
core.cozystack.io/v1alpha1 Option that serves named lists of dropdown
options for the dashboard. Each object (one per source: gpu,
instancetype, instanceprofile, network, image, storagepool,
storageclass, vmdisk, backupclass) is computed on read by an in-process
provider using a privileged dynamic client, so tenants need no direct
access to Nodes, the KubeVirt CR, cluster instancetypes, etc.
Tenants get read access via the aggregated cozy:tenant roles; the
apiserver ServiceAccount gains read on the underlying sources.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The roundtrip test for the core API group panicked (Filler.Funcs:
customFuncs' second argument must be a randfill.Continue) because the
fuzzer still used github.com/google/gofuzz while apimachinery switched
to sigs.k8s.io/randfill. Port the fuzz functions to randfill.Continue
/ FillNoCustom.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
tenant-rd now carries release.cozystack.io/helm-install-timeout alongside
kubernetes-rd (its parent chart bootstraps the seaweedfs-db CNPG cluster, whose
first reconcile exceeds flux's default wait budget). Update the comment that
claimed only kubernetes-rd carries it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## Summary
Annotate `storageClass` as immutable across every stateful chart that
exposes one: clickhouse, foundationdb, harbor, http-cache, kafka (Kafka
+ ZooKeeper), kubernetes (top-level + nodeGroups[]), mariadb, mongodb,
nats, openbao, opensearch, postgres, qdrant, rabbitmq, redis, vm-disk.
The PVC-pinning argument is identical for every chart — Kubernetes fixes
`storageClassName` on each PVC at creation, so editing this field on an
existing resource never migrated data; the schema annotation makes that
contract explicit and lets schema-aware consumers refuse the edit
up-front.
Schema-aware consumers today: the Cozystack UI
(cozystack/cozystack-ui#6) reads the rule from the chart's openAPISchema
and renders the field read-only on edit forms.
This PR is **schema-side only**. The cozystack aggregated apiserver does
not yet evaluate CEL rules in `openAPISchema` (`MariaDB` / `Postgres` /
etc. are application-backed, not CRDs); tracking issue
cozystack/cozystack#2657 covers wiring CEL into
`pkg/registry/apps/application/rest.go::Update`. Until that lands, an
apiserver-level `kubectl patch` that changes `storageClass` is still
accepted — but the underlying StatefulSet PVCs do not move, so there's
no data-corruption risk, only a confusing UX. The UI catches that case
today; the apiserver gate closes it permanently.
## What changes
For each app in the list above:
- `packages/apps/<app>/values.yaml` — gain a `## @immutable` directive
directly above (or below, for fields inside a `@typedef`) the
`storageClass` line.
- Regenerated via `make generate` in each package:
- `api/apps/v1alpha1/<app>/types.go` — kubebuilder `XValidation` marker
on the relevant field(s).
- `packages/apps/<app>/values.schema.json` — `x-kubernetes-validations:
[{rule: "self == oldSelf", message: …}]` on `storageClass`.
- `packages/apps/<app>/README.md` — regenerated parameter table.
- `packages/system/<app>-rd/cozyrds/<app>.yaml` — the embedded
`openAPISchema` on the ApplicationDefinition reflects the change.
Refactor: `pkg/registry/apps/application/rest.go` factors the inline
schema-build path into a new `buildSpecSchema` helper so it can be
unit-tested without standing up a fake apiserver. Three regression tests
in `pkg/registry/apps/application/rest_spec_schema_test.go` pin the
contract that `x-kubernetes-validations` survives the
v1→internal→structural conversion (test loads the actual MariaDB
ApplicationDefinition fixture on disk, asserts the rule reaches the
structural schema). Empty-input and malformed-JSON paths are also
pinned.
No e2e admission test is added because the apiserver does not yet
enforce the rule — see cozystack/cozystack#2657.
No changelog entry: schema-side annotation is invisible at the apiserver
layer today (UI-only consumer) and the release-cut PR will add the entry
when admission enforcement lands.
## Dependencies
Requires cozystack/cozyvalues-gen#24 to be merged and released, and the
`cozyvalues-gen` binary pin in `.github/workflows/pre-commit.yml` line
50 bumped to the release that includes that PR. Pre-commit CI will fail
until both happen — the stock `v1.3.0` binary silently ignores `##
@immutable` and regenerates artefacts without the rule, producing a
drift against what's committed.
Once that ordering is satisfied, rebase this branch and the pre-commit
job will be green.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* storageClass is now treated as immutable after creation for many
stateful apps (enforced by chart/CRD schema validation).
* **Documentation**
* Added a storage immutability guide explaining the contract and
enforcement.
* Updated chart READMEs and values docs to mark storageClass as
immutable and clarified nodeGroup storage remains intentionally mutable.
* **Tests**
* Added tests validating schema parsing and preservation of immutability
rules.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2639?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Annotate storageClass as immutable across every stateful app whose
chart exposes one — clickhouse, foundationdb, harbor, http-cache,
kafka (Kafka + ZooKeeper), kubernetes (top-level only), mariadb,
mongodb, nats, openbao, opensearch, postgres, qdrant, rabbitmq,
redis, vm-disk. Without coverage across the catalogue the UI would
render storageClass read-only on some apps and editable on others —
a worse mental model than uniform behaviour.
The annotation lands at the schema layer (CEL XValidation rule
'self == oldSelf' on values.schema.json / types.go / embedded
openAPISchema in cozyrds/<app>.yaml). The cozystack aggregated
apiserver does not yet evaluate CEL on application-backed resources;
that is tracked at cozystack/cozystack#2657. The UI
(cozystack/cozystack-ui#6) consumes the marker today.
The PVC-pinning argument is identical for every chart: Kubernetes
fixes a PVC's storageClassName at creation time, so editing this
field on an existing resource never migrated data. The schema
annotation makes that contract explicit.
kubernetes.nodeGroups[name].storageClass is intentionally NOT
annotated: the field is optional and undefaulted, so a strict
'self == oldSelf' rule would block any future attempt to set it on
an existing node group. A richer ratcheting form is needed there —
see docs/storage-immutability.md.
A refactor in pkg/registry/apps/application/rest.go extracts the
inline schema-build path into buildSpecSchema so it can be
unit-tested. Three regression tests in rest_spec_schema_test.go pin
the contract that x-kubernetes-validations survives the
v1->internal->structural conversion (loaded from the actual MariaDB
ApplicationDefinition fixture on disk, not a hand-edited mock).
Each per-app README gains a one-line blockquote pointing to a new
docs/storage-immutability.md that explains the contract and which
consumers enforce it today.
Requires cozystack/cozyvalues-gen with @immutable support — bump the
pre-commit pin in .github/workflows/pre-commit.yml once the release
exists.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from coderabbitai on
pkg/apis/apps/presets/legacy_test.go:151.
Replace the index-by-index loop over `got` with a single t.Fatalf
that prints both slices via %#v. The previous loop iterated only
across `got`, so extra entries in `want` would slip past unreported
when `got` was shorter than `want`; t.Fatalf with full slice
representations names both sides regardless of length.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
pkg/registry/apps/application/rest.go:1777.
When app.Spec.Raw cannot be unmarshalled to map[string]any the
deprecation walker now logs a klog.V(2) line naming the kind, name,
namespace and the underlying error before returning. Upstream
validation still owns rejecting invalid specs (this hook is purely
informational), so the log stays at debug verbosity rather than
surfacing as a warning that would fire on transient inputs the
caller is about to reject anyway.
Helper signature changed to (kind, namespace, name string, raw []byte)
so the malformed-input path is reachable from existing unit tests
without forcing a fake Application wrapper.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
cozystack-api now scans each app CR Create and Update for the legacy
flat resourcesPreset names (nano, micro, small, medium, large, xlarge,
2xlarge) and emits a klog warning per occurrence with the dotted JSON
path inside the spec and the suggested 1:1 instance-type replacement.
Example log line:
Postgres/foo in tenant-x uses deprecated resourcesPreset "small" at
spec.resourcesPreset; migrate to "t1.small" (1:1 equivalent CPU and
memory)
Warnings are non-blocking. Legacy values still render correctly through
the cozy-lib legacy alias table; the warning helps operators discover
remaining legacy data after the platform upgrade.
The mapping table lives in pkg/apis/apps/presets/legacy.go and is the
single Go-side source of truth. The walker handles top-level, nested,
in-array, non-string and malformed-spec inputs. The wrapper warns are
emitted via a pure helper deprecationMessagesFor so the message format
is unit-testable without intercepting klog.
Co-authored-by: Matthieu ROBIN <matthieu@hidora.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Replace the ephemeral `emptyDisk` volume on tenant Kubernetes worker VMs
with persistent PVC-backed storage via KubeVirt `dataVolumeTemplates`.
Previously, kubelet certificates, kubeconfig, and containerd state were
lost on VM reboot, causing the node to fall out of the tenant cluster.
**Changes:**
- Rename `ephemeralStorage` field to `diskSize` in NodeGroup
configuration
- Replace `emptyDisk` volume with `dataVolumeTemplates` using CDI blank
source
- Add `cdi.kubevirt.io/storage.usePopulator: "false"` annotation to work
around CDI volume population issues with LINSTOR CSI
- Add optional per-nodeGroup `storageClass` field (defaults to cluster
default)
- Set fixed 2Gi ephemeral-storage resource for virt-launcher pod
- Rename internal mount paths from `/ephemeral` to `/persistent`
- Add helm unit tests (`cluster_test.yaml`) covering dataVolumeTemplates
defaults, custom disk size, storageClass presence/absence, fixed
ephemeral-storage, disk name consistency, and CDI populator annotation
- Quote storage values in templates (`diskSize`, `storageClassName`) for
Helm correctness
- Guard bind-mount commands with `mountpoint -q` checks and create
target directories to handle fresh images and cloud-init retries
- Set `blockSize.matchVolume` on worker node disks for automatic block
size detection, fixing DRBD 4Kn compatibility with QEMU
**Known gotchas:**
- CDI volume population (prime PVC rebind) causes `ClaimMisbound` errors
with LINSTOR CSI driver, regardless of StorageClass (`replicated` or
`local`). The `usePopulator: false` annotation switches to the legacy
import path which works correctly. This is a known CDI issue documented
in
[kubevirt/containerized-data-importer#3146](https://github.com/kubevirt/containerized-data-importer/issues/3146).
- The `replicated` StorageClass with DRBD `suspend-io` quorum policy can
cause transient VM IO errors during LINSTOR satellite restarts (quorum
loss). The `local` StorageClass is recommended for worker node disks.
When `storageClass` is not specified in a nodeGroup, the cluster default
StorageClass is used.
- Existing clusters will trigger a rolling replacement of all worker
nodes on upgrade due to changed KubevirtMachineTemplate hash.
- DRBD volumes use 4096-byte logical sectors (`block-size 4096`).
Without explicit block size configuration, QEMU defaults to 512-byte
sectors, causing immediate IO errors and VM pause. The
`blockSize.matchVolume` setting in the disk spec ensures KubeVirt
auto-detects the correct sector size from the volume.
**Tested on a live cluster:**
- Verified DataVolume provisioning with `local` StorageClass
- Verified nodes join the tenant cluster and reach Ready state
- Verified node survives VM reboot (pod deletion) and returns to Ready
with the same Node object, reusing kubelet certificates from persistent
disk
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes
Worker node VMs now use persistent PVC-backed disks instead of ephemeral emptyDisk volumes. Kubelet certificates, kubeconfig, and containerd state survive VM reboots, preventing nodes from falling out of the tenant cluster.
BREAKING CHANGE: The `ephemeralStorage` field in nodeGroups has been renamed to `diskSize`. A new optional `storageClass` field allows per-nodeGroup storage class selection. Users must update their values overrides accordingly.
Worker node disks now use `blockSize.matchVolume` to automatically detect the underlying volume's sector size, ensuring compatibility with DRBD-backed storage classes that use 4096-byte logical sectors.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Optional storageClass parameter to select storage class for worker
node persistent disks (defaults to cluster default).
* Makefile test target added to run Helm unit tests for the chart.
* **Configuration Changes**
* Worker node storage parameter renamed from ephemeralStorage to
diskSize (default: 20Gi).
* Kubelet/containerd state moved to persistent disk paths and node
ephemeral-storage accounting fixed to 2Gi; disk provisioning aligns
block sizing with backing volumes.
* **Documentation**
* Updated docs and chart/schema to reflect the persistent disk model and
new fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Gate defensive label post-filter on event type: DELETED events now always
pass through so cached clients can evict stale entries when a Secret's
labels mutate out of the watched selector
- Extract buildTenantSelector helper to eliminate duplicated selector-merge
code between List and Watch
- Add TestList_WithNothingSelector_ReturnsEmpty and
TestWatch_WithNothingSelector_ClosesImmediately to cover the previously
untested labels.Nothing() path
Signed-off-by: IvanHunters <ivan.okhotnikov@aenix.io>
Watch() in the TenantSecret REST handler hard-coded the selector to the
internal.cozystack.io/tenantresource=true label and discarded
opts.LabelSelector entirely. Watch streams therefore returned every
tenant Secret in the namespace regardless of the user-provided selector,
while List() filtered correctly. Clients that switch to Watch after the
initial List surfaced unrelated Secrets.
Merge opts.LabelSelector into the underlying watch selector the same
way List() already does, handle non-selectable selectors by returning a
closed watcher, and add a defensive post-filter in the streaming
goroutine so events never escape the requested selector even if a future
client implementation skips label filtering on Watch.
Adds rest_test.go covering label-selector merging in List and a Watch
test that creates Secrets after the watch starts (since fake.Client.Watch
does not emit initial ADDED events) to assert only matching events are
observed.
Signed-off-by: IvanHunters <ivan.okhotnikov@aenix.io>
Integrate the HAMi GPU virtualization addon from main while
keeping the persistent-storage diskSize and storageClass fields
introduced by this branch.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## Summary
Fixed two IDOR (Insecure Direct Object Reference) vulnerabilities in the
TenantNamespace API handlers that allowed authenticated users to access
metadata of tenant namespaces without proper authorization checks.
## Changes
### New optimized function: hasAccessToNamespace()
- Lists RoleBindings **only in the target namespace** instead of all
cluster RoleBindings
- Used by Get() and Watch() for single-namespace access checks
- Order of magnitude faster than the previous approach
### Get() handler
- Uses `hasAccessToNamespace()` for efficient authorization
- Returns `NotFound` instead of `Forbidden` to prevent tenant
enumeration
- Now correctly enforces RoleBinding-based access control
### Watch() handler
- Uses `hasAccessToNamespace()` for per-event authorization
- Logs authorization errors with klog for security audit
- Events for unauthorized namespaces are silently filtered out
- Ensures users only receive watch events for namespaces they have
access to
### Additional fixes
- Fixed ServiceAccount subject handling when namespace is empty
(defaults to RoleBinding namespace)
- Added proper error logging in Watch handler
## Performance Impact
| Handler | Before | After |
|---------|--------|-------|
| List() | List all RoleBindings × 1 | No change ✅ |
| Get() | List all RoleBindings × 1 | List RoleBindings in 1 namespace 🚀
|
| Watch() | List all RoleBindings × N events | List RoleBindings in 1
namespace × N events 🚀 |
**For Watch with 100 events:**
- Before: 100 × (all cluster RoleBindings) = catastrophic
- After: 100 × (1-5 RoleBindings in namespace) = fast + cached by
controller-runtime
## Security Impact
**Before**: Any authenticated user could:
- Read metadata (labels, annotations, creation time) of any `tenant-*`
namespace via `Get()`
- Stream all tenant namespace events via `Watch()`, including
creation/modification/deletion
**After**: Users can only access tenant namespaces they have explicit
RoleBindings for, matching the behavior of the `List()` handler.
## Testing
Manually verified:
- Users can only `get` their own tenant namespaces
- Users can only `watch` events for their own tenant namespaces
- Unauthorized access returns `NotFound` (not `Forbidden`) to prevent
enumeration
- `List()` behavior remains unchanged and consistent with `Get()` and
`Watch()`
- Authorization errors are logged for security audit
## Checklist
- [x] Code follows project style and conventions
- [x] Security vulnerability is fully mitigated
- [x] Authorization logic is consistent across List/Get/Watch handlers
- [x] Performance optimized per feedback from @timofei.larkin
- [x] Error logging added for security audit
- [ ] Unit tests added (can be done in follow-up PR)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enforced per-namespace access control for Get and Watch; inaccessible
namespaces return Forbidden or NotFound as appropriate
* Forwarded field/label selectors to upstream watches and added
defensive filtering to skip inaccessible events (logged)
* Improved ServiceAccount subject namespace fallback and
privileged-group bypass
* **Tests**
* Added security tests covering RoleBinding subjects, groups, privileged
bypasses, ServiceAccounts, and access-denied behaviors
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Hoist user identity extraction out of the Watch goroutine; reuse a
cached username and groups map across events instead of re-fetching
them per event. Watch now returns Unauthorized up front when no user
is present in the context, rather than failing silently per event.
- Switch the per-event access-check error log to structured klog.ErrorS
to comply with the project Go style guide.
- Strengthen TestGet_WithAccess to assert the concrete *TenantNamespace
type plus Name, Kind, and APIVersion, so type or metadata regressions
fail fast.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from sircthulhu and CodeRabbit:
1. Return Forbidden instead of NotFound for unauthorized access
- Get() now returns 403 Forbidden to follow standard K8s RBAC behavior
- Previously returned 404 NotFound for security-by-obscurity
- Updated test expectations to match new behavior
2. Propagate field and label selectors in Watch handler
- Pass opts.FieldSelector and opts.LabelSelector to upstream Watch
- Add defensive filtering before authorization to prevent RBAC bypass
- Fixes potential issue with resourceNames restrictions
3. Refactor subject-matching logic to eliminate duplication
- Extract matchesSubject() helper for Group/User/ServiceAccount checks
- Remove duplicated code from filterAccessible and hasAccessToNamespace
- Consolidates ServiceAccount namespace fallback logic
All tests pass successfully.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Incorporate images.waitForKubeconfig support from main while
keeping the persistent-storage diskSize and storageClass fields
introduced by this branch.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Andrei Kvapil
