Commit Graph
146 Commits
Author SHA1 Message Date
Myasnikov Daniil 5265ee662c fix(tenant): stop the gateway release waiting on TenantGateway listeners and certificates
helm-controller waits on every object of a release with the kstatus
poller on install and on every upgrade. kstatus reads a TenantGateway at
Ready=False/ListenersNotReady as InProgress, and a TenantGateway sits
there whenever one listener certificate is pending, which depends on
ACME or DNS and, under HTTP-01, on routes that other releases own. After
a platform upgrade the gateway release therefore times out, retries
without end under RetryOnFailure, holds a helm-controller worker
meanwhile, and fails the tenant release that waits on it. It used to
pass because health is judged only inside a Helm action: the release
was installed while the TenantGateway had no status yet or was still
Ready, and the listeners whose certificates do not issue came later
with other releases' routes.

An always-true expression replaces kstatus for TenantGateway and keeps
the wait on. Flux checks status.observedGeneration against the
generation before it evaluates any expression, so the release still
waits until the controller has observed the new spec, but not for
listener or certificate health. The poller strategy is set explicitly
because expressions are only evaluated under it.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-23 14:34:55 +00:00
Aleksei Sviridkin 6addef4d3c fix(kubevirt): route the export proxy to the Service KubeVirt creates (#4637)
## What this PR does

The VM export routes point at a Service that does not exist, and the
tenant network policy would block the Ingress even if it did.

KubeVirt creates the export proxy Service under the name
`virt-exportproxy` (`VirtExportProxyName` in
`pkg/virt-operator/resource/generate/components`, v1.9.0). The Ingress
and the TLSRoute that publish `vm-exportproxy.<host>` name a
`vm-exportproxy` Service, and nothing creates one. The backend was
`virt-exportproxy` when the Ingress was added in 0ddaff938. Then
3780925a6 renamed the hostname to `vm-exportproxy` and the backend along
with it. The hostname stays, only the backend goes back.

Renaming the backend is not enough for the Ingress, because it is served
by the ingress controller in tenant-root and tenant egress is
default-deny. Each allowed destination has its own `allow-to-*` policy.
The CDI upload proxy has `allow-to-cdi-upload-proxy`, the export proxy
had nothing. The new `allow-to-vm-exportproxy` policy renders for the
same tenants and the same source pods as the CDI one. Its destination is
narrower: only the `kubevirt.io=virt-exportproxy` pods in
`cozy-kubevirt`, on TCP 8443, the container port virt-operator gives
them.

The unit tests pin the backend name for the Ingress and the TLSRoute,
and the placement, selector and port of the new policy.

I checked this on a test cluster with KubeVirt v1.9.0 and ingress-nginx.
Without the fix the controller logs `Error obtaining Endpoints for
Service "cozy-kubevirt/vm-exportproxy"`. With the backend renamed and
the tenant allowed egress to virt-exportproxy, a passthrough request to
`vm-exportproxy.<host>` reaches the export proxy, and it answers with
its own serving certificate.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubevirt): requests to vm-exportproxy.<host> now reach the KubeVirt export proxy: the Ingress and TLSRoute route to the virt-exportproxy Service KubeVirt creates, and tenants are allowed egress to the export proxy.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Tenant workloads can now connect to the KubeVirt export proxy over TCP
port 8443.
* **Bug Fixes**
* Updated export-proxy Ingress and TLSRoute routing to use the
`virt-exportproxy` backend, keeping export-proxy traffic directed to the
correct service.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 11:19:32 +03:00
Aleksei Sviridkin a1416655f4 fix(tenant): allow tenant egress to the KubeVirt export proxy
The vm-exportproxy Ingress is served by the ingress controller in
tenant-root, and tenant egress is default-deny. Nothing allowed traffic
to virt-exportproxy in cozy-kubevirt, so the Ingress stayed unreachable
even with the right backend Service. The new policy renders wherever
allow-to-cdi-upload-proxy does, but narrows the destination to the
export proxy pods and their container port.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 06:23:28 +03:00
Alexey Artamonov e63b9e439c fix(monitoring): count a central request as the receiver admitted it
The limiter refuses for good any request it counts above burst, and
burst was the receiver's 4 MiB cap. A request grows on its way to the
limiter: the sampler adds a tracestate to every span it keeps, and the
limiter counts a request as the collector re-encodes it, which writes
the empty value of every attribute the wire may omit, so a valid
request of spans with empty-valued attributes is counted 1.4x. A
request the receiver admitted near its cap was therefore refused on
every retry.

The limiter now runs right after memory_limiter, so it counts the
request before sampling. The receiver caps a request at 3 MiB and
burst stays at 5 MiB, which holds 1.4x the cap; a larger burst would
also be what each pod start lets through above the rate. The rate is
counted before sampling, so the store keeps only the sampled share of
it. The texts say that a request above the cap gets a 400, which
senders do not retry, and that each collector rollout a tenant causes
lets that burst through again on every pod.

The replica bound that keeps that allowance small now applies only to
a central tenant's collector, whose render fails above ten replicas: a
collector writing to the tenant's own store has no rate, and the
replica counts it shipped with stay valid.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 177b207a27 feat(monitoring): hold each shared-central tenant to a write rate
The design has the per-tenant collector rate-limit its tenant, so that
a noisy or hostile tenant cannot exhaust a shared backend. The disk cap
alone does not do that: VictoriaTraces always keeps the last two days,
so a tenant writing faster than the cap allows over two days fills the
disk for every tenant on the store.

The platform sets the rate with
monitoring.tracingCentralTenantBytesPerSecond, and it reaches every
tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each
monitoring HelmRelease reads, so Flux upgrades them all; a rate read
with lookup at render time would never reach a tenant whose release is
already in sync. The platform writes the key only when it is set, since
every release in every tenant reads that Secret. Without it the
collector holds the tenant to the rate of a default 10Gi store.

Each central tenant's collector splits the rate across its replicas
and refuses data above it before the batch processor, so the sender
sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the
limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which
the OTLP specification makes final, so central tenants should export
over HTTP.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Aleksei Sviridkin 3b5af9dc55 fix(tenant): skip Cilium policies on isp-hosted without Cilium
The isp-hosted variants install no CNI of their own, so on a host
without Cilium the cilium.io/v2 kinds do not exist and the tenant
install fails, taking the root tenant and everything that depends on
cozystack-basics down with it.

The policies now render on isp-hosted only when the host serves both
Cilium policy kinds. On every other variant Cozystack installs Cilium
itself, so they keep rendering unconditionally: a CRD that is missing
at render time there fails the install loudly instead of leaving the
tenant without network isolation, because helm-controller does not
re-render a release when only API discovery changes.

On isp-hosted without Cilium the tenant gets no network isolation from
Cozystack; isolating tenants is then up to the host's CNI.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 12:06:54 +03:00
Alexey Artamonov 69a16bd2da docs(tenant): shorten the tracingCentral description
The description ships in the godoc, the schema, the README, the
ApplicationDefinition and the dashboard's field help, and at about
2,800 characters it buried what an operator needs to decide on the
field. It now says what the field does, what tenant-root has to host
first, and who is inside the shared store's trust boundary; the
lifecycle details move to the user guide.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-01 18:02:22 +03:00
Alexey Artamonov 2d10565873 feat(tenant): let a tenant opt into shared-central tracing
tracingCentral turns the shared-central topology on for a tenant with
its own Monitoring: it publishes _namespace.tracingCentral for the
tenant's monitoring charts and renders the only two cross-boundary
paths that topology needs, the collector and Grafana each to the
tenant-root vmauth. Both come from one condition, so a tenant never
exports to the shared store without the path, or gets the path without
opting in, and tenant-root, which hosts the store, gets neither. The
flag is written only for tenants that opt in: every application in a
tenant reads that Secret, and a new key for all of them would
re-upgrade them all on the platform upgrade.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 22:23:47 +03:00
IvanHunters de359f858c docs(tenant): correct gateway=false line in the test rules table
The rules-under-test header still described gateway=false as "parked,
no inherit". It behaves like unset and still inherits. Align it with the
fixed comment below it.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 18:23:03 +03:00
IvanHunters eedb446e8a docs(tenant): clarify that gateway=false behaves like unset
The rewritten gateway description claimed setting false opts out of
Gateway publishing entirely, including inheritance. It does not:
tenant.gatewayEffective returns "false" for both an explicit false and
an unset key, so namespace.yaml keeps _namespace.gateway pointing at the
nearest ancestor that owns a Gateway and the tenant still publishes
through it. State that false currently behaves the same as unset, and
fix the matching stale comment in gateway_default_test.yaml.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 15:52:46 +03:00
IvanHunters 2dc884fe05 docs(tenant): fix stale gateway comment in namespace.yaml
The inline comment still described the old, incorrect behavior, that
tenant.gatewayEffective returns true for a custom non-derived host.
The resolver keys only on an explicit tenant.spec.gateway=true. Align
the comment with the code and the corrected parameter description.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 15:12:33 +03:00
IvanHunters 784d290c42 docs(tenant): correct gateway param description to match behavior
The gateway parameter description claimed the chart auto-enables a
Gateway for tenants whose apex is derived from the parent (empty host)
and leaves it off for custom-apex tenants. The resolver
tenant.gatewayEffective never inspects host: an unset gateway key
always means the tenant owns no Gateway and publishes through the
nearest ancestor that owns one, regardless of host. The unit tests in
gateway_default_test.yaml already pin this real behavior.

Rewrite the description to match the code and regenerate the schema,
README, Go types and the tenant-rd CRD.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 14:07:51 +03:00
Aleksei Sviridkin a2db2aeb8c fix(seaweedfs): raise the number of buckets a volume server can serve
A volume server started with -max 0 takes a slot count derived from disk
rather than a fixed one, replication 001 spends a slot on each of the
two servers per logical volume, and a collection claims COPY_2 volumes
whenever it grows. So the bucket ceiling is set by the product of the
size limit and COPY_2, and moving either number alone moves it. The
mechanism is upstream in MaybeAdjustVolumeMax (weed/storage/store.go)
and findVolumeCount (weed/topology/volume_growth.go).

Do not divide these out of the nominal PVC size, because the count
follows the space the filesystem makes available and not the volume's
advertised size: the shipped 10Gi volume PVC was seen to give nine slots
per server at the upstream 1000 and 19 at 500, which puts 400 at 23 to
24 by the same division, or eight collections against two. The span is
the observation's own, since the free space behind 19 at 500 fits more
than one filesystem geometry and those divide differently at 400; eight
either way, because a collection that cannot get a whole batch of COPY_2
keeps the volumes it did get and writes into them. The division itself
is measured: on a 1 GB image the master reports max as the filesystem's
available space over the limit in every run, 25 slots and nine
collections at the 40 and 3 that scale to this pair. Two is what the
platform's own cozy-backups bucket and a single tenant bucket already
use. Past that every S3 PUT to a new bucket times out while the disk
still reads as nearly empty, and no COSI object reports anything wrong,
because the claim and the access are both granted.

400 and 3 make the budget 1200 MB per collection. Splitting the
reduction across both numbers rather than taking it out of the size
limit alone keeps the cost off the curve that grows: open descriptors
track stored bytes divided by the limit, so a 10 TiB server keeps
roughly 52k of them at this limit against 21k upstream, where moving the
size limit alone would have put it at 105k. A .dat and an .idx stay open
per volume; the .vif beside them is written and closed. The ceiling
reached is the same either way.

Both figures describe the shipped replicationFactor of 2. The master
picks its growth count from the replication the filer asks for, so at 1
and 3 copies it reads COPY_1 and COPY_3 instead and this COPY_2 does not
apply; lowering the size limit still raises the ceiling there.

The size limit was out of an operator's reach: it lived in the
data-plane chart, which is pulled by ExternalArtifact from a component
that declares no install section, so a Package-level values override for
it is dropped before any merge. The app chart now declares
master.volumeSizeLimitMB and sends it on every release, and 400 is set
there. The data-plane chart keeps upstream's 1000 as its fallback on
purpose. A platform upgrade can re-render that release from its artifact
before the parent has sent anything new, and a lower fallback would then
reach the master first and drop the limit ahead of an operator's own
setting. COPY_2 is sent by nothing and lives in the data-plane chart, so
it does land in that window, where a grow costs 3000 MB against the 6000
MB it cost before.

The app chart also writes maxVolumes into dataDirs on every release, and
because dataDirs is a list Helm replaces the data-plane chart's copy
wholesale, so the zero that makes the slot count auto-derived at all
comes from there. Losing it drops the server to the subchart's fixed
seven without failing any render.

The field is bounded at both ends, because either end takes an instance
down. Zero is not a smaller limit but an off switch: the volume server
leaves its slot calculation untouched at zero, keeps reporting no
capacity, and nothing is ever grown. Past 30000 the master does not
start at all. The bound sits in the values schema rather than in a
template guard, so Helm refuses the value before any template runs. The
apps API does not: it reads that schema only for defaults, so it accepts
a value out of range, and the release then fails to render. An explicit
null gets past both bounds, because Helm drops the key before it
validates, and would render an empty -volumeSizeLimitMB= that the master
exits on at start, measured on the shipped image, so the template
requires the value.

No bound against the disk goes with it, though the field invites one. A
limit at or above the volume's own size does not break an instance:
MaybeAdjustVolumeMax floors the slot count at one, measured on 4.31 with
a 255 MiB disk against a 1000 MB limit, which reported Max 1 and served
a write it then read back. Such a server is capped at a single slot
rather than dead, so refusing it at render would be a capacity policy
that stops a working release on every reconcile, not only on the
upgrade. The field's description carries the relation instead, where the
value is chosen.

The growth count is deliberately not exposed beside the limit. The
master picks which of its four growth counts to apply from the
replication the filer requests, and the filer takes that from
replicationFactor, so a field wired to COPY_2 would do nothing at 1 or 3
copies while still appearing to be a setting. A knob that is silently
inert for two of the three supported values is worse than no knob.

Two further prices. A collection absorbs the budget before it needs
another grow, so growth requests arrive five times as often as at the
upstream pair. A collection also starts with half as many writable
volumes, and a volume server takes the writes into each volume one at a
time (syncWrite holds that volume's dataFileAccessLock), so a heavily
parallel writer spreads over fewer volumes until the collection grows
again; that holds for instances kept at 1000 too. And lowering the limit
on an instance that already holds data carries the other one: its
volumes between the new limit and the old go oversized on the next
heartbeat and leave the master's writable set, and each affected
collection needs one successful grow before its next write lands.
Nothing is lost and the volumes stay readable; the room left inside them
for further writes is given up and does not return as capacity.

Those grows compete for one pool, and the pool is smaller than it looks.
An oversized volume is charged no headroom, while one below the limit is
charged all of its unfilled room; a grow then claims a whole COPY_2
batch and takes as much of it as is there, and the volumes it creates
are empty and charge their headroom straight back. Measured on a local
SeaweedFS 4.31 setup: two oversized collections with two free slots
between them, the first to write took both, and the second stayed
refused with the disk still reporting a quarter of itself free.
Collections that need no grow are not free either, since their volumes
below the new limit were created six at a time and each can hold back a
whole limit. Also measured there, with the limit and batch scaled down:
a server with free space for one batch per collection refused the grow
its one affected collection needed, because a second, nearly empty
collection held the room, and the same free space without it let the
grow through. The minFreeSpacePercent reserve only comes into it on a
server whose writable volumes have all passed the limit, where nothing
is charged headroom: from 8000 MiB of filesystem upward, a 10Gi PVC
measuring 9979, five percent is at least 400 MB, and such a server then
refuses a grow only once its disk is down to the reserve. Everywhere
else the headroom charge decides, and a refusal can arrive with the disk
far from full whatever the reserve. Deleting a collection gives the room
back a slot per volume, however little it stored, since the headroom
charged to those volumes leaves with them.

So, on a variant that runs platform migrations, the upgrade does not
change the limit on an existing instance unless its operator already set
one, which the previous chart accepted and ignored. Platform migration
57 writes master.volumeSizeLimitMB: 1000 into every existing SeaweedFS
instance that has not chosen a limit, and 400 applies only to instances
created from here on. It leaves alone only an instance whose master is
not an object, which no v1 chart renders, and names that one in its log.
The default variant runs no platform migrations, so a cluster there that
installed SeaweedFS by hand gets no pin, and the document below gives
the same patch to apply before upgrading. A chosen limit is a whole
number the new chart accepts; anything else counts as not chosen and is
replaced with 1000, which is what the previous chart ran in its place,
since it ignored the key and the new chart would refuse to render it. A
valid value already set is kept, and since the previous chart ignored
it, the upgrade is when it starts to apply; the document below lists the
ones that change the limit ahead of it. Raised values are listed as
well: existing buckets keep writing, but every volume below the new
limit is charged up to it, and on a local SeaweedFS 4.31 setup seven
nearly empty volumes left no free slot once the limit went up tenfold,
so a new bucket was refused. The value goes into the application's
HelmRelease, whose spec.values is the application spec, for an instance
created through the API and for one the tenant module renders alike. The
tenant module never renders spec.values on that release, and a Helm
upgrade leaves a key alone that neither manifest carries: measured with
Helm 4.3.0 on client-side, server-side and forced server-side apply, and
read in createPatch at the v4.1.1 that helm-controller v1.5.0 runs.
Where the migration runs, it is a pre-upgrade hook of the platform
release, and the cozystack-packages source every application chart comes
from is a manifest of that same release, applied only after its hooks;
so no instance meets the new chart unstamped, and a failed migration
stops the platform upgrade before that source moves. That holds while
the platform source is pinned by digest, as it is out of the box; one
that follows a tag, a semver range or a branch lets source-controller
move cozystack-packages on its own schedule. Until cozystack-packages
moves, each patched release keeps its old chart, which ignores the key,
since no v1 release reads it or closes the master object in its schema.
An instance kept at 1000 still gains, because at a replication factor of
2 a grow now claims three volumes rather than six, so it has room for
roughly twice as many new buckets.

Lowering an existing instance is left to its operator, and
docs/operations/seaweedfs-lowering-the-volume-size-limit.md is the check
for it. It lists each server's volume files and adds one limit for every
volume below the limit and a batch for every collection with no volume
at or below 90 percent of it, since the master grows such a collection
on its own schedule without a write; measured the same way, a collection
with every volume in that band and no writes got two new volumes per
server within one pass. Where each disk type is served by as many volume
servers as the replication factor, as the shipped two at a factor of 2
are, each collection's volumes sit on every server of that type, and
that bounds the claim from above without assuming how the collections
grew. The master places a new volume on any server of the requested disk
type with room rather than where its collection already lives, whichever
group or pool that server belongs to, and it reads an empty disk type as
hdd; under MultiZone it also grows each zone on its own. So the document
keeps zoned instances, and any disk type served by more servers than
copies, at 1000.

BREAKING CHANGE: on the default variant, which runs no platform
migrations, set master.volumeSizeLimitMB to 1000 before upgrading on
every SeaweedFS instance that has no valid value and whose master is an
object, with the step in
docs/operations/seaweedfs-lowering-the-volume-size-limit.md. On the
variants that run platform migrations, migration 57 does this. If the
step is skipped, every such instance drops to 400 on the upgrade: each
of its volumes at or above 400 MB leaves the master's writable set, and
a bucket whose volumes have all crossed that line takes no writes until
it can grow.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-17 00:54:48 +03:00
Aleksei Sviridkin 487cafac3e feat(platform): let a tenant pick its Gateway class from an allowlist
Tenant Gateways were pinned to the bundled Cilium class. Give the
platform a default in gateway.className, the set a tenant may choose
from in gateway.tenantSelectableClasses, and the set whose provider
terminates TLS upstream of the Gateway in
gateway.edgeTerminatedClasses. A tenant picks its own with
tenant.spec.gatewayClass, read only when it owns a Gateway; an
inheriting tenant runs on its ancestor's.

The selectable set is an allowlist rather than free choice because
Tenant is a tenant-writable object while a GatewayClass is
cluster-scoped: without it a tenant could attach its Gateway to a class
an admin keeps for internal-only traffic. Naming anything outside the
set fails that tenant's own gateway release and reaches no other
tenant.

Where TLS ends is a property of the class, so the gateway chart derives
certMode from whichever class the tenant landed on instead of taking it
as a separate input. The two therefore cannot be configured out of
step.

The publishing tenant cannot cross that line by itself. The
cluster-wide ACME issuers point their HTTP-01 solver at its Gateway and
are rendered from the platform default alone, so a mismatch would pin
the solver to a listener that Gateway does not have. Those issuers now
drop the sectionName pin when the default class is edge-terminated,
letting the challenge attach by hostname instead.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-04 14:26:54 +03:00
Aleksei Sviridkin c681ae1243 test(tenant): anchor the operator-gate absence and refresh a spent reason
The two "carries no operator gate" tests asserted only
`notExists: spec.dependsOn`. A bare absence assert passes on a template
that renders no document, so toggling `seaweedfs` and `gateway` off left
both green while the releases they guard were gone. Measured: all 5 tests
passed with the templates rendering nothing, and 2 of them fail once the
name is pinned alongside the absence.

An earlier check on these two asserts injected a `dependsOn` and watched
them redden, which proves only that they catch an added gate. That is the
other direction; the one that matters for a negative assert is the
document disappearing.

Both suite headers also justified their `templates:` list with a reason
that has expired: keycloakgroups.yaml now guards its `.Values._cluster`
index, so a no-values render of the whole chart succeeds. The list is
still load-bearing, for the reason the sibling suite already records. It
keeps an unrelated template out of this suite's result: with a nil
dereference added outside the guard of a template neither suite asserts
on, both suites stay green with their list and error without it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:43:08 +03:00
Aleksei Sviridkin 92ec49e64e fix(tenant): retry failed monitoring and gateway installs in place (#3633)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

The tenant chart renders monitoring and gateway with no install
strategy, so they get Flux's default, and install remediation is an
uninstall in every case Flux offers. Neither sets
`spec.uninstall.disableHooks`, which defaults to false. So a release
that missed nothing but a readiness budget gets uninstalled, and the app
chart's `post-delete` cleanup Job runs against it.

Those Jobs delete on purpose, and they are right to. The monitoring one
removes PVCs labelled
`apps.cozystack.io/application.name=<release>-system`. That label is not
incidental either: the metrics and logs claim templates stamp it so the
operator copies it onto the storage PVCs, and the Job selects on it to
stay release-scoped. Both ends were built to meet. The result is that a
slow first install deletes the tenant's metrics and logs, and nothing
reconstructs them.

The gateway Job deletes the `cozystack-acme-account` Secret. That key
regenerates, so this one costs no data. It costs a re-registration
against the ACME provider, and repeated cycles run into its rate limits,
which can hold certificate issuance for the tenant down for hours.
Smaller stake, same mechanism, and the two are not one class: only one
of them owns storage.

`RetryOnFailure` keeps the manifests applied and retries the failed
release, so no uninstall happens and neither Job runs. Both actions
carry the strategy because the controller performs that retry as an
upgrade, so with install alone the retry's own failure falls through to
upgrade remediation. The remediation blocks stay: an absent release
consults install, an out-of-sync one consults upgrade, both by retry
count rather than by strategy.

Two things are traded away. A real upgrade failure now retries instead
of rolling back, which gives up no stable safety net, since with
`retries: -1` the default strategy was already rolling back and
re-upgrading without end. And the uninstall was the only automatic clean
slate, so an install wedged on a bad resource state retries against that
state until an operator removes the release. For a release that owns
data that is the right exchange.

This removes the consequence rather than the trigger, and for monitoring
the trigger is worth naming separately: the parent HelmRelease here
waits `10m`, while the child `monitoring-system` release sets `15m`
deliberately, with a comment explaining that 10m is too short for the
OIDC users-Job inside its `activeDeadlineSeconds: 900`. The parent's
wait covers the child, so the parent can fail while the child is
behaving exactly as designed. That mismatch is untouched here and wants
its own change.

The chart's etcd and seaweedfs releases lose storage the same way and
are outside this change. Beyond those four, computeplane, ingress and
info stay on the default strategy, but their charts ship no destructive
`post-delete` hook, so remediation costs them a wasteful teardown rather
than data.

Same shape as #3552 and #3581. Relates to #3624.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(tenant): retry a failed monitoring or gateway install instead of uninstalling the release, so a slow first install no longer runs the cleanup hooks that delete the tenant's metrics and logs volumes or its ACME account key
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Reliability Improvements**
* Monitoring and gateway deployments now automatically retry failed
installs and upgrades every 30 seconds.
* Existing unlimited remediation retries remain available, improving
recovery from deployment issues.
* **Validation**
* Added automated coverage to verify retry behavior for both monitoring
and gateway deployments.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-08 14:04:20 +03:00
Aleksei Sviridkin 36f83fa9b7 fix(tenant): retry failed etcd and seaweedfs installs in place (#3630)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

The tenant chart renders the etcd and seaweedfs releases with no install
strategy, so they get Flux's default, and install remediation is an
uninstall in every case Flux offers. Neither release sets
`spec.uninstall.disableHooks`, which defaults to false, so that
uninstall runs the app chart's `post-delete` cleanup Job. Nothing
unusual is configured anywhere along that chain: fail early, uninstall,
hooks.

Those Jobs delete data on purpose. The etcd one removes the
`data-etcd-<ordinal>` PVCs by name, because the etcd-operator does not
reclaim them when the EtcdCluster goes. The seaweedfs one removes the
volume PVCs by label and explicitly reclaims the keep-protected CNPG
Cluster holding the filer metadata, because nothing else would. Both are
right for a real uninstall, and neither hook is at fault here.
Remediation is what calls them, on a release that is only starting
slowly.

So an install that misses its budget does not restart. It destroys the
datastore the tenant control planes keep their state in, or the object
store's data together with the metadata database that indexes it, and
the next attempt begins from nothing.

That budget is not a comfortable margin. The seaweedfs release has
`timeout: 10m`, and `hack/e2e-install-cozystack.bats` already records
that on a loaded runner the tenant stack only starts creating pods
around 9 to 10 minutes in. The window CI observes and the window that
triggers this are the same window.

`RetryOnFailure` keeps the manifests applied and retries the failed
release as an upgrade. No uninstall happens, so neither cleanup Job runs
and the install already under way carries on instead of starting over.

Both actions carry the strategy. The controller runs the retry of a
failed install as an upgrade, so with install alone the retry's own
failure falls through to upgrade remediation. The remediation blocks
stay: an absent release consults install, an out-of-sync one consults
upgrade, both by retry count rather than by strategy.

One trade is that a real upgrade failure retries instead of rolling
back. That costs no stable safety net. With `retries: -1` and the
default strategy, upgrade remediation was already rolling back and
retrying without end, so the release never rested in the rolled-back
state either.

The other side of the trade is that the uninstall was also the only
automatic clean slate. On a tenant holding no data yet, an install that
wedges on a bad resource state now retries against that state until an
operator removes the release. For a release that owns data that is the
right exchange, because lost data does not come back and a wedged
install does.

This covers two of the tenant's data-owning releases. The monitoring
release carries the same default and a cleanup Job that deletes its PVCs
as well, and is left for its own change, as is the rest of the catalog,
whose nested releases want the question asked one at a time. Same shape
as #3552 and #3581. Relates to #3555 and #3576.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(tenant): retry a failed etcd or seaweedfs install instead of uninstalling the release, so a slow first install no longer runs the cleanup hooks that delete the etcd data volumes, the SeaweedFS volumes and the filer metadata database
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Reliability Improvements**
* Improved installation and upgrade recovery for etcd and SeaweedFS by
automatically retrying failed operations every 30 seconds.
* Preserved ongoing remediation attempts to help services recover
without uninstalling or rolling back after failures.

* **Tests**
* Added coverage to verify retry behavior and remediation settings for
both services.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-08 14:04:17 +03:00
Aleksei Sviridkin d956536461 fix(tenant): render the chart without the platform's injected values (#3636)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

`helm lint packages/apps/tenant` fails on main with three errors. This
removes two of the three; the branch name notwithstanding, lint still
exits 1 afterwards on the third, which is a repository-wide convention
question rather than a defect in this chart.

**The chart cannot be rendered outside a cluster at all.**
`templates/keycloakgroups.yaml` reads `index .Values._cluster
"oidc-enabled"`, and `_cluster` is injected by the platform, so it is
absent whenever the chart is rendered on its own. `index` on a nil map
is a hard error, not an empty result, so both `helm lint` and a plain
`helm template` die on line 1.

The idiom the neighbouring charts use for the same lookup does not
rescue this on its own, which is worth stating because it looks exactly
like a guard. `(index .Values._cluster "x") | default "y"` still fails:
`index` is evaluated first and errors on the nil map, so `default` never
receives a result to substitute; it only ever covered a key missing from
a map that exists.

The spelling alone does not decide whether a chart breaks, though.
`packages/system/cozystack-basics` uses it and lints clean, because line
1 of its `values.yaml` is `_cluster: {}`, which makes the map non-nil
before `index` runs. A chart is broken only when it neither declares the
empty map nor guards at the use site, and a lint sweep across the
catalog found 27 in that state. Guarding at the use site is the choice
here because it makes the template correct whatever `values.yaml`
contains, and because `_cluster` is platform-internal while this chart's
`values.yaml`, schema and README are generated together from
`api/apps/v1alpha1/tenant/types.go` as user-facing output. The guard has
to sit on the map rather than on the value it returns:

```
{{- $oidcEnabled := index (.Values._cluster | default dict) "oidc-enabled" }}
```

**This is behaviour-preserving, and that is checked rather than
asserted.** With `_cluster` present, `| default dict` returns it
untouched and the lookup is what it was; with it absent the lookup is
nil, the `eq` below is false, and the block stays unrendered, which is
what an unconfigured tenant should produce. Rendering before and after
gives byte-identical output:

| values | before | after |
| --- | --- | --- |
| `_cluster.oidc-enabled: "true"` | 4 KeycloakRealmGroups |
byte-identical |
| `_cluster.oidc-enabled: "false"` | no groups | byte-identical |
| no values at all | hard error | renders, no groups |

Worth recording how that comparison has to be run, because the obvious
way proves nothing. The KeycloakRealmGroup block sits behind
`.Capabilities.APIVersions.Has "v1.edp.epam.com/v1"`, which is false
under a plain `helm template`, so a first pass rendered the same output
in *both* the enabled and disabled cases, and a diff of those two would
have compared one empty result against another. The renders above pass
`--api-versions v1.edp.epam.com/v1` so the guarded branch actually
executes. Anyone proving neutrality on a capability-gated template needs
that flag or the proof is vacuous.

**`Chart.yaml` does not declare `cozy-lib`**, which is present as a
symlink under `charts/`, so lint reports missing chart metadata. It is
declared the way `packages/apps/qdrant` does it, that being the one
chart in the tree which declares it and does not carry this error.
Rendering is unchanged by the declaration, checked the same way as
above.

Two things that could plausibly have broken and did not, checked rather
than assumed: rendering from a packaged `helm package` artifact is
byte-identical, and the `charts/cozy-lib` symlink is still expanded into
the tgz with the dependency declared. And neither `hack/update-crd.sh`
nor the `fix-charts` target in `packages/apps/Makefile` reads the
`dependencies:` block, so nothing needs regenerating.

**The icon error is left in place on purpose, and the relative path is
load-bearing.** `helm lint` wants a URL with a scheme, and
`/logos/tenant.svg` has none. But `hack/update-crd.sh` consumes exactly
this form: it rewrites a leading slash to `./`, then reads that file off
disk and base64-encodes it into `.spec.dashboard.icon` of the generated
ApplicationDefinition. `packages/apps/tenant/logos/tenant.svg` is that
file. Give the chart a URL with a scheme and the generator's `[[ ! -f
"$ICON_PATH" ]]` check no longer finds anything to encode. So the
relative form is a requirement of the pipeline rather than a stylistic
convention, all 31 icon-carrying charts use it, and satisfying helm lint
here means changing the generator, not the charts. That is a
repository-wide decision and not this PR's.

`tests/keycloakgroups_oidc_test.yaml` pins the gate in all three
directions: four realm groups when `oidc-enabled` is `"true"`, none when
it is `"false"`, and none when the platform injected no `_cluster` at
all. The last case is the one the guard exists for, and it needs no
mocking, since `_cluster` is absent from `values.yaml` and "not
injected" is just the chart's default state. Against the unguarded
template that suite fails with `index of untyped nil`. The chart's other
seven suites pass identically with and without the fix, which is
precisely why the change needed its own.

Two limits of the new suite, stated rather than left to be found. It
pins how many documents render, not which: renaming the four groups or
breaking a `realmRef` would pass. And the template's second gate,
`.Capabilities.APIVersions.Has "v1.edp.epam.com/v1"`, is enabled for the
whole suite, so no case covers its absence. Both are worth closing on
the next touch of this file; neither weakens what the suite does pin,
which is that the chart renders at all when the platform injected
nothing.

A trap for anyone reproducing this by hand: the gate compares against
the string `"true"`, so `--set _cluster.oidc-enabled=true` passes a YAML
boolean and renders nothing, while `--set-string` renders the four
groups. The platform writes a genuine string, and it survives the
round-trip through a Secret because `toYaml` quotes it. Note also that
this chart's helpers reject a release name that does not begin with
`tenant-`, which will silently render nothing for either form.

One note in case a future change teaches the linter to inject the
platform's values, which would hide this class of error without fixing
it: the guard is still worth having, because it is what lets `helm
template packages/apps/tenant` succeed with no values file at all. A
chart that renders standalone can be inspected, diffed and unit-tested
without reproducing the platform around it.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(tenant): the tenant chart now renders without the values the platform injects, so `helm template` works on it standalone, and its `cozy-lib` dependency is declared in Chart.yaml; this clears two of the three `helm lint` errors it reported, the remaining one being the repository-wide relative icon URL
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Added support for using the shared Cozy library in tenant deployments.

- **Bug Fixes**
- Improved tenant configuration rendering when cluster settings are
unavailable.
- OIDC-related Keycloak groups now render reliably when OIDC is enabled
and required platform capabilities are present.
- Preserved expected behavior when OIDC is disabled or configuration is
omitted.

- **Tests**
- Added coverage for enabled, disabled, and missing cluster
configuration scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-08 10:55:04 +03:00
Aleksei Sviridkin 7b36a5d2f2 test(tenant): give these template lists their real reason (#3651)
## What this PR does

Two helm-unittest suites in `packages/apps/tenant/tests/` said they
restrict rendering because `namespace.yaml`'s `lookup` errors under
helm-unittest. It does not. The lookup returns empty offline and the
template renders, which `namespace.yaml`'s own header states and which
`tests/namespace_ancestor_labels_test.yaml` depends on.

I measured what each list actually does by removing it, and the two
suites turned out to be different cases rather than two copies of one.

In `cleanup_role_watch_test.yaml` the list is load-bearing, for a
template the comment never mentioned. That suite sets no values, and
`keycloakgroups.yaml` indexes `.Values._cluster` on its first line, so
the whole-chart render fails with `index of untyped nil` and all 8 tests
error. The comment now says that, and says it conditionally: a guard on
that index is already in flight, and once it lands the list stops being
load-bearing. I checked that too, by applying the guard and re-running
with the list removed, which passes 8/8.

In `vmo_webhook_dependson_test.yaml` the list is not load-bearing at
all. That suite sets `_cluster`, the chart renders without the list, and
all 5 tests pass. It stays only to scope the suite, and the comment says
so instead of inventing a technical reason.

Each comment now states what holds for its own file and nothing wider.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. It touches two files, both
under `packages/apps/tenant/tests/`, and no chart, template, values file
or tooling. Nothing in the map keys on test files.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Clarified test-rendering rationale and suite-scoping comments.
* Documented that rendering requires `_cluster` values and that template
lists control test scope.
* **Tests**
* Updated explanatory comments only; test behavior and configuration
remain unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-08 10:46:41 +03:00
Aleksei Sviridkin 1cb51c92aa test(tenant): give these template lists their real reason
Both suites here said they restrict rendering because namespace.yaml's
`lookup` errors under helm-unittest. It does not. The lookup returns empty
offline and the template renders, which namespace.yaml's own header states
and which tests/namespace_ancestor_labels_test.yaml depends on.

Measured what each list actually does by removing it.

In cleanup_role_watch_test.yaml the list is load-bearing, for a template
the comment never mentioned: the suite sets no values and
keycloakgroups.yaml indexes `.Values._cluster` on its first line, so the
whole-chart render fails with "index of untyped nil". Stated
conditionally, because a guard on that index is already in flight and
would end it.

In vmo_webhook_dependson_test.yaml the list is not load-bearing at all,
because that suite sets `_cluster` and the chart renders without it. It
stays only to scope the suite.

Each comment now states what holds for its own file and nothing wider.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 04:49:23 +03:00
Aleksei Sviridkin eac7e1ea15 test(tenant): anchor the upgrade.force guard to its HelmRelease
Each test asserted only `notExists: spec.upgrade.force`, which passes on a
template that renders no document at all. Renaming the value that gates one
of these releases makes the template render nothing and leaves its test
green: measured on etcd.yaml, where the guard stayed green while a sibling
suite's `equal` was the only thing that noticed the release had vanished.
Pin each HelmRelease name alongside the absence so the assertion has a
document to be about.

Cover computeplane.yaml. Seven templates in this chart emit a HelmRelease
and the list held six; the module arrived after the suite was written. The
invariant holds there today, so this closes an unguarded template rather
than a live regression — which is the same gap that produced this suite,
whose first pass missed gateway.yaml.

Correct two claims in the header. The template restriction was said to be
there because namespace.yaml's `lookup` errors under helm-unittest; it does
not, the lookup returns empty offline and the template renders, which is
what its own header says and what namespace_ancestor_labels_test.yaml
relies on. The restriction stays, for the reason that actually applies: it
keeps an unrelated template breaking out of this suite's result.

helm-controller also does not deprecate upgrade.force. The CRD vendored at
internal/fluxinstall/manifests/fluxcd.yaml, which is what the pinned Flux
2.8.x ships, describes it as a replacement strategy and carries no
deprecation marker; `recreate` is the neighbouring field that does. Later
upstream documentation adds that the field is ignored under server-side
apply, and that sentence is not in the pinned CRD, so the comment says
which is which. The reason to keep the flag off is client-side apply,
where it replaces resources instead of patching them.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 03:09:46 +03:00
Aleksei Sviridkin 6d403c3015 fix(tenant): retry failed etcd and seaweedfs installs in place
The tenant chart renders both releases with Flux's default install
strategy, and install remediation is an uninstall in every case Flux
offers. Neither release disables hooks on it, so a remediated install
runs each chart's post-delete cleanup Job, and those Jobs delete data on
purpose. The etcd one removes the data-etcd-<ordinal> PVCs by name,
because the etcd-operator does not reclaim them when the EtcdCluster
goes. The seaweedfs one removes the volume PVCs by label and explicitly
reclaims the keep-protected CNPG Cluster holding the filer metadata,
precisely because nothing else removes it. Both are correct for a real
uninstall. Remediation is what calls them, on a release that is only
starting slowly.

So an install that misses its budget does not merely restart. It
destroys the datastore the tenant control planes keep their state in, or
the object store's data together with the metadata database that indexes
it, and the next attempt begins from nothing.

RetryOnFailure keeps the manifests applied and retries the failed
release, so no uninstall happens and neither cleanup Job runs. Both
actions carry the strategy because the controller performs that retry as
an upgrade, so on install alone the retry's own failure falls through to
upgrade remediation. The cost is that a real upgrade failure retries
rather than rolls back, which gives up no stable safety net: with
retries: -1 and the default strategy, upgrade remediation already rolled
back and retried without end. The remediation blocks stay because an
absent release consults install and an out-of-sync one consults upgrade,
both by retry count rather than by strategy.

The other side of the trade is that the uninstall was also the only
automatic clean slate. On a tenant holding no data yet, an install that
wedges on a bad resource state now retries against that state until an
operator removes the release. For a release that owns data this is the
right exchange, because lost data does not come back and a wedged
install does.

This covers two of the tenant's data-owning releases. The chart's
monitoring release destroys its storage on a remediated install as well
and is outside this change, as is the rest of the catalog, whose nested
releases want the question asked one at a time.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 02:26:31 +03:00
Aleksei Sviridkin bd3f3b93d0 fix(tenant): render the chart without the platform's injected values
`helm lint packages/apps/tenant` fails, and two of its three errors are
the chart's own.

`templates/keycloakgroups.yaml` reads `index .Values._cluster
"oidc-enabled"`. `_cluster` is injected by the platform, so it is absent
whenever the chart is rendered on its own, and `index` on a nil map is a
hard error: `index of untyped nil`. The chart cannot be rendered or
linted outside a cluster at all.

The idiom the neighbouring charts use for the same lookup does not
rescue this on its own, which is worth stating because it looks like it
does. `(index .Values._cluster "x") | default "y"` still errors: `index`
evaluates first and fails on the nil map, so `default` never sees a
result to substitute; it only ever covered a key missing from a map that
exists.

The spelling alone does not decide whether a chart breaks, though.
`packages/system/cozystack-basics` uses it and lints clean, because line
1 of its values.yaml is `_cluster: {}`, which makes the map non-nil
before `index` runs. So a chart is broken only when it neither declares
the empty map nor guards at the use site. Guarding at the use site is
preferred here because it makes the template correct whatever
values.yaml contains, which is what lets the chart render standalone;
and `_cluster` is platform-internal, while this chart's values.yaml,
schema and README are generated together from
api/apps/v1alpha1/tenant/types.go as user-facing output.

The guard has to sit on the map rather than on the value it returns,
hence `index (.Values._cluster | default dict) "oidc-enabled"`.

That is behaviour-preserving rather than merely quieter. With `_cluster`
present, `| default dict` returns it untouched and the lookup is what it
was. With it absent the lookup is nil, the `eq` below is false, and the
block stays unrendered, which is what an unconfigured tenant should
produce. Rendered both ways before and after, with
`--api-versions v1.edp.epam.com/v1` so the capability-gated block
actually executes, the output is identical byte for byte: four
KeycloakRealmGroups when `oidc-enabled` is `"true"`, none when it is
`"false"`. Rendering with no values at all goes from a hard error to a
clean render with no groups.

`Chart.yaml` also does not declare `cozy-lib`, which is present as a
symlink under `charts/`, so lint reports missing chart metadata. Declare
it the way `packages/apps/qdrant` does, that being the one chart in the
tree that declares it and does not carry this error. Rendering is
unchanged by the declaration, checked the same way.

`tests/keycloakgroups_oidc_test.yaml` pins all three branches of the
gate: four realm groups when `oidc-enabled` is `"true"`, none when it is
`"false"`, and none when the platform injected no `_cluster` at all.
That third case is the one the guard exists for, and it needs no mock,
because `_cluster` is absent from values.yaml and so "not injected" is
simply the chart's own default state. Against the unguarded template the
suite fails with `index of untyped nil`; the chart's other seven suites
pass identically either way, which is what left the change uncovered.

The third error, `invalid icon URL '/logos/tenant.svg'`, is deliberately
left. Lint wants a URL with a scheme, and all 31 charts in the tree that
carry an icon use a relative path, none an absolute one. Changing this
one would make it the only chart with a different convention, and
choosing the URL a relative path should become is a decision about where
those assets are served from, not about lint.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 02:25:05 +03:00
Aleksei Sviridkin a27c341843 fix(tenant): retry failed monitoring and gateway installs in place
Both releases carry Flux's default install strategy, and install
remediation is an uninstall in every case Flux offers. Neither disables
hooks on it, so a remediated install runs the app chart's post-delete
cleanup Job against a release whose only fault was missing a readiness
budget.

The monitoring Job deletes PVCs labelled
apps.cozystack.io/application.name=<release>-system. That label is not
incidental: the metrics and logs claim templates stamp it precisely so
the operator copies it onto the storage PVCs, and the cleanup Job
selects on it to stay release-scoped. A slow first install therefore
deletes the tenant's metrics and logs, and nothing reconstructs them.

The gateway Job deletes the cozystack-acme-account Secret. That key
regenerates on the next install, so this one costs no data. It costs a
re-registration against the ACME provider, and repeated cycles run into
its rate limits, which can hold certificate issuance for the tenant down
for hours.

RetryOnFailure keeps the manifests applied and retries the failed
release, so no uninstall happens and neither cleanup Job runs. Both
actions carry the strategy because the controller performs that retry as
an upgrade, so on install alone the retry's own failure falls through to
upgrade remediation. Two things are traded away. A real upgrade failure
now retries rather than rolls back, which gives up no stable safety net,
since with retries: -1 and the default strategy upgrade remediation
already rolled back and retried without end. And the uninstall was the
only automatic clean slate, so an install wedged on a bad resource state
retries against that state until an operator removes the release. The
remediation blocks stay because an absent release consults install and
an out-of-sync one consults upgrade, both by retry count rather than by
strategy.

The chart's etcd and seaweedfs releases lose storage the same way and
are outside this change. Beyond those four, computeplane, ingress and
info remain on the default strategy, but their charts ship no
destructive post-delete hook, so remediation costs them a wasteful
teardown rather than data. That is a different question from this one.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 02:06:53 +03:00
Aleksei Sviridkin cf85716444 fix(tenant): bound the pre-delete cleanup waits and the hook Job
kubectl reads --timeout=0 as "wait forever" and substitutes a week,
so the two `kubectl delete helmreleases --wait=true` calls in the
tenant pre-delete hook hang for as long as any child HelmRelease
cannot finalize. helm returns from Uninstall.Run before deleting
anything, so the release stays in storage and flux retries the same
teardown; helm.sh/hook-delete-policy carries no hook-failed, so the
waiting Job survives between attempts as well.

Bound each wave at 90s. Neither is guarded: nothing stands behind
these deletes, so a wave that did not finish ends the run rather than
report a cleanup that did not happen, and ending it keeps the tenant
modules from being torn out from under applications that are still
deleting. Whatever was deleted stays deleted, so the next attempt
resumes from there.

The bound has a price and it is not zero. A nested Tenant, or an app
of kind Kubernetes whose own hook spends two 120s waits first, misses
90s and converges across uninstall retries instead; neither fits the
300s budget as it stands either. Between those and the fast case sits
a band that does pay: a wave inherits its slowest child, so one that
settles around 95s completes inside the budget today and expires here
instead, trading a clean attempt for one more retry. An unbounded
wave does not fail, it spends the whole budget and reports a timeout
naming nothing, so the trade is worth taking.

Give the Job an activeDeadlineSeconds of 270. It bounds what the
waits do not — the DELETE calls carry kubectl's default
--request-timeout of 0 — and sits under the 300s uninstall budget
flux applies when the HelmRelease carries no timeout of its own,
because the deadline runs from the Job's startTime while that budget
runs from just before the Job is created. At equal values helm gives
up first, on a Job still running.

Cap the pod's termination at 10s, which is part of that arithmetic
rather than a detail of it. Since 1.31 the Job controller withholds
the terminal condition until every pod has finished terminating, so
helm learns of the failure at the deadline plus this period; the 30s
default would land it at 270+30, level with the budget the deadline
is set under. The period is spent in full: the SIGTERM goes to PID 1,
which is a shell running a multi-command script, and a default
disposition sent to PID 1 is discarded by the kernel with no handler
installed to catch it. kubectl is a child and is never signalled.

Set backoffLimit to 0 and restartPolicy to Never. The limit is needed
under either policy, since the default of 6 keeps the failure from
reaching helm inside the budget. The policy matters because under
OnFailure the pod survives its own failure and the kubelet re-runs
the container in place, with the controller applying the cap only
afterwards by summing restartCount: the script re-enters the wave
that just expired and is cut down mid-run. Never leaves a failed pod
the controller sees directly.

Add resource requests to match the sibling hook in
packages/apps/kubernetes. Without them the pod is BestEffort and
first out under node pressure, which a retry inside the Job used to
absorb and a single attempt does not.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-07 14:38:06 +03:00
Andrei Kvapil 8fc8d23dfd fix(computeplane): address review round three
Gate the Proxied ingress Service on _namespace.etcd like the addon
HelmRelease it fronts, so an etcd-less tenant with the addon pinned on does
not get a selector-less Service (+ test). Correct the secrets framing: the
tenant-visible surface is an allowlist (include: [] withholds everything),
so drop the never-load-bearing exclude entry, the bats cross-check and the
rename-coupling comments, and state the real mechanism. Fix the etcd
prerequisite docs (an ancestor's etcd satisfies the wrapped chart; the
awaiting-etcd beacon is the diagnostic), reword the tenant toggle to stop
advertising the not-yet-shipped placement routing, require maxReplicas >= 1
on the ingress-role node group, and fix comment drift (Service name after
the release rename, placement-hook label, package-guard lookup semantics).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:03 +02:00
Andrei Kvapil 6dcf5373e7 fix(computeplane): let Flux derive both release names
The aggregated API cannot represent spec.releaseName: the conversion
rebuilds HelmRelease specs without it, so the first tenant edit through the
API dropped the module's explicit computeplane-module release name, let it
default to the object name computeplane, and collided with the Helm storage
record the cluster release owned — with the tenant chart writing the old
name back every interval. Drop releaseName from both HelmReleases and derive
names from object names, like every other tenant module. Kamaji now writes
the admin kubeconfig to computeplane-cluster-admin-kubeconfig; the
computeplane-rd secrets.exclude entry follows in the same change, and a bats
test pins the exclusion to the name the cluster release actually produces so
a future rename cannot silently re-expose the credential. Also add the
check-release-name guard the other extra modules carry, and pin the fluxcd
addon's distribution artifact to the flux-operator version vendored in
system/fluxcd-operator instead of :latest.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:03 +02:00
Andrei Kvapil 76374f6472 fix(tenant): fail loudly when computeplane is enabled without its package
The computeplane-application package ships in the iaas bundle while the
toggle arrives with tenant-rd in the system bundle, and bundles are
independent booleans. Refuse the render with a clear diagnostic instead of
producing a HelmRelease that retries a nonexistent artifact forever.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
Andrei Kvapil b2232904b6 feat(tenant): add the computeplane module toggle
Wire the computeplane module into the tenant chart as a bool toggle, matching
the other tenant modules. The module's own Helm release takes the -module
suffix so the canonical release name computeplane (and with it the
computeplane-admin-kubeconfig Secret) stays reserved for the cluster
HelmRelease the module renders.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
Andrei Kvapil 1c7eb22918 feat(tenant): allow egress to the grafana-dashboards plugin server
Tenant egress is default-deny plus explicit allow-to-* policies, so the
Grafana pods deployed per tenant could not reach the in-cluster plugin
server in cozy-grafana-operator. Mirror the allow-to-keycloak pattern
and cover the policy with a test.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-24 16:08:51 +02:00
IvanHunters 1ac9398a4b fix(tenant): emit full ancestor label chain on tenant namespaces
The tenant.cozystack.io/<ancestor> namespace labels drive the
<ancestor>-egress CiliumClusterwideNetworkPolicy, which lets an ancestor
tenant reach a descendant namespace only when the descendant carries
tenant.cozystack.io/<ancestor>. namespace.yaml derived that chain by
splitting .Release.Namespace, which only reached one level up: a direct
child of tenant-root got tenant-root, but a grandchild (tenant-ktj-htdev
under tenant-ktj) got only tenant-ktj and dropped tenant-root entirely.
The root ingress controller in tenant-root then could not reach nested
tenants, surfacing as upstream timeouts.

Derive the chain deterministically from the namespace name instead. Every
tenant descends from tenant-root, and tenant.name builds each child as
<parent-namespace>-<word>, so every progressive dash-prefix of the name is
a real ancestor namespace. This needs no cluster lookup, renders offline
(so it is unit-testable), and converges on every reconcile regardless of
parent/child HelmRelease ordering. The ownerReference lookup is made
non-fatal so the template still renders when the parent namespace lookup
is empty; in a real reconcile the parent always exists and the
ownerReference is always emitted.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-03 09:51:49 +03:00
Aleksei Sviridkin 6db5ae9c6f fix(tenant): gate VM-CR-creating HelmReleases on victoria-metrics-operator
The tenant etcd, ingress and monitoring HelmReleases create VM* custom
resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert,
VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system
child of monitoring) guarded by the victoria-metrics-operator validating
webhook with failurePolicy: Fail. On a cold install the operator pod can
be briefly unavailable on :9443, so these releases reconcile before the
webhook backend is serving and are rejected with connection refused. They
carry install.remediation.retries: -1 and would eventually recover, but
the failed-install -> rollback -> retry churn can outlast a cold-install
readiness deadline and surfaces as spurious install failures.

Add a Flux spec.dependsOn from each of these releases to the
victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator,
so they wait for the operator to be Ready before creating VM* resources.
The system monitoring components gate on the operator the same way, via a
variant-level dependsOn. The cross-shard reference (tenant releases are
sharded, the operator release is not) resolves via helm-controller's
uncached dependency check (APIReader, v1.1.0+).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-01 12:05:23 +03:00
Aleksei Sviridkin e1f6a9c2aa fix(tenant): run the cleanup hook as non-root under a restrictive securityContext
The pre-delete cleanup Job ran kubectl as root with no securityContext.
Add a pod-level runAsNonRoot + runAsUser 65534 + seccompProfile
RuntimeDefault and a container-level allowPrivilegeEscalation: false plus
capabilities drop ALL, mirroring the dashboard adopt hook that runs the
same clastix/kubectl image non-root. Pin the securityContext in
helm-unittest.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 15:40:26 +03:00
Aleksei Sviridkin 5e7936c811 fix(tenant): replace bitnami/kubectl with maintained clastix image in cleanup hook
The pre-delete cleanup Job ran kubectl from `bitnami/kubectl:latest`. The
bitnami Docker Hub catalog has moved to a legacy/sunset state, so that pull is
at risk of breaking — which would stop the cleanup hook from starting at all —
and the `:latest` tag leaves the kubectl version unpinned and non-reproducible.

Switch to `docker.io/clastix/kubectl:v1.32`, the version-pinned image already
used by every other first-party cleanup/delete hook in the repo (kubernetes,
keycloak-configure, dashboard, mariadb). That image is Alpine-based and ships
only `/bin/sh`, so the hook script (pure POSIX) now invokes `/bin/sh` instead
of `/bin/bash` — matching those sibling hooks.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 15:40:26 +03:00
Aleksei Sviridkin ed30425b17 fix(tenant): grant watch verb to cleanup Role so pre-delete hook can wait
The pre-delete cleanup hook deletes the tenant's HelmReleases with
`kubectl delete --wait=true`, which opens a watch on helmreleases to block
until finalization. The hook Role granted only get/list/delete, so the watch
was rejected with 403, the delete call hung, the Job never finished, and
tenant deletion stalled on the hook.

Add the watch verb, matching the kubernetes app chart's cleanup Role which
already includes it for the same --wait deletes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 15:40:26 +03:00
Myasnikov Daniil 57c0ce66bc test: cover flux v2.8 chart fixes and wire missing test targets
- kafka: assert WorkloadMonitor reads .Values.kafka.replicas /
  .Values.zookeeper.replicas (regression for the phantom .Values.replicas),
  and that enableServiceLinks lives under entityOperator.template.pod.
- tenant: pin that no swept HelmRelease carries upgrade.force (would have
  caught the missed gateway.yaml).
- seaweedfs-db: baseline render cover for the new split-out chart.
- kafka and seaweedfs-db had no `test` Makefile target, so their suites never
  ran in CI; add one to each.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-31 15:02:33 +05:00
Myasnikov Daniil e3f4e34a0b fix(tenant): drop deprecated upgrade.force from gateway HelmRelease
The flux-v2.8 sweep removed `upgrade.force: true` from the sibling tenant
HelmReleases (etcd, info, ingress, monitoring, seaweedfs) for helm-controller
v1.5 SSA compatibility, but gateway.yaml was missed. Complete the sweep so no
tenant HR keeps the flag that conflicts with Server-Side Apply.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-31 15:02:11 +05:00
Myasnikov Daniil 7f94182a87 fix(platform): drop deprecated upgrade.force from HelmReleases; fix kafka WorkloadMonitor replicas paths
Flux v2.8 / helm-controller v1.5.0 embeds Helm v4, which renamed --force
to --force-replace and rejects combining it with SSA's --force-conflicts.
Existing HelmReleases with spec.upgrade.force: true fail every upgrade with
"cannot use force conflicts and force replace together", looping
upgrade -> rollback -> upgrade and blocking dependants. Observed on PR #2619
CI for tenant-root/{ingress-nginx-system,seaweedfs-db}, tenant-test/info,
and others.

Per Flux docs, upgrade.force has been a no-op since v2.8 anyway, so removing
it is safe on green clusters and unblocks broken ones.

Also fix packages/apps/kafka/templates/workloadmonitor.yaml: it referenced
.Values.replicas (no such field; values.yaml has kafka.replicas and
zookeeper.replicas). Flux v2.7 silently dropped the null; v2.8 SSA forwards
it, and the WorkloadMonitor CRD rejects "spec.replicas: null" as not
type:integer, which blocks every kafka-test install.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 7e286cf8e1)
2026-05-28 12:43:50 +05:00
Aleksei Sviridkin 97dff75b5f feat(tenant): inherit _namespace.gateway from parent
Mirror the existing _namespace.ingress inheritance for Gateway API
attachment. A child tenant under a Gateway-owning ancestor publishes
its routes through that ancestor's Gateway by default, without burning
a dedicated LB IP / Certificate per child.

Resolution rule (tenant.gatewayEffective):
  * gateway=true explicit  → own Gateway (LB IP + cert burned)
  * gateway=false explicit → parked (no inherit, no own)
  * gateway unset          → inherit $parentNamespace.gateway

Removes the auto-default for derived-apex tenants — the previous shape
gave every derived-apex child its own Gateway / IP / cert, which scales
linearly with tenant depth and produces ~N×K LB IPs on a tree of N
top-level × K derived-apex tenants. Inheritance collapses that to one
LB IP per Gateway-owning ancestor.

Custom-apex tenants (host explicitly set to a non-derived value) now
fall through to inheritance instead of auto-firing a separate Gateway.
The previous auto-fire conflated 'custom apex' with 'want public
exposure' and would have silently failed TLS termination anyway (the
ancestor's cert does not cover an unrelated apex). The new rule:
explicit gateway=true is the only way to ask for a separate Gateway.

The namespace.cozystack.io/gateway label and _namespace.gateway in the
rendered cozystack-values Secret now both carry the owner-tenant name
— could be self (when this tenant owns one) or any ancestor up the
chain. Apps in the tenant render HTTPRoutes whose parentRefs.namespace
points at that owner.

helm-unittest pinned for the HelmRelease render side. The
cross-namespace attach end of the flow (ReferenceGrants, listener /
SAN expansion on the owner Gateway) lands separately in the
TenantGateway controller.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-25 18:18:54 +03:00
Aleksei Sviridkin 039924f59d feat(gateway): align _namespace.gateway propagation with tenant.gatewayEffective
Two related fixes for the gap where the tenant chart's gateway HelmRelease materialises but child apps (harbor, bucket) silently fall back to ingress-nginx:

- packages/apps/tenant/templates/namespace.yaml: switch the
  _namespace.gateway resolution from raw .Values.gateway to
  tenant.gatewayEffective. This is the same helper consulted by
  gateway.yaml (HelmRelease render), so per-tenant Gateway
  materialisation and child-app HTTPRoute / Ingress selection now
  stay in lockstep — including the auto-default for derived-apex
  tenants.
- packages/system/cozystack-basics/templates/cozystack-values-secret.yaml:
  add a conditional gateway field to the tenant-root cozystack-values
  Secret. cozystack-basics is the only writer of _namespace.* for
  tenant-root (apps/tenant chart skips tenant-root via the existing
  ne 'tenant-root' gate). Without this line, tenant-root apps read
  _namespace.gateway as empty and route to ingress even when the
  platform's gateway.enabled flag is on. Conditional on
  _cluster.gateway-enabled='true' so non-Gateway clusters keep the
  empty value.

Test fixture in cozystack-basics/tests/cozystack-values-secret_test.yaml
pins the conditional and the regression-guard that the other four
_namespace keys (etcd, ingress, monitoring, seaweedfs) stay
unconditional.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-25 17:07:02 +03:00
Aleksei Sviridkin d2e851f63b feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.

The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.

publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.

Security model -- seven independent layers, all admission-time gates
fail-closed:

  1. Listener allowedRoutes namespace whitelist via the
     kube-apiserver-written kubernetes.io/metadata.name label;
     plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
     passthrough listeners restrict allowedRoutes.kinds to
     HTTPRoute / TLSRoute respectively.
  2. cozystack-gateway-hostname-policy VAP on Gateway.
  3. cozystack-gateway-attached-namespaces-policy VAP on Package --
     blocks tenant-* in gateway.attachedNamespaces.
  4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
     writes from non-trusted callers.
  5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
     writes to namespace.cozystack.io/host from non-trusted callers.
  6. Render-time fail in cozystack-basics for tenant-* in
     gateway.attachedNamespaces.
  7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
     scoped to tenant-* namespaces.

Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).

Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.

Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.

Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.

Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.

Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.

Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-25 17:07:02 +03:00
Andrei KvapilandClaude f919afe15f revert(ingress)!: remove broken publishing.exposure=loadBalancer mode
Reverts the ingress loadBalancer exposure mode added in #2468.

The mode rendered the ingress-nginx Service as type: LoadBalancer and a
CiliumLoadBalancerIPPool over publishing.externalIPs, as a migration path
off Service.spec.externalIPs (deprecated upstream in Kubernetes v1.36 per
KEP-5707). In default Cozystack the implementation produces two failure
modes:

- publishing.externalIPs is the operator-supplied list of node public IPs
  (addresses the cluster nodes already hold and listen on), not a
  virtual-VIP pool. Treating it as one and DNATing those IPs via Cilium
  LB-IPAM collides with the existing ingress-nginx DNAT on the same IPs
  (Service.spec.externalIPs path), so traffic is forwarded
  non-deterministically.
- Cilium in default Cozystack runs with ipam.mode: kubernetes and no
  L2/BGP announcer, so a pool over addresses outside the node range is
  allocated but never advertised — the Service stays unreachable.

The KEP-5707 deprecation is real, but the right migration is a
mechanism-agnostic tenant API knob translated by a Cozystack-side
controller into whichever backend the cluster admin has configured
(MetalLB pool, Cilium LB-IPAM, robotlb, externalIPs pin) — not a
chart-side IPPool baked into the tenant chart. That follow-up belongs in
its own PR.

Removed:
- publishing.exposure value (and the inline comment) from
  core/platform/values.yaml
- expose-mode plumbing line in core/platform/templates/apps.yaml
- CiliumLoadBalancerIPPool template at
  apps/tenant/templates/cilium-lb-pool.yaml
- type: LoadBalancer + externalTrafficPolicy: Local branch in
  extra/ingress/templates/nginx-ingress.yaml (Service renders back as
  ClusterIP + spec.externalIPs when externalIPs are set, falling through
  to type: LoadBalancer when they are not — pre-#2468 shape)
- Exposure mode section from extra/ingress/README.md
- exposure_test.yaml fixtures in both charts

Operators who explicitly set publishing.exposure=loadBalancer will see
their ingress Service revert to ClusterIP + spec.externalIPs on upgrade.
Anyone on the default value (externalIPs) is unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-05-11 10:42:29 +02:00
Aleksei Sviridkin 500816b71b refactor(ingress): move CiliumLoadBalancerIPPool to tenant chart
The pool is now rendered from packages/apps/tenant/templates/cilium-lb-pool.yaml instead of packages/extra/ingress/templates/cilium-lb-pool.yaml. Cilium LB IPAM forbids overlapping CIDRs across pools regardless of serviceSelector, so both the ingress-loadBalancer path and the upcoming per-tenant Gateway in #2470 cannot each own their own pool on the same publishing.externalIPs range. The tenant chart is the natural per-tenant owner — it already creates the Namespace, the cozystack-values Secret, and the HelmReleases for both ingress and gateway.

The new pool uses a namespace-only serviceSelector (io.kubernetes.service.namespace: <ns>), which matches any LoadBalancer Service in the tenant namespace. The metadata.name changed from <trim>-ingress to <trim>-exposure to reflect that the pool is not ingress-specific.

Only the ingress-loadBalancer signal is wired in this commit (_cluster.expose-mode=loadBalancer plus .Values.ingress=true on the publishing tenant). The gateway branch is added in #2470 on top of this commit — it rebases, drops its own packages/extra/gateway/templates/cilium-lb-pool.yaml, and adds an OR branch for .Values.gateway in the tenant template.

Pool-rendering unit tests moved from packages/extra/ingress/tests/ to packages/apps/tenant/tests/. The ingress chart tests keep the Service-level asserts. packages/apps/tenant/Makefile gains a test target so hack/helm-unit-tests.sh picks up the new suite.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-24 16:44:46 +03:00
myasnikovdaniil cbc79600dd [linstor-gui] Add Keycloak-protected Ingress for the UI (#2390)
## What this PR does

Stacked on top of #2382. Publishes the linstor-gui UI on
`https://linstor-gui.<root-host>` behind the cluster Keycloak realm
using the same oauth2-proxy gatekeeper pattern that the dashboard uses,
so the LINSTOR REST API is never exposed unauthenticated through
Ingress.

The Ingress is rendered only when **both** gates are satisfied:
- `_cluster.oidc-enabled == "true"`
- `linstor-gui` is listed in `_cluster.expose-services` (i.e. in
`publishing.exposedServices` in the core cozystack values)

If OIDC is disabled cluster-wide we intentionally do **not** ship an
unauthenticated ``token-proxy`` fallback — unlike dashboard, there is no
reason to front a raw storage-management API with k8s bearer tokens.
Operators can still reach the UI via \`kubectl port-forward\` to the
ClusterIP service.

## What's new

- \`templates/gatekeeper.yaml\` — oauth2-proxy Deployment (OIDC mode),
upstream \`linstor-gui.<ns>.svc:80\`
- \`templates/gatekeeper-svc.yaml\` — ClusterIP :8000 in front of it
- \`templates/gatekeeper-sa.yaml\` — dedicated SA, no auto-token
- \`templates/keycloakclient.yaml\` — persistent \`linstor-gui-client\`
+ \`linstor-gui-auth-config\` Secrets, \`KeycloakClient\` CRD that
auto-provisions the OIDC client with \`redirectUris:
[/oauth2/callback/*]\`
- \`templates/ingress.yaml\` — Ingress to the gatekeeper Service with
cert-manager ClusterIssuer, gated on expose-services + oidc-enabled
- \`tests/ingress_auth_test.yaml\` — unit tests for each conditional
branch, KeycloakClient rendering, oauth2-proxy args

README updated with Option 1 (Keycloak-protected Ingress) and Option 2
(port-forward).

### Release note

\`\`\`release-note
Added an opt-in Keycloak-protected Ingress for the linstor-gui package.
Add \`linstor-gui\` to \`publishing.exposedServices\` to publish
\`https://linstor-gui.<root-host>\` behind the cluster OIDC realm.
\`\`\`

## Test plan

- [ ] Deploy to dev10 with \`linstor-gui\` added to
\`publishing.exposedServices\`
- [ ] Verify cert-manager issues the TLS certificate
- [ ] Open \`https://linstor-gui.<root-host>\` in a browser, confirm
Keycloak login challenge
- [ ] After login, confirm LINSTOR node list loads via the nginx → mTLS
proxy
- [ ] Verify helm-unittest cases pass in CI
- [ ] Sanity-check that when \`oidc-enabled=false\` or the service is
not in \`expose-services\`, the Ingress + gatekeeper resources are
skipped
2026-04-15 17:52:29 +05:00
Myasnikov DaniilandClaude Opus 4.6 80968410b0 [tenant] Widen linstor-gui egress to whole cozy-linstor namespace
Narrowing to (instance=linstor-gui, name=gatekeeper) also blocked the
transient cm-acme-http-solver pods that cert-manager spins up in the
same namespace during http-01 challenges — the solver pods carry
different labels, so the Certificate never became Ready and TLS
handshakes failed.

Broaden the rule to the whole cozy-linstor namespace to mirror
allow-to-dashboard / allow-to-keycloak. The ingress controller only
initiates upstream HTTP calls driven by Ingress resources, so allowing
namespace-wide egress is not a practical escalation.

Verified on dev10 after patching the live CNP: Certificate
linstor-gui-ingress-tls becomes Ready (Let's Encrypt prod), and the
Ingress returns HTTP 302 to the Keycloak authorize endpoint with the
correct client_id and redirect_uri.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-14 12:34:15 +05:00
Myasnikov DaniilandClaude Opus 4.6 e2b6c2c3dd [tenant] Allow ingress to linstor-gui gatekeeper
The tenant chart declares per-namespace egress CNPs so the tenant-root
ingress-nginx can reach auth-fronted UIs in the system namespaces
(dashboard, keycloak, cdi-upload-proxy). linstor-gui's gatekeeper in
cozy-linstor needs the same allowance — without it, requests to
https://linstor-gui.<root-host> time out at the Ingress with 504
because tenant-root → cozy-linstor egress is denied by default-deny.

Scoped to the gatekeeper pods (instance=linstor-gui, name=gatekeeper)
rather than the whole cozy-linstor namespace — the ingress controller
does not need to reach the LINSTOR controller, satellites, or CSI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-14 12:34:15 +05:00
Aleksei Sviridkin fef0f10bfd docs(tenant): align chart README with tenant naming rules
The README stated that dashes are 'allowed but discouraged' in tenant names, but the tenant.name Helm helper explicitly fails on release names that contain more than one dash, and the platform guide on the website already says tenant names must be alphanumeric. Bring the chart README in line with both the website documentation and the enforcement now present on the aggregated API.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-12 14:16:41 +03:00
Myasnikov Daniil 9e55552910 [docs] Updated app go types
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-03-25 15:57:25 +05:00
Myasnikov Daniil e090e2f317 [docs] Added go types codegeneration for managed apps
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-03-25 15:57:13 +05:00
Timofei LarkinandClaude Opus 4.6 bdeb3a7e8c feat(lineage-webhook): import scheduling constants from cozystack-scheduler
Import SchedulingClassAnnotation, SchedulingClassLabel, SchedulerName,
and GVR components from the cozystack-scheduler/pkg/apis sub-module
instead of defining them locally. This ensures the webhook and scheduler
stay in sync on label/annotation keys.

Also standardize the namespace label from scheduling.cozystack.io/class
to scheduler.cozystack.io/scheduling-class for consistency with the
scheduler, and resolve scheduling class from the owner Application CR
(via a new SchedulingClass() stub method) before falling back to the
namespace label.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-03-18 14:08:29 +03:00