Commit Graph
4761 Commits
Author SHA1 Message Date
Michel Jansen 9b6128d3ca fix(kubernetes): reject invalid UTF-8 sysctl file values
Raw Helm file inputs can carry invalid UTF-8 that Go quoting turns into
different Unicode characters. Reject these bytes rather than silently
changing the requested worker configuration.

Assisted-by: LLM
Signed-off-by: Michel Jansen <mjansen@whero.xyz>
2026-11-12 21:35:53 +01:00
Michel Jansen 7d9f913350 feat(kubernetes): configure per-pool Talos sysctls
Worker sysctl requirements extend beyond user namespaces. A string map
keeps those settings in the Talos machine config without expanding the
application API for each kernel knob.

Tenant input crosses an unquoted shell heredoc in the reconcile Job,
so preserve scalar contents without allowing shell or YAML injection.

Assisted-by: LLM
Signed-off-by: Michel Jansen <mjansen@whero.xyz>
2026-11-12 21:35:53 +01:00
Aleksei Sviridkin 238b3ee491 fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but
only the `s3_lifecycle` job of its plugin worker applies it, and that
job is scheduled by the admin server. Cozystack deployed neither, so a
lifecycle rule returned 200 and no object ever expired, on the platform
backup bucket and on tenant buckets alike. That also made the documented
way to bound backup storage a no-op.

This PR adds the admin server and a worker to Simple and MultiZone
SeaweedFS instances, behind a new `lifecycle.enabled` value that is
**off by default**. A Client instance is unchanged, because its rules
are enforced where the instance it connects to runs. The worker
registers for `s3_lifecycle` only, so vacuum, balancing and erasure
coding are not started as a side effect. The admin keeps its job state
on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default
when that is empty; `volume.storageClass` is not reused, because in
MultiZone it can name a class local to one zone or node. The admin UI is
protected by a generated password, because a tenant namespace admits
traffic from the whole cluster. The dashboard Role lists the two new
WorkloadMonitors.

Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart
renders no `[grpc.s3]` section, so the s3 gateway served its internal
gRPC in plaintext while the worker dials it with mTLS, and every
lifecycle delete failed the TLS handshake. The section is backported
from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart
patch, applied with `--forward` so the next `make update` stops at the
point where the patch must be dropped instead of silently reversing it.
The s3 pods are restarted once so they load the section. This also
repairs IAM cache propagation between s3 replicas, which failed in the
same way, and it applies whether or not lifecycle is enabled.

The data-plane release now retries a failed install or upgrade in place
(`RetryOnFailure`) instead of rolling it back. With the previous default
remediation and unlimited retries, an upgrade that waited out its
timeout, for example on a tenant quota with no room for the admin
volume, rolled back and retried forever, restarting the s3 pods twice
per cycle. The worker's hostname anti-affinity is relaxed to a
preference so a rollout finishes on a single-node cluster.

#### Why lifecycle is off by default

The daily walk that SeaweedFS uses to expire objects sends its deletes
without an object identity, and the s3 server skips its compare-and-swap
when no identity is given; it does not re-check the object's age either.
A key overwritten between the walk's listing and the delete loses its
new content. This path is unchanged from 4.45 through 4.52 and on
upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`,
`identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup
buckets write unique keys and are barely exposed; a tenant bucket that
rewrites a fixed key is. On top of that, turning enforcement on would
start applying, on upgrade, every rule that was stored and ignored until
now. Both are retention changes an operator should choose, so nothing
changes for existing instances until `lifecycle.enabled` is set. Making
it the default again is left to a separate PR that carries object
identity on the walk path; upstream report:
https://github.com/seaweedfs/seaweedfs/issues/12451.

#### Turning lifecycle on and off

When `lifecycle.enabled` is set, rules that have been stored but ignored
start applying within minutes, and objects they have already expired are
deleted on that first pass, so review existing bucket lifecycle rules
first. Set `lifecycle.storageClass` before enabling, and prefer a
replicated class: the class is part of the admin StatefulSet's volume
claim template, which cannot change, so changing it while lifecycle is
on makes every upgrade of the instance fail until it is reverted. Moving
the volume means disabling lifecycle, deleting the
`admin-data-seaweedfs-admin-0` PVC, changing the class and enabling
lifecycle again. Each instance gains two small pods and a 1Gi PVC; when
the tenant quota has no room for them, the data-plane release keeps
retrying in place until it has.

When `lifecycle.enabled` is turned off again, the parent release removes
the admin credentials Secret while the data-plane release removes the
admin StatefulSet only on its own reconcile, shortly after. An admin pod
that restarts in that window fails to start on the missing Secret until
the StatefulSet is gone; nothing is lost, and the order is left as is.
The admin PVC stays until the instance is deleted, and a later re-enable
starts the admin with a new password.

Verified on a test cluster with the 4.45 chart: without the backport,
the worker found the expired objects and every delete failed with `tls:
first record does not look like a TLS handshake`; with this branch and
lifecycle enabled, a rule with a past expiration date removed the
matching objects and left the others, both on a manual run and on the
first scheduled pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up needed, not opened yet:
`operations/services/backup-classes.md` (v1.7 and next) advises an S3
lifecycle policy, which the bundled SeaweedFS applies only once
`lifecycle.enabled` is set; the SeaweedFS reference page gains
`lifecycle.enabled` and `lifecycle.storageClass` from the regenerated
README on the next release
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The terraform provider does not model the SeaweedFS application, so the
new values do not reach it.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added optional, daily S3 bucket lifecycle-rule processing for
supported SeaweedFS configurations. It is disabled by default; Client
instances rely on lifecycle rules configured on the connected SeaweedFS.
* Added configurable storage for lifecycle job state and dashboard
visibility for lifecycle components.
* Enabled the S3 gRPC server to use its internal TLS identity when
security is configured.

* **Bug Fixes**
* Improved storage-class handling so numeric-looking names remain
strings and unset classes are omitted where appropriate.
* Clarified backup cleanup guidance, including when lifecycle rules
apply and how to reclaim storage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:21:52 +03:00
Aleksei Sviridkin 3badbdef74 fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:15:26 +03:00
Aleksei Sviridkin aa97451bc2 fix(platform): leave tenant-root's ingress empty when the root tenant has none (#5226)
## What this PR does

Enabling SeaweedFS on the root tenant without ingress no longer hangs.
`cozystack-basics` wrote `_namespace.ingress: tenant-root` for the root
tenant whatever its `spec.ingress` said, so `seaweedfs-system` depended
on an `ingress-nginx-system` release that nothing creates, and the
platform backup bucket waited with it. Every other tenant already
derives the key in `packages/apps/tenant/templates/namespace.yaml` and
leaves it empty without ingress.

`cozystack-basics` cannot read the root Tenant, so the platform now
carries its ingress state the same way it carries monitoring: the new
platform value `ingress.rootEnabled` reaches the `cozystack-values`
Secret as `_cluster.ingress-enabled`, and `_namespace.ingress` is
written only when it is true. The default is `true`, which keeps the
value every existing cluster renders today, and a Secret written before
the key existed keeps it as well. An operator running the root tenant
without ingress sets `ingress.rootEnabled: false`, kept in sync with
`Tenant/root spec.ingress` by hand, as with `monitoring.rootEnabled`.

With the key empty, `seaweedfs-system` drops the dependency through the
check it already had (`eq $ingress .Release.Namespace`), and nothing
reads live cluster state for it.

An empty `_namespace.ingress` now reaches tenant-root and every tenant
inheriting from it, so the consumers that would break on it are adjusted
in the same PR. The API rejects `ingressClassName: ""` (checked with a
server-side dry run), and two consumers quote the value:

- `kubernetes`: the proxied ingress-nginx addon Ingress is not rendered
without an ingress; its Service still is.
- `monitoring`: Grafana's `spec.ingress` is left out, since
grafana-operator copies it into the Ingress verbatim.
- `seaweedfs`: the S3 Ingress is disabled instead of rendering with no
controller to serve it.

The `KamajiControlPlane` `network.ingress` block is unchanged on
purpose: its `hostname` is what the Kamaji provider uses as the control
plane endpoint, and Kamaji itself skips an empty class name. Harbor,
bucket, bootbox and alerta write the class unquoted, so an empty value
renders as an unset class, which the API accepts.

Tests:
`packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml`
pins the flag reaching `_cluster` both ways and with `ingress` set to
null;
`packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml`
pins `_namespace.ingress` for a missing, string, bool and `"false"`
flag, and that the other keys stay when ingress is off;
`packages/extra/seaweedfs/tests/ingress_dependency_test.yaml` pins the
dependency and the S3 Ingress for an own, empty and inherited ingress;
the kubernetes and monitoring suites pin the gated Ingress. Reverting
any of the gates, or flipping the missing-key default, fails the suites.

Not covered here:

- `_namespace.etcd` has the same hardcoded shape (#4783) and is left for
its own change.
- Toggling `Tenant/root spec.ingress` without changing
`ingress.rootEnabled` leaves the value stale, as
`monitoring.rootEnabled` does today.

### Screenshots

Not a UI change.

### Downstream repositories

The diff adds a key to `packages/core/platform/values.yaml`, which the
website documents by hand.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#800 documents `ingress.rootEnabled` in the
platform Package reference
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a root tenant without ingress no longer leaves seaweedfs-system waiting for an ingress-nginx-system release that is never created. Set the new platform value ingress.rootEnabled to false to match Tenant/root spec.ingress: false.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added a root-tenant ingress setting, enabled by default, that can be
turned off.
* **Bug Fixes**
* Kubernetes and Grafana ingress resources are now rendered only when
ingress is configured.
  * SeaweedFS S3 ingress is enabled only when ingress is configured.
* Ingress-dependent components now respect the root-tenant setting,
while other platform services remain available when root ingress is off.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 15:06:12 +03:00
Myasnikov Daniil 6a93765351 fix(seaweedfs): bump the COSI driver to v0.4.0 for the lost-grant fix
seaweedfs-cosi-driver v0.3.1 grants access by rewriting the legacy
/etc/iam/identity.json. Each S3 gateway migrates that file to
per-identity files and renames it away whenever it changes, so a grant
written while a gateway is migrating is lost: the BucketAccess reports
accessGranted and its Secret is written, but every gateway rejects the
key with InvalidAccessKeyId and never recovers. Users added to a Bucket
in one change lose a key regularly. v0.4.0 also writes the per-identity
file on grant.

The rest of v0.4.0 does not reach the platform. The
readwrite-governance-bypass access policy is opt-in and no
BucketAccessClass uses it. The refusal to delete a bucket that still
holds locked objects applies only to object-lock buckets, and their
classes are Retain, so the sidecar never calls DriverDeleteBucket for
them.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 07:53:34 +00:00
Alexey Artamonov 67fcadf689 fix(seaweedfs): disable the S3 Ingress in a tenant without ingress
The S3 Ingress kept rendering with no ingress-nginx to serve it. Now
that seaweedfs-system no longer waits on ingress-nginx-system there, the
release completes and the unserved Ingress would stay behind.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov 275c840a8c fix(monitoring): leave Grafana's ingress out in a tenant without ingress
grafana-operator creates the Ingress from spec.ingress verbatim, so an
empty _namespace.ingress reached the API as ingressClassName "" and was
rejected. With no ingress-nginx in the tenant there is nothing for that
Ingress to attach to.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov c61e6a842e fix(kubernetes): skip the proxied addon Ingress in a tenant without ingress
An empty _namespace.ingress means no ingress-nginx serves the tenant,
which tenant-root now reports when the root Tenant has ingress off. The
Ingress would carry ingressClassName "", which the API rejects, and the
release would fail to install where it used to leave an unserved
Ingress behind.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov b26001b6f9 test(seaweedfs): pin the ingress-nginx dependency to the namespace's ingress
seaweedfs-system waits on ingress-nginx-system only when _namespace.ingress
names its own namespace. With tenant-root's key now empty when the root
Tenant runs no ingress, an empty value has to drop the dependency, and a
value inherited from a parent must not add it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:04:36 +03:00
Alexey Artamonov 5b6708c7f9 fix(platform): leave tenant-root's ingress empty when the root has none
cozystack-basics wrote _namespace.ingress: tenant-root whatever the root
Tenant's spec.ingress said, so applications in tenant-root were told an
ingress-nginx runs there when nothing deploys it. SeaweedFS then waited
forever on ingress-nginx-system, and the platform backup bucket with it.
Every other tenant already derives the key and leaves it empty without
ingress.

cozystack-basics cannot read the root Tenant, so the platform carries
its ingress state the way it carries monitoring: ingress.rootEnabled
reaches the Secret as _cluster.ingress-enabled and gates the key. The
default keeps the value existing clusters render, and a Secret written
before the key existed keeps it too.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:04:35 +03:00
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Alexey Artamonov 8ba0380043 docs(clickhouse): name the clickhouse-backup handlers that reload the config
clickhouse-backup reloads its config file in each handler that reaches
S3, among them the create_remote, restore_remote and list calls the
strategy makes. The status and action-log endpoints the strategy polls
read in-memory state and do not reload it, so the mounted file reaches
the sidecar on the calls that need credentials rather than on every
request.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:11 +03:00
Alexey Artamonov f5114ae04c fix(seaweedfs): keep the metadata database storage class a string
db.storageClass went unquoted into the values of the database release
and again into its CNPG Cluster. A class name made only of digits,
which Kubernetes allows, became a number that the Cluster rejects, and
a line break in the value could add keys to the database release
values. It is quoted at both steps.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 10:30:11 +03:00
Alexey Artamonov e6eb3345c6 fix(seaweedfs): escape pool and zone storage classes in the release values
Pool and zone storage classes were wrapped in quotes by hand, which
does not escape a quote or a backslash in the value: a quote closes the
string early and the data-plane release values stop parsing, with a
YAML error that does not name the field. They are now quoted like the
volume and admin classes.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 10:29:48 +03:00
Alexey Artamonov f83db514b0 fix(seaweedfs): quote every storage class the upstream claims carry
The master, filer, volume index, log and all-in-one claims of the
upstream chart write storageClassName unquoted, like the data claims
did, so a class name made only of digits renders as a number and the
claim is rejected. None of them is set by Cozystack today, but they
are values of the same chart that a later change can start handing on,
and the patch already exists for the data claims. They follow the same
rule: quoted, and left out when no class is named so that the cluster
default applies. The deprecated filer.enablePVC claim is left alone:
it renders only with filer.data unset, and the template fails earlier on
an unset filer.data, so it cannot be reached.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 10:29:28 +03:00
Alexey Artamonov e7b5a5a1a9 fix(seaweedfs): keep storage class names strings in the volume claims
The upstream admin and volume StatefulSets write storageClassName
unquoted, so a class name made only of digits, which Kubernetes allows
and the parent chart now hands on as a string, turned back into a
number there and the claim was rejected. The field is quoted, and left
out when no class is named: quoting an empty value would set an empty
storageClassName, which disables dynamic provisioning instead of using
the cluster default. Upstream 4.52 and master still write both fields
unquoted, so this is a chart patch, applied with --forward so make
update stops once upstream carries an equivalent change.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 02:18:50 +03:00
Alexey Artamonov 7c9aa2fe10 fix(seaweedfs): hand storage class names to the data plane as strings
The volume and admin storage classes were written unquoted into the
values of the data-plane release. A class name made only of digits,
which Kubernetes allows, arrived there as a number, and a value with a
line break could add keys next to the field, past the schema that is
the tenant's input gate. Both are now quoted. lifecycle.storageClass,
new in this release, also carries the StorageClass name rules in its
schema, so a value that cannot name a class is refused when the
application is written instead of leaving the admin claim pending.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 23:43:06 +03:00
Aleksei Sviridkin ba0312b35f chore(kubernetes): align monitoring fix with current platform base
The previous base differs from the published package configuration,
so the image overlay retains release controllers instead of current
builds.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-10 11:45:36 +03:00
Aleksei Sviridkin b53ca1fede fix(postgres): refuse system databases and never drop them (#5155)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

`databases` accepts `postgres`, `template0` and `template1`, and the
init job treats them like application databases: it comments them
`database managed by helm`, creates `<db>_admin` and `<db>_readonly`,
makes `<db>_admin` the owner of the database and its schemas, and
installs the `trigger_auto_grant` event trigger. When the entry later
leaves values, the delete pass runs `DROP DATABASE` on it, which can
only fail:

- `template1`, `template0`: `cannot drop a template database`;
- `postgres`: `cannot drop the currently open database`, since the job
is connected to it.

The init job then fails on every upgrade, so the HelmRelease sits in
`UpgradeFailed` with `post-upgrade hooks failed`, and putting the entry
back does not help. `template1` also hands the takeover (owner of
`public`, default privileges, trigger) to every database created
afterwards. Once its roles are dropped by hand, the trigger left behind
names a role that no longer exists, so `CREATE SCHEMA` fails in every
new database, even as superuser.

This PR:

- **Refuses** the three names at render with a message that says what to
do. `cozyvalues-gen` has no way to constrain the keys of a
`map[string]Database`, so the refusal sits next to the existing
database-name guard, like the `users.postgres` refusal. The field
description says it too.
- **Never selects them for `DROP DATABASE`.** The delete pass excludes
them whatever values say.
- **Hands a taken-over system database back.** In a new `release system
databases` pass before the delete pass, for each of the three still
commented `database managed by helm`: drop the trigger and
`auto_grant_schema_privileges()`, `REASSIGN OWNED` / `DROP OWNED` for
its two roles in that database, give `public` back to
`pg_database_owner` (PostgreSQL 15+), and clear the comment. The
existing orphaned-roles pass then drops the roles; it never visits
`template1`, hence the per-database step. `template0` accepts no
connection and the job that commented it never got further, so only its
comment is cleared.

**Upgrade note.** A release that declares one of the three names fails
to render until the entry is removed. Removing it is now safe: the next
run releases the database instead of dropping it.

**Testing**

- `tests/system_databases_test.yaml`: 5 of its 6 cases fail on `main`
without the change. `helm unittest packages/apps/postgres`: 215 passed.
- I replayed the scenario against a `postgres:17` container, running the
rendered `init.sh` with `PGUSER=postgres`:
1. The `main` script with `databases: {appdb, template1, postgres}`
reproduces the takeover.
2. The `main` script with only `appdb` then fails with `cannot drop the
currently open database`.
3. This branch's script on that same state exits 0. It releases both
databases (owner `postgres`, no comment, no event trigger, `public`
owned by `pg_database_owner`) and drops `postgres_admin`,
`postgres_readonly`, `template1_admin` and `template1_readonly`. A
database created afterwards gets no trigger, and `CREATE SCHEMA`
succeeds in it.
  4. A second run is a no-op.
5. A `template0` commented and given roles by hand is released the same
way.

Open PR #5091 touches the same script (stable database order); the hunks
do not overlap.

Backport: the automatic cherry-pick onto `release-1.6` conflicts in
`init-script.yaml` and the generated cozyrd, so the backport to
`release-1.6` is done by hand in #5157 (same SQL, replayed against that
branch's script). `release-1.7` keeps `kind/backport`.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change (walked the
map: a template guard, the init script, a test suite and one field
description regenerated into README, schema, Go types and cozyrd; no
field, default, enum or name change)
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): databases now refuses the PostgreSQL system databases postgres, template0 and template1, and the init job never runs DROP DATABASE on them. A system database a previous version took over is handed back to the superuser on the next upgrade (event trigger removed, ownership and public schema restored, its <db>_admin and <db>_readonly roles dropped). A release that declares one of them must remove the entry before upgrading.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Behavior Changes**
* PostgreSQL system databases (`postgres`, `template0`, and `template1`)
are refused as managed application databases.
* Previously managed system databases are released during cleanup and
excluded from database deletion. Cleanup restores eligible ownership and
security settings and clears system database comments.
* **Documentation**
* Updated PostgreSQL configuration guidance to clarify that these system
database names are refused. Names that merely contain these strings
remain accepted.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 21:09:23 +03:00
ZverGuy f2944c572d fix(dashboard): answer not-installed when the observability proxy is absent
The proxy can be left out of an installation, and then the console
nginx answered the observability roots with its own 502. The console
read that as an unreachable store: three errors, a Retry button and a
live tail reconnecting forever, instead of a notice that the feature
is not available on this cluster.

nginx now turns the 502 and 504 it generates itself, when it cannot
resolve or reach the proxy, into a 404 Kubernetes Status, which the
console already treats as not installed. Upstream errors are not
intercepted, so the proxy's own 502 and 504 for a store still reach
the console unchanged. The bats test runs the extracted locations in a
real nginx when one is installed.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 2b6a1f4548 test(observability-proxy): drop the tenant egress policy from test notes
The chart tests named the tenant egress policy as one of the parties
the pod label and port are pinned for. That policy is gone, so the
notes now name only the Service and the console nginx, which still
depend on them.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy b703a54cb4 fix(tenant): drop the tenant egress rule to the observability proxy
The rule was added for a tenant Grafana reading through the proxy, but
the tenant Grafana datasources still point straight at vlselect and
vmselect, and the console reaches the proxy from cozy-dashboard. The
rule therefore served no caller and only let every pod of every tenant
reach the proxy directly, outside the console route, where each
request with a fresh token costs a TokenReview.

It can come back together with a tenant client that needs it.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 889712d244 fix(observability-proxy): grant the proxy only the API calls it makes
Nothing the proxy does reads ApplicationDefinitions, so the rule for
them is dropped. Namespaces are read through an informer, which only
lists and watches, so get is dropped as well.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
myasnikovdaniil 91f03dd90e fix(platform): remove the dead registries value and patch-containerd Secret (#5190)
Nothing reads the `patch-containerd` Secret: the kubeadm certs.d mount
it fed is gone, and the v1.6.0 changelog already says the passthrough is
removed. The `registries` value stayed in the Package spec and takes
mirrors that do nothing, which is worse than no knob for someone setting
up an air-gapped cluster. Removed both.

A Package that still sets `registries` keeps installing, the platform
chart has no values schema so the key is ignored. On upgrade the Secret
in `cozy-system` is deleted by the release.

Mirrors for tenant workers are the per-pool `talos.registryMirrors` of
KubernetesNodes.

I checked that nothing is left: `git grep` for `.Values.registries` and
for `patch-containerd` outside the changelogs finds nothing but the
kubernetes README sentence that says the Secret was removed. Full `make
unit-tests` passes.

Fixes https://github.com/cozystack/cozystack/issues/4006


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration Changes**
* The platform chart no longer supports the `registries` configuration
option or generates containerd registry secrets. Mirror endpoint
settings, including credentials and TLS verification options, are no
longer applied through this chart.
* Registry mirrors can be configured per worker pool using
`talos.registryMirrors` on `KubernetesNodes`. Existing packages that set
`registries` continue to install, but the setting is ignored.
* **Documentation**
  * Updated guidance for configuring registry mirrors on worker pools.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 19:36:30 +05:00
Mattia Eleuteri f003851338 fix(postgres): refuse system databases and never drop them
postgres, template0 and template1 were accepted in databases and handled
like an application database. The init job made <db>_admin their owner,
installed the auto-grant event trigger in them and, once the entry left
values, ran DROP DATABASE on them. That DROP can only fail (template
databases cannot be dropped, and the job is connected to postgres), so
every later upgrade failed in the init job and the release stayed in
UpgradeFailed even after the entry was put back. template1 also passed
the takeover on to every database created from it afterwards, where the
trigger, naming a role about to be dropped, makes CREATE SCHEMA fail.

A release a previous version let take one of them over has to be
repaired without dropping it, and without touching what this chart did
not create: a role or owner that merely carries an expected name may
belong to a tenant or an administrator, and a role comment can be set
by any CREATEROLE holder, so only the cozystack:managed mark is trusted.
What the marked roles own goes to cozystack:orphaned rather than to the
superuser, since a tenant's SECURITY DEFINER function would otherwise
run with superuser rights. A failed lookup of the databases cloned from
template1 has to stop the run before template1 stops being selected, or
those clones would never be repaired.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-09 11:18:27 +01:00
Aleksei Sviridkin da7f2c2f50 fix(tests): keep two unit tests green after the release image restamp (#5180)
## What this PR does

Same change as #5179, on `main`. Both tests pass here today, because
`main` still carries the pre-release image pins; they fail on every
release line as soon as the first rc is cut, since the `Prepare release`
commit rewrites `packages/*/values.yaml` and two unit tests read exactly
the fields it rewrites. On `release-1.7` this has kept `Unit &
controller tests` red on every PR since v1.7.0-rc.3
([job](https://github.com/cozystack/cozystack/actions/runs/41162009892/job/124421449910)
on #5172), and `release-1.8` would repeat it.

| test | reads | before the release build | after (release-1.7 today) |
|---|---|---|---|
| `hack/cni-plugins-staging-contract.bats` "the kube-ovn release the CNI
plugin pin was checked against is the one vendored" |
`.global.images.kubeovn.tag` in `packages/system/kubeovn/values.yaml` |
`v1.15.26@sha256:…` | `v1.7.0@sha256:…` |
| `packages/system/monitoring/tests/vtraces_collector_test.yaml` "pins
the collector image through the cozy-lib.image helper" |
`tracingCollector.image` |
`docker.io/otel/opentelemetry-collector-contrib:0.136.0@sha256:…` |
`ghcr.io/cozystack/cozystack/otelcol-tracing:v0.136.0-cozystack.0@sha256:…`
|

Changes:

- The bats test takes the vendored kube-ovn release from `appVersion` in
`packages/system/kubeovn/charts/kube-ovn/Chart.yaml`. That is the value
`packages/system/kubeovn/Makefile` builds the image from, and a release
build leaves it alone. The hint printed on a mismatch names the new
file.
- The helm unittest case no longer names the image at all. It sets
`_cluster.images-registry` and asserts that the registry lands in front
of the rendered image and that the image stays digest-pinned (the
`grafana_upstream_image_plugins_test.yaml` pattern). Removing the
`cozy-lib.image` call from the template makes it fail; the old regex did
not.

Same single commit as #5179; the only difference is the unrelated `DOC=`
line next to the edit, kept as `main` has it.

Verified locally on this branch: `bats
hack/cni-plugins-staging-contract.bats` 10/10, `helm unittest
packages/system/monitoring -f tests/vtraces_collector_test.yaml` 28/28.

Refs #5178 (the release-1.7 report), #5179, #5172.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Tests**
* Version validation now compares the checked version with the chart’s
application version and reports missing or mismatched versions with the
chart path.
* Monitoring image checks now verify use of the configured mirror
registry and require a 64-character lowercase SHA-256 digest, without
requiring a specific image name.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:19:21 +03:00
Aleksei Sviridkin 23461e065d fix(redis): run one Sentinel when Redis runs a single pod (#5151)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The chart pins `spec.sentinel.replicas: 3` whatever `replicas` says, so
an instance created with `replicas: 1` runs one Redis pod and three
Sentinel pods that have nothing to fail over to. This makes the Sentinel
count follow the data pods: one Sentinel when `replicas` is 1, three
otherwise.

The operator cannot run zero Sentinels (freshworks `Validate()` resets
`sentinel.replicas <= 0` to 3), and it needs at least one to locate the
master. With one Sentinel its quorum is `replicas/2+1` = 1, so the
single Sentinel works on its own.

The operator also gives the Sentinel Deployment a PodDisruptionBudget
with `minAvailable: 1` whenever Redis has two replicas or fewer
(`ensurePodDisruptionBudget` in
`operator/redisfailover/service/client.go`). On a single Sentinel that
budget refuses every eviction and would block node drains, so the chart
sets `sentinel.disablePodDisruptionBudget: true` in that case only. The
Sentinel `WorkloadMonitor` now expects the rendered count instead of a
fixed 3/2.

The operator only stops ensuring that PDB once the flag is set; it never
deletes the one it created while three Sentinels ran
(`EnsureSentinelDeployment` in freshworks v3.3.5). An existing instance
scaled to one replica would keep it, so
`templates/hooks/sentinel-pdb.yaml` adds a post-upgrade hook that
deletes `rfs-<release>`, with a Role scoped to that one name. It renders
on upgrade only, since a fresh one-replica install never gets that PDB.

From `replicas: 2` up nothing changes. Scaling an existing instance
between 1 and 2 replicas scales the Sentinel Deployment between 1 and 3
accordingly.

Not in scope: with `replicas: 1` the Redis data pod keeps its own
operator PDB (`minAvailable: 1`), which already blocks drains of its
node today; that is unchanged here.

Tests: `tests/sentinel_replicas_test.yaml` covers 1, 2, 3 and the
default, the PDB switch, and the `WorkloadMonitor`. Three of its cases
fail on `main` without the template change.
`tests/sentinel_pdb_hook_test.yaml` covers the hook. `helm unittest
packages/apps/redis`: 79 passed. Open PR #4990 also edits
`redisfailover.yaml` (Sentinel spreading); the two touch different keys.

Backport: `release-1.7` ships the same freshworks operator and CRD, so
`kind/backport` applies. `release-1.6` is not backportable as is: it
still runs spotahome `v1.3.0-rc1`, whose vendored CRD (chart 3.3.0) does
not declare `sentinel.disablePodDisruptionBudget`, and the operator's
CRDs are installed with Flux's default `crds: Create` policy, so they
are never updated on an existing cluster. The API server would prune the
field and the single Sentinel would keep a `minAvailable: 1` PDB that
blocks node drains. A 1.6 fix would need a CRD update path first.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change (walked the
map: the diff changes a template, a test, and the wording of two
descriptions; no field, default, enum or name changes)
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(redis): an instance with `replicas: 1` now runs one Sentinel instead of three, without a Sentinel PodDisruptionBudget; on upgrade, the PDB the operator created earlier is deleted. Instances with two replicas or more are unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Redis deployments configured with one replica now use one Sentinel;
other replica settings continue to use three.
* Workload monitoring now reflects the configured Sentinel count. When
changing to one replica, upgrades remove the Sentinel
PodDisruptionBudget.
* **Documentation**
* Updated Redis configuration guidance to explain how Sentinel count
varies with the replica setting and clarify Sentinel resource defaults.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:17:32 +03:00
Aleksei Sviridkin d61db784a6 fix(mariadb-operator): derive the cluster name from the platform domain (#5123)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Fixes #2348.

`packages/system/mariadb-operator/values.yaml` pins
`mariadb-operator.clusterName: cozy.local`, and the vendored chart turns
it into the operator's `CLUSTER_NAME` and into a SAN of the webhook
certificate. On a platform installed with any other
`networking.clusterDomain`, the operator resolves its own pods as
`<name>.<ns>.svc.cozy.local` and gets NXDOMAIN. Observed on a cluster
with `clusterDomain: lab.local`: a 2-replica MariaDB never gets
replication configured (`status.replication` stays `{}`, the secondary
runs as a standalone semi-sync master), and User, Database and Grant
fail with `lookup mariadb-<name>-primary.<ns>.svc.cozy.local ... no such
host`, while the operator only retries at verbosity 1.

`clusterName` is a value of the vendored subchart, which the `_cluster`
lookup used by package templates cannot reach. The platform now carries
`networking.clusterDomain` under the `mariadb-operator` subchart key in
the shared `cozystack-values` Secret, the same route
`keda.clusterDomain` and `opensearch-operator.manager.dnsBase` already
take. The literal in the package stays as the fallback for a render
without that Secret, so a cluster on `cozy.local` renders exactly as
before.

Tests cover both ends of the path:
`packages/core/platform/tests/apps_mariadb_operator_cluster_name_test.yaml`
pins that the Secret carries the domain under the anchored subchart key
and tracks a non-default value (it fails with the `apps.yaml` change
reverted), and
`packages/system/mariadb-operator/tests/cluster_name_test.yaml` pins
that a domain set on that key reaches the operator's `CLUSTER_NAME` and
the webhook certificate, with `cozy.local` kept as the standalone
default. `make helm-unit-tests` passes locally.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(mariadb-operator): the MariaDB operator now uses the platform's networking.clusterDomain instead of a hard-coded cozy.local, so replication and User/Database/Grant reconciliation work on clusters with a non-default cluster domain.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* MariaDB now uses the configured networking domain as its cluster name
and webhook certificate address. If no custom domain is configured, it
continues to use the default cluster name and address.
* **Tests**
* Added coverage for default and custom domain settings, including their
effect on the MariaDB cluster name and webhook certificate address.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:07:37 +03:00
Aleksei Sviridkin 0c486263f9 fix(postgres): render the declared databases in a stable order (#5091)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The postgres init script lists the declared databases with `keys
.Values.databases`, which returns them in Go's randomized map order. A
release with two or more databases therefore renders a different
`DEFINED_DBS` line on each upgrade, so the `<release>-init-script`
Secret and the `checksum/config` annotation of the init Job change on
upgrades that did not touch the databases. Sorting the keys makes the
render a function of the values alone.

Found while measuring a platform-wide values change on a production
cluster (#5092): of 36 releases that ran Jobs again, 31 rendered a
byte-identical manifest across the two Helm revisions and the other 5
were Postgres releases whose only difference was this ordering, e.g.
`DEFINED_DBS="db_a db_b"` → `DEFINED_DBS="db_b db_a"`.

This does not stop the init Job from running on every upgrade (it is a
`post-upgrade` hook); the issue covers that part.

Verification: the new `tests/init_script_deterministic_test.yaml`
renders eight databases declared out of order and expects them sorted.
Without the fix it failed 5 runs out of 5 with a different order each
time; with it, `helm unittest packages/apps/postgres` passes (27 suites,
210 tests).


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): the init script renders the declared databases in a stable order, so an upgrade that leaves the databases unchanged no longer rewrites the init-script Secret.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Database initialization now lists configured databases in alphabetical
order, providing consistent output across deployments.
* **Tests**
  * Added coverage to verify the alphabetical ordering.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:06:40 +03:00
Aleksei Sviridkin 9e8e10c307 fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does

Refs #5205.

The S3 endpoint the platform backups use is copied twice, and neither
copy is refreshed after `publishing.host` changes. The COSI sidecar
writes `BucketInfo` once per `BucketAccess`, so the credentials Secret
`bucket-cozy-backups-system-credentials` rendered from it keeps the old
endpoint. The Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of `backupstrategy-controller` saw. The default
objects gate only acted on missing objects, so backups kept going to the
old host and nothing reported it.

The gate now also forces a render when a copy is stale:

- the bucket's `<bucket>-system` release, when the credentials Secret's
endpoint differs from the COSI Secret it is rendered from;
- `backupstrategy-controller`, when the endpoint an object was rendered
with differs from the credentials Secret's. The chart stamps the
rendered endpoint on the BSL and on every default Strategy as
`backups.cozystack.io/endpoint`, because the endpoint itself sits at a
different path in every kind. Objects without the annotation count as
stale until a render stamps it; `MinForceInterval` bounds the forces.

Stale credentials are forced before the BackupClass and the routed
objects are read, so an absent BackupClass or an unreadable Strategy
still fails the check but does not keep the old endpoint in the
credentials Secret. A `BucketInfo` without an endpoint is not compared,
since rendering from it would write an empty endpoint or fail the bucket
release. Repairing missing objects keeps working while the credentials
comparison is stale or fails, and a `systemSecretName` that is not
derived from a COSI Secret (no `-credentials` suffix, or the raw COSI
Secret itself) is not compared at all. RBAC is unchanged: the controller
already holds cluster-wide `get` on Secrets. No new metric: nothing
would consume it.

What this does not do: the COSI Secret still holds the endpoint the
`BucketAccess` was granted with, so after a host change the endpoint
follows only once an operator deletes the `BucketAccess` and forces
`bucket-cozy-backups` to recreate it (this issues new S3 keys). From
there the gate replaces the remaining manual steps of the recovery
described in the issue. Automatic re-grant is left for a separate issue,
and the operator runbook goes to the website, hence `Refs` rather than
`Fixes`.

Known limitation: the credentials comparison runs only when
`backupStorage.systemSecretName` is
`bucket-<bucketName>-system-credentials`, the Secret the forced
`bucket-<bucketName>-system` release renders. If an operator sets
`bucketName` and `systemSecretName` inconsistently, the comparison is
switched off rather than forcing a release that never rewrites the
compared Secret, so a stale endpoint in that Secret is not repaired.
Repairing a missing credentials Secret still forces the release derived
from `bucketName`, as before this PR.

Tested with unit tests for the gate (including the comparison
normalisation shared with the chart helpers), helm-unittest for the
annotation and the wiring, and a bats guard that every rendered Strategy
CR and the BSL carry the annotation. On a live cluster, a server-side
render of the chart produced exactly the annotation the gate derives
from the real credentials Secret, so the steady state does not force
anything.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: an operator runbook for changing `publishing.host` with
platform backups enabled (re-grant the `BucketAccess`, then let the gate
converge; restart ClickHouse pods that read the credentials from
environment variables). Not filed yet, pending approval of the text.
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup strategies and storage locations now record their storage
endpoint, enabling detection and repair of outdated endpoint settings
after bucket access changes.
* Bucket credentials and backup objects are refreshed independently when
their endpoints are missing or out of date.
* **Documentation**
* Clarified when endpoint changes trigger updates and how suspended
releases are handled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:02:42 +03:00
ZverGuy 74a7a44423 feat(dashboard): route the observability proxy roots from the console
The logs and metrics roots under /apis/cozystack.io and
/apis/apps.cozystack.io are served by the observability proxy, not by
the API server, so the console nginx sends them there ahead of its
/apis prefix location, unbuffered and with the same long timeouts, so
that a live tail reaches the browser as it is written.

nginx resolves a literal upstream once at startup and refuses to start
when the name does not resolve, which would take the console down on
every cluster that does not run the proxy. The upstream is therefore a
variable resolved per request, and such a cluster answers 502 on those
paths only. That resolver does not apply the pod's search domains, so
the config becomes a template the image's entrypoint renders with the
pod's nameservers and the platform cluster domain, which the chart now
passes to the console.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy f7719cf6e2 feat(tenant): allow egress to the observability proxy
A tenant's Grafana reads the tenant's logs and metrics through the
observability proxy, and tenant egress is default-deny with one
allow-to-* policy per platform service. The rule is narrowed to the
proxy pods and their port, since the proxy's namespace holds nothing
else a tenant should reach.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy bbdf05c6c7 feat(platform): deploy the observability-proxy
The proxy only answers for stores that the monitoring stack runs, so it
follows monitoring-agents: on by default in the full variants and
opt-in through enabledPackages on the slim ones, where monitoring is
opt-in too. It is a stateless reader with no CRDs, so networking is its
only dependency.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy 7ec648d863 feat(observability-proxy): add the chart and image build
Tenants read their logs, and later their metrics, through an
authorizing proxy that sits in front of the VictoriaLogs and
VictoriaMetrics read paths, so the proxy needs a deployable package
before the console or tenant Grafana can reach it.

The proxy reviews every token and access decision against the API
server, so its own role is limited to creating TokenReviews and
SubjectAccessReviews and reading ApplicationDefinitions and namespaces.
The namespace logs root is authorized as list on cozystack.io logs,
which no API server serves; the rule is aggregated into the tenant view
role so that a namespace viewer passes that review, as upstream grants
pods/log at view.

The image ref lands without a digest until the first build stamps it,
and the package joins the root build list so every build path refreshes
it.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
Mattia Eleuteri 9fba8af69c test(redis): pin the release name in the Sentinel PDB hook for the longest release
The long-name case checked only that the Job name fits in 63 characters and that the Job targets the right PDB. A Job name that dropped the release, letting two releases' hooks replace each other, or a Role granting a different PDB would both have passed.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-08 19:45:18 +01:00
Marian Koreniuk e92bc8664b fix(tests): keep two unit tests green after the release image restamp
The release build rewrites packages/*/values.yaml: the kube-ovn image tag
becomes the Cozystack version and the traces collector becomes the
first-party otelcol-tracing build. Two unit tests read exactly those
fields, so every PR against release-1.7 has failed Unit & controller
tests since the first rc, and release-1.8 would repeat it.

Same change as #5179 on release-1.7.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-11-08 19:37:03 +01:00
Mattia Eleuteri a7f84242d1 fix(redis): keep three Sentinels for a zero or negative replica count
The operator runs three Redis pods when the count is zero or negative, but the chart treated any count up to 1 as the single-pod case. Such a cluster got one Sentinel and no Sentinel PDB, while three Redis pods ran under it.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-08 19:36:44 +01:00
Mattia Eleuteri 55d6c52fab fix(kubernetes, monitoring): send tenant cluster metrics to the tenant's vminsert
The monitoring-agents chart reads vmagent.remoteWrite.urls as a list and
takes the tenant label from global.target, while the kubernetes app still
passed a single remoteWrite.url and vmagent.externalLabels.tenant. Both
keys were dropped in silence, so the vmagent of every tenant cluster fell
back to the chart defaults: it wrote to vminsert services in
cozy-monitoring that do not exist inside the tenant cluster, queued the
failed writes on the node disk, and labelled every series
tenant="cozy-monitoring". No tenant cluster metric reached the tenant's
Monitoring.

The tenant label has to name the tenant that owns the cluster, which is
not always the tenant whose Monitoring receives the data: a tenant
without its own Monitoring inherits a parent's.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-08 19:33:54 +01:00
Alexey Artamonov c6c107fd08 docs(backups): state when the gate compares endpoints in the values reference
The values reference read as if the credentials endpoint were always
compared and an object without the endpoint annotation never forced a
render. Operators sizing the force counter or choosing a custom
systemSecretName need both conditions spelled out.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:18:29 +03:00
Alexey Artamonov dc909c778c fix(backups): compare the credentials endpoint only for the forced bucket's Secret
The gate forces bucket-<bucketName>-system, which renders the
credentials Secret of that bucket's system user and nothing else. The
COSI Secret name was derived from systemSecretName instead, so a value
naming another bucket's Secret made a lagging endpoint there force the
platform bucket release every throttle window without ever refreshing
the Secret that was compared.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:18:29 +03:00
Alexey Artamonov d0f47f7dd5 fix(backups): compare the credentials endpoint only for a Secret derived from COSI
backupStorage.systemSecretName may name a Secret the bucket chart does
not render from a COSI Secret: the raw COSI Secret itself, which the
projector accepts, or any Secret without the -credentials suffix. The
COSI Secret name was then the source name unchanged, so the gate
compared a Secret with itself and either reported it stale on every
tick or failed the check, and stopped repairing missing default objects
as well. Skip the comparison when there is no distinct COSI Secret or
no endpoint key to compare.

The gate compares the annotation with what the chart's endpoint
helper renders, and the helper renders a present but blank endpoint as
a bare https:// instead of falling back, so the gate has to compute
the same value or its expectation and the chart disagree for that
Secret. The gate's docs, metric help and log lines named only missing
objects, so they did not cover a stale endpoint they now report.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:18:29 +03:00
Kirill Ilin cb3fcd3357 feat(operator): let the platform source trust a private registry CA (#5237)
## What this PR does

The operator creates the `cozystack-platform` OCIRepository from its own
command-line arguments and rewrites the whole object on every start;
`cozystack-packages` copies that spec. When the platform source is a
registry whose certificate is issued by a private CA (an internal
pull-through cache in an air-gapped install, for example), there is no
durable way to make Flux trust it: a `certSecretRef` patched onto the
live object is dropped on the next operator restart, source-controller
then fails TLS, and platform updates silently stop arriving.

This adds `--platform-source-cert-secret` to the operator and
`cozystackOperator.platformSourceCertSecret` to the installer chart. The
named Secret in `cozy-system` is set as `spec.certSecretRef` on the
generated OCIRepository, so it holds `ca.crt` for a private CA and/or
`tls.crt`+`tls.key` for client certificate auth, with the Flux key
layout. `cozystack-packages` inherits it through the existing spec copy.

The flag is OCI only. `GitRepository` has no `certSecretRef` (Flux reads
a Git CA from the `ca.crt` key of the `secretRef` Secret), so a Git
platform source with the flag set is refused at startup instead of
having its CA silently ignored.

Tests: unit tests for the generated spec, a regression test that the CA
survives the operator's update of an existing OCIRepository, a test that
a Git source with the flag writes nothing, and a helm-unittest suite for
the new chart value.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/791
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The new installer value is optional, so `ansible-cozystack` is not
reached. The website's air-gapped guide
(`content/en/docs/next/install/kubernetes/air-gapped.md`) added the
private CA to the nodes only, while Flux fetches the platform source
itself; cozystack/website#791 adds a step for `platformSourceUrl`,
`platformSourceSecret` and `platformSourceCertSecret`.

### Release note

```release-note
feat(operator): the platform source can be served from a registry behind a private CA. Set `cozystackOperator.platformSourceCertSecret` in the installer to a Secret in `cozy-system` with `ca.crt` (and optionally `tls.crt`/`tls.key`); the operator sets it as `certSecretRef` on the `cozystack-platform` OCIRepository, so it survives operator restarts.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added the `platformSourceCertSecret` setting for OCI platform sources.
It allows a Secret containing certificates for private CA trust or
client authentication to be configured separately from registry
credentials.
* The configured certificate Secret is applied to the OCI source and
inherited by its package source.
* **Configuration**
* The setting is empty by default. Git platform sources do not support a
separate certificate Secret; use the existing platform source Secret for
Git TLS certificates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-06 22:21:19 +05:00
Alexey Artamonov 86c541368d Merge branch 'main' into fix/clickhouse-backup-late-credentials
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Alexey Artamonov 9376ba0078 fix(backups): hand every driver the same forcePathStyle
backupStorage.forcePathStyle reaches drivers that read it differently.
The controller, Velero and the ClickHouse sidecar parse it with
strconv.ParseBool, the etcd and MongoDB strategies write it into a
boolean field that Helm decodes as YAML 1.1, and the Kafka and
RabbitMQ scripts take anything but the string "false" as path style.
So "False" meant path style to Kafka and RabbitMQ only; "yes" meant
path style to etcd, MongoDB, Kafka and RabbitMQ and was refused by
Velero and the ClickHouse sidecar; "T" and "1" could not be applied to
the boolean fields.

The chart now accepts the spellings of strconv.ParseBool and of YAML
1.1 booleans, a superset of the boolean spellings the etcd and MongoDB
strategies accepted, and passes every consumer "true" or "false", so
they agree. Unset stays unset, so each driver keeps its own default.
Any other value fails the render with a message naming it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Myasnikov Daniil 41507e36aa docs(kubernetes): say where worker registry mirrors are configured
The air-gapped note said in-guest image pulls honour no per-tenant
mirror until a Phase 2 that has landed, so an operator setting up an
air-gapped cluster would read that mirroring is unavailable and file
an issue. It now says the Secret and the platform registries value are
removed, that a Package still setting the value keeps installing, and
that worker mirrors are set per pool through talos.registryMirrors of
the KubernetesNodes resource.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-06 18:56:09 +05:00
Kirill Ilin 0a448963e3 docs(installer): say a CA-only cert secret must be Opaque
kubernetes.io/tls requires both tls.crt and tls.key, so a Secret that
carries only ca.crt is rejected under that type.

Assisted-by: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-11-06 18:29:18 +05:00
Kirill Ilin fbfc270c71 test(installer): tie each platform source flag to its own value
Both secrets were only ever set together, so a cert flag keyed on
platformSourceSecret stayed green. A pull-through cache behind a private
CA usually needs the CA without credentials, which is exactly the case
such a slip breaks. The absence checks match the flag by prefix, since
a whole-element comparison misses the flag rendered with any value.

Assisted-by: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-11-06 18:29:18 +05:00
Alexey Artamonov 5cbb3a633e fix(seaweedfs): give the lifecycle admin volume its own storage class
volume.storageClass is picked for volume server data, and in a
MultiZone instance it can name a class whose volumes exist in one zone
or on one node. Reusing it for the admin volume would tie the admin to
that zone or node, so losing it would stop lifecycle enforcement for
the whole instance, a placement nobody chose for the admin. The admin
volume now takes lifecycle.storageClass, and the cluster default when
that is empty.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 16:04:24 +03:00
Alexey Artamonov 62b4213784 fix(seaweedfs): keep bucket lifecycle enforcement off unless enabled
The daily walk that expires objects sends its deletes without an object
identity, and the s3 server skips its compare-and-swap when none is
given (walker_dispatcher.go and identityMatches, unchanged from 4.45
through 4.52). It does not re-check the object's age either, so a key
overwritten between the walk's listing and the delete loses its new
content. Backup buckets write unique keys and barely meet that window;
a tenant bucket that rewrites a fixed key does.

Turning enforcement on by default would also start applying, on
upgrade, every rule that was stored and ignored until now. Both are
retention changes an operator should choose, so the admin server and
the worker now run only where lifecycle.enabled is set. On by default
can return once the walk carries identity.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 15:56:51 +03:00