The table test builds its Running conditions by hand, so it can only
confirm the shape it already assumes. A DataVolume read off a cluster
where the importer had failed on a 404 for days carries a Bound
condition with the same error, which the message must not pick up, and
proves the error sits where the controller looks for it.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A disk whose HTTP source answers 404 or does not resolve never leaves
ImportInProgress: CDI fails an import only on a fatal error and restarts
the importer pod otherwise, so the condition only ever said the disk was
still importing. The reason it will not finish was visible only in the
importer logs and on the DataVolume, whose Running condition carries the
error through the back-off between restarts.
Appending that message lets the user see why the disk is stuck from the
application status and the console, which already shows the
WorkloadsReady message for a disk.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
controller-runtime 0.25 made FakeInformer's sync state a private
channel, so the DataVolume watch tests no longer compile: they toggled
the exported Synced field and its lock. FakeInformers also builds every
informer it creates already synced, so an unsynced informer has to be
seeded into InformersByGVK and marked synced with Synced() later.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A reconcile that runs before the DataVolume watch starts finds no
reader, keeps the monitor's stored verdict and is not requeued. The
watch's initial replay then delivers only the DataVolumes that still
exist, so a monitor whose DataVolume was deleted while the controller
was down kept reporting it not ready until the periodic resync, hours
later.
Installing the reader now queues every monitor that holds a DataVolume
verdict, through the controller's own queue, so each is recomputed
against the synced cache. Monitors without a verdict are left alone:
the replay already reaches every DataVolume they could select. If the
monitors cannot be listed, the next discovery poll retries.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
CDI copies the DataVolume labels onto the PVC it creates, so the vm-disk
monitor also publishes a Workload for that PVC, and its readiness was
bind state alone. Mid-import the PVC is already Bound: the Workload read
operational while the monitor, and the Application's WorkloadsReady,
said the disk was not populated.
A PVC whose controller is a DataVolume the monitor finds not ready now
reads not operational. While the DataVolumes cannot be read, its
Workload keeps the verdict it has, as the monitor does.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
While the DataVolume watch syncs, every WorkloadMonitor of every kind
is requeued every few seconds, and an informer that cannot sync keeps
that window open. Each pass wrote status unconditionally, so a cluster
with many monitors sent the API server a steady stream of writes that
changed nothing. Comparing against the stored status first leaves the
requeue as cache reads only.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A failed DataVolume List keeps the monitor's last verdict and retries,
and the only trace of it was a controller log line. A monitor with no
verdict yet then reads operational while its disks are unreadable, with
nothing a tenant can see saying why. A Warning event on the monitor
makes the failure visible where the bucket metrics failure already is.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A controller-runtime Kind source adds an event handler to the informer
when it starts and never removes it; cancelling it on a sync timeout
only ends its context. Building a new source on every failed sync
attempt therefore left one more handler on the DataVolume informer per
retry, each enqueueing the same events, until the controller restarted.
The source is now started once and later attempts only wait for the
informer to sync. Because the started source keeps delivering events
between attempts, the syncing flag stays set until the reader is
installed, so a reconcile queued in that gap is requeued as well.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Starting the DataVolume source replays every DataVolume to its
handler at once, and the handler queues the matching monitors, while
the reader is installed only after the source has synced. A monitor
reconciled in that window saw no reader, kept its last verdict and
returned with nothing left to queue it again, so a disk that failed
or finished while the controller was down could keep a wrong
WorkloadsReady until the next unrelated event.
Such a reconcile still publishes the replica counts and now asks to
be requeued after a few seconds, until the reader is in place or the
sync attempt gives up. It is a requeue rather than an error because
the window is expected: an error would count as a reconcile failure
and log a line for every monitor in the cluster while it lasts.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
WorkloadsReady=False covers different situations: a stopped VM has
no pod and its monitor reports it not operational on purpose, while a
disk whose DataVolume never populates is broken. A consumer that wants
to react to one and not the other had only the message text to go by.
The WorkloadMonitor status gains a Reason next to Message, set to
DataVolumeNotReady by the DataVolume check, and the Application API
carries it into the WorkloadsReady condition in place of the generic
reason whenever a monitor that is not operational names one.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The DataVolume List sits on the reconcile path of every
WorkloadMonitor, and returning its error before the status write froze
the replica counts and Operational of a monitor that selects no
DataVolume at all whenever CDI's API hiccupped.
The status is now written either way, with the last DataVolume verdict
kept whenever the DataVolumes could not be read, and the error still
returned so the request is retried. The same applies while no reader
is installed yet, so a stuck disk does not read as ready for the
window between a controller restart and the DataVolume informer
syncing.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
For a vm-disk, WorkloadsReady is the one condition that reports
whether CDI populated the disk, and its generic "One or more workloads
are not operational" gives an operator nothing to act on: a disk has
no workloads, and the stuck DataVolume and its phase are exactly what
they need to know.
The WorkloadMonitor status gains a message that the controller fills
with each selected DataVolume that is not ready and its phase, and the
Application API uses the messages of the monitors that are not
operational in WorkloadsReady instead of the generic text whenever one
of them names a cause.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A vm-disk application reported its workloads operational whatever
state its DataVolume was in, so an import or clone that never
finished left the disk looking healthy while it was unusable. Its
monitor has minReplicas 0 and only pods fed Operational.
The monitor now also lists the DataVolumes its selector matches and
reports not operational while any of them is in flight, Failed,
Unknown or without a phase. UploadReady counts as in flight: CDI
holds it from the moment the upload server is ready until the
transfer succeeds, so it covers an empty disk and a half-uploaded
one alike. Waiting for the first consumer counts as ready, since
that is where such a disk rests, and so does a phase this
controller does not know, so a CDI release that renames one cannot
turn working disks red.
This is deliberately not a Helm wait: a readiness gate holds a
helm-controller worker for its whole window, and a disk that never
populates would keep taking it back. The signal lands in the
application's WorkloadsReady condition instead, and Ready keeps
following the HelmRelease.
CDI ships with the IaaS bundle only and can be installed while this
controller is running. A watch on a kind whose CRD is absent keeps
the controller from starting, so the DataVolume watch is added from a
runnable that polls discovery until the kind is served. It starts on
the first reconcile, the earliest point at which a new watch is
started at once, rather than on the next tick, so a disk created
just after a restart is not read as populated for a poll interval.
The reader is installed only once that watch has synced within a
bound: a reconcile context has no deadline, and a List on an
informer that never syncs would park every WorkloadMonitor reconcile
for good.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Two small bug fixes, one per commit. Both were first proposed by
@officialasishkumar in #2334 and #2336. Those PRs went stale, and part
of each was fixed on main in the meantime, so this carries only what is
still broken.
ClickHouse Keeper hosts rendered without a cluster domain, as
`chk-<name>-keeper-cluster1-0-0.<namespace>.svc.` with nothing after
`svc.`. The zookeeper block in `clickhouse.yaml` redeclared
`$clusterDomain` from `.Values.clusterDomain`, which the chart never
defines, and that shadowed the value read from `_cluster.cluster-domain`
at the top of the template. Keeper is enabled by default, so every
ClickHouse release got these hosts. The keeper name test set
`clusterDomain` itself and hid the bug. It now sets only
`_cluster.cluster-domain`, like the platform does, and fails without the
fix.
The WorkloadMonitor controller panicked on a PVC with no
`storageClassName`, which is any PVC that takes the cluster default
class. The check used `||` where it needed `&&`. controller-runtime
recovers the panic, but the reconcile of that monitor fails on every
retry and never reaches its status update. A PVC with an empty
`storageClassName` also got a resource key with an empty class name
instead of `default`. A new table test covers unset, empty and explicit
classes.
The duplicate `template` key in the ClickHouse backup CronJob from #2334
and the `Operational` status fix from #2336 are already on main, so they
are not in this PR.
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(clickhouse, workloadmonitor): ClickHouse Keeper node hostnames now include the cluster domain, and the WorkloadMonitor controller no longer panics on a PVC without a storage class.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Workload resource labels now use the default storage class when a
PVC’s storage class is unset or empty, while continuing to reflect
explicitly configured storage classes.
* ClickHouse ZooKeeper hostnames now consistently use the configured
cluster domain, or `cozy.local` when no domain is set. This helps
hostnames align with the cluster’s domain configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
maps.Copy(workload.Labels, bc.Labels) can bring in a workloads.cozystack.io/bucket-class label directly set on the BucketClaim, which would survive unmodified when spec.bucketClassName is empty or invalid, leaving the Workload reporting a class that does not match the source BucketClaim.
Signed-off-by: harshitrwt <harshit.rawat1804@gmail.com>
A BucketClassName longer than 63 chars (valid as a Kubernetes object name, up to 253 chars) is not a valid label value, so the bucket-class label is skipped with an Info log rather than failing the Workload write. This can happen with SeaweedFS pool class names (<namespace>-<pool>-lock) when the tenant and pool names are both long -- a missing label in that case does not mean no class was set.
Signed-off-by: harshitrwt <harshit.rawat1804@gmail.com>
The storage class check used || where it needed &&, so a PVC with no
storageClassName (one that takes the cluster default) was dereferenced
as a nil pointer. controller-runtime recovers the panic, but the
reconcile of that WorkloadMonitor fails on every retry and never
reaches its status update. A PVC with an empty storageClassName
produced a resource key with an empty class name instead of falling
back to "default".
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
getWorkloadMetadata filled the workloads.cozystack.io/kubevirt-vmi-instance-profile
label from the annotation kubevirt.io/cluster-instanceprofile-name, a key KubeVirt
never stamps. KubeVirt records the cluster preference (which carries the guest OS
profile) under kubevirt.io/cluster-preference-name, so the profile label was never
emitted and a VM's instance profile never reached the billing metric. The sibling
instance-type label worked only because its key is the real one.
Read the correct annotation. The emitted label name is unchanged, so the downstream
metric pipeline is untouched.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The controller discovers where to query SeaweedFS bucket size metrics
via the namespace.cozystack.io/monitoring label, which assumes the
monitoring stack that scrapes SeaweedFS runs in the same cluster. Some
deployments host SeaweedFS and its metrics in a separate cluster, so
add a --seaweedfs-metrics-endpoint flag to cozystack-controller (and a
matching cozystackController.seaweedfsMetricsEndpoint chart value): an
explicit base URL of a Prometheus-compatible query API, with scheme,
host, optional port and path prefix; /api/v1/query is appended. Basic
auth may be embedded as URL userinfo. The flag is validated and
normalized at startup, so a malformed URL fails the rollout rather
than the queries. Empty keeps label-based discovery unchanged.
Bucket sizes feed billing, so metrics failures no longer collapse to
"no size": previously any query failure yielded an empty result map
and the reconciler overwrote Workload resources with it, silently
wiping the recorded sizes. The reconciler now retains the last known
s3-storage-bytes and s3-physical-storage-bytes values, emits a
BucketMetricsUnavailable warning event on the WorkloadMonitor, and
returns the error from Reconcile so it lands in the reconcile error
metrics.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pass bucket names from BucketClaim.Status.BucketName into the PromQL
query as a bucket=~"name1|name2" filter. This prevents O(N²) load
where N WorkloadMonitors each fetch all buckets globally.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Replace 2×N per-bucket HTTP requests with a single query that fetches
all SeaweedFS bucket size metrics at once:
{__name__=~"SeaweedFS_s3_bucket_(size|physical_size)_bytes"}
Results are keyed by bucket name in memory, then looked up per
BucketClaim. This reduces HTTP round-trips from 2N+1 to 2 (one
namespace lookup + one Prometheus query) regardless of bucket count.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Change queryPrometheusMetric to return (int64, bool) so callers can
emit s3-storage-bytes=0 for empty buckets while omitting the field
entirely when monitoring is not configured.
Also:
- Add RBAC marker for core/namespaces GET (used by resolvePrometheusURL)
- Log namespace read errors instead of silently returning empty URL
- Add TestQueryPrometheusMetricZeroValue test
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Bucket existence is already tracked via LifetimeHours. The s3-buckets
count resource produced a meaningless "s3-buckets-Hours" fallback type
in billing. Only storage size metrics (s3-storage-bytes,
s3-physical-storage-bytes) are now set on bucket Workloads.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Replace static --prometheus-url flag with dynamic resolution from
namespace.cozystack.io/monitoring label. Each tenant namespace knows
which tenant hosts its monitoring stack, so the controller constructs
the vmselect URL automatically. This correctly handles multi-tenant
setups where different tenants may use different monitoring instances.
- Remove PrometheusURL field from WorkloadMonitorReconciler struct
- Remove --prometheus-url flag and prometheusUrl chart value
- Add resolvePrometheusURL() that reads namespace label
- queryPrometheusMetric() now accepts prometheusBaseURL as parameter
- Add tests for resolvePrometheusURL with and without label
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
- Check resp.StatusCode before parsing Prometheus response
- Limit response body read to 1 MB via io.LimitReader
- Use strings.HasPrefix in test instead of fragile slice indexing
- Add TestQueryPrometheusMetricServerError for 500 responses
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Query SeaweedFS_s3_bucket_physical_size_bytes alongside the logical
size metric. Physical size includes all replicas and reflects actual
disk usage, while logical size reflects what the user stored.
Refactor queryBucketSizeBytes into generic queryPrometheusMetric
to reuse for both metrics.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Query SeaweedFS_s3_bucket_size_bytes from a Prometheus-compatible API
to populate s3-storage-bytes resource on bucket Workloads. The
Prometheus URL is configurable via --prometheus-url flag. When set,
bucket WorkloadMonitors are requeued every 60s to keep sizes current.
When Prometheus is not configured, buckets still get tracked with
s3-buckets=1 for existence-based billing.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Add reconcileBucketClaimForMonitor() that watches COSI BucketClaim
objects and creates Workload CRDs with s3-buckets resource, following
the same pattern as PVC and Service reconcilers. This enables the
billing pipeline to discover and track S3 buckets per tenant.
Changes:
- Add COSI API types dependency (container-object-storage-interface-api)
- Register cosiv1alpha1 scheme in controller main
- Add BucketClaim watch in SetupWithManager
- Add BucketClaim list + reconcile in Reconcile loop
- Add RBAC annotation for objectstorage.k8s.io/bucketclaims
- Add unit tests for BucketClaim reconciliation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Add two features to improve application observability in the dashboard:
1. WorkloadsReady condition on Application status
- Query WorkloadMonitor resources to determine if all pods are running
- Expose WorkloadsReady as a separate condition alongside Ready
- Ready continues to reflect HelmRelease state only (no override) to
preserve backward compatibility with tooling and avoid false-negatives
during normal startup windows when pods are still coming up
- Handle three states: operational, not operational, unknown (pending)
- Fail-open on WorkloadMonitor query errors
- Integrate with Application Watch to emit MODIFIED events on
WorkloadMonitor changes (with initial-events-end safety)
- Register cozystack.io/v1alpha1 types in API server scheme
2. Events tab in dashboard
- Show Kubernetes Events scoped to application namespace
- Use status.namespace for Tenant applications
- Include both lastTimestamp and eventTime columns for K8s compat
3. Bug fix: WorkloadMonitor Operational status persistence
- Operational was written to stale 'monitor' instead of 'fresh'
inside RetryOnConflict, so it was never persisted to the cluster
Closes#2359Closes#2360
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Alexey Artamonov