## What
Consolidated landing branch for a set of interdependent platform/CI
stabilization fixes. It began as the source PRs in the table below (in
dependency order) and has since grown with review-driven fixes and a few
production-behavior changes — the full current inventory is under
**Scope** below. Landing them as one unit lets CI run against the real
combined state instead of a fragile bottom-up merge train.
## Source PRs (dependency order)
| # | Commit | Source PR | Addresses |
|---|--------|-----------|-----------|
| 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi
crashloop during cert provisioning |
| 2 | test(e2e): bucket.bats port-forward + S3 client reliability |
#2944 | flaky bucket test |
| 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop
volumeSizeLimitMB | #2943 | missing bucket/access classes |
| 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps)
| #2874 | cilium "IP already in use" leak |
| 5 | refactor(build): mode=max registry cache + main-only warmer |
#2938 | build cache |
| 6 | ci(build): isolate each PR build on its own ephemeral runner VM |
#2939 | shared-buildkit contention |
| 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline |
#2928 | LINSTOR provisioning timeouts |
| 8 | test(metallb): assert digest-pinned image form, not version
literal | #2873 | brittle metallb assertion |
## Why consolidated
These form a dependency DAG (verified from CI logs): e.g. 2943 needs
2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested
bottom-up, each lower PR runs with none of the fixes above it and can't
go green alone. This branch carries all of them, so CI runs against the
real combined state.
## Verification status
CI is green on the latest head (`5653c30`): full-suite E2E passes
end-to-end and is reproducible (2 of 3 runs on this SHA green). The one
red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the
sandbox, unrelated to the diff — a different environmental subsystem
failed each run (details in the comments below). The earlier 3-hour
crust-gather snapshot hang is fixed in `5653c30`.
## Scope beyond the original 8
The branch has grown past the 8 source PRs above with review-driven
fixes and a few production-behavior changes. For reviewer transparency,
the full set:
**Review fixes (@lexfrei review):**
- **B1 (blocker)** — `8f41910` converts the SeaweedFS
`-lock`/`-readonly` BucketClasses and the s3 service-name override into
`patches/`, wired into `make update`, so a re-vendor no longer drops
them.
- **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods
(both delete remedies were RBAC-forbidden).
- **FU3** — `014de3f` adds a unit test for the HelmRelease update
conflict-retry path.
- **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any
non-terminal owner phase, not just `Running`.
- **FU7** — `f3914d2` sets `persist-credentials: false` on the
build-cache checkout (SHA-pinning is handled repo-wide in #2849).
**Production-behavior changes (not in the table above):**
- VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring
(`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops
install-time eviction churn.
- cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease
update path (`728f676`).
**E2E robustness follow-ups:** per-test crust-gather snapshots
(`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node
single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget
(`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin
(`95f42aa`).
**Not addressed (by decision):** FU2 — the s3 Service keeps
`seaweedfs.name`-based naming via patch (rendered output unchanged;
cozystack renders `fullname == seaweedfs`, so it matches the
`componentName`/`fullname` siblings in practice). FU6 — fork PR build
push is handled by the existing mirror-to-same-repo-branch workflow.
## Relates to
#2946#2944#2943#2874#2938#2939#2928#2873 — if this lands, those
can be closed; otherwise they remain the granular per-PR review path.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added CI build cache warming for main branch builds.
* Added a best-effort in-cluster Cilium endpoint self-heal watchdog for
e2e installs.
* Extended SeaweedFS COSI with object-lock support plus separate
readonly access.
* **Improvements**
* Reduced e2e install flakiness with a shared readiness deadline, safer
waits, and improved cleanup/diagnostics capture.
* Prevented install-time churn by switching VPA update mode to
**Initial**.
* Added container startup probes and updated SeaweedFS volume sizing.
* **Tests**
* Improved e2e robustness (etcd backup schedule waiting, digest-pinned
image checks, and corrected S3 port-forwarding).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
When publishing.certificates.wildcardSecretName is set, the root
tenant's ingress-nginx controller now serves the operator-supplied
wildcard Secret as its --default-ssl-certificate, and the platform
service Ingresses stop requesting their own per-host ACME certificates.
The flag is set only on the publishing controller (Release.Namespace ==
expose-ingress) because ingress-nginx loads the Secret from its own
namespace; child-tenant controllers cannot reach a Secret in another
namespace, so they keep the existing ACME path.
Each service Ingress (dashboard, keycloak, grafana, alerta, bucket,
linstor-gui, harbor, seaweedfs S3, bootbox) drops its cert-manager
cluster-issuer annotation, the http01 challenge-class annotation, and
its tls block in wildcard mode; nginx then falls back to the default
certificate for those hosts. Other annotations and routing are
unchanged. Each file keeps its existing render guard intact, and the
annotations block collapses cleanly where the cert annotations were its
only content.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new HelmRelease configurations for cert-manager, monitoring
agents, and Victoria Metrics Operator in Kubernetes.
- Added resource specifications for `vmselect` in the VMCluster
configuration.
- Enhanced resource management for `vmselect` with defined limits and
requests for memory and CPU.
- **Bug Fixes**
- Adjusted resource limits for Redis failover memory allocation.
- **Documentation**
- Updated README and release notes for various components, enhancing
clarity and usability.
- **Chores**
- Updated image versions across multiple components for consistency and
performance improvements.
- Modified migration scripts to facilitate transitions and manage
resources effectively.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Andrei Kvapil <kvapss@gmail.com>
- Remove grafana-oncall
- Add Alerta
- Configure basic alerts
- Update grafana 10 --> 11
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added new configuration options for the Alerta service, enhancing user
customization.
- Introduced a new Helm chart for the VictoriaMetrics Kubernetes stack,
enabling comprehensive monitoring solutions.
- Added VMAuth feature for enhanced authentication in the Kubernetes
stack.
- **Bug Fixes**
- Fixed issues with the ETCD dashboard and improved ingress path prefix
handling.
- **Documentation**
- Updated README and release guide for the VictoriaMetrics stack with
installation and configuration instructions.
- Introduced a changelog for organized tracking of changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
This change is aimed at improving the development experience.
- The option `make delete` has been added.
- Added check for `NAME` and `NAMESPACE` variables
- Now, any package (not just system ones) can include options such as
make show, make diff, make apply.
- Applications from packages/extra require explicit specification of the
`NAMESPACE`.
- Applications from packages/apps require explicit specification of both
`NAME` and `NAMESPACE`.
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
myasnikovdaniil