The minio client was downloaded without -L, so when dl.min.io served
a 302 redirect, curl wrote the HTML body ("<a href=...>Found</a>") to
/usr/local/bin/mc, chmod +x'd it, and shipped that as the binary. When
the bucket E2E test ran "mc alias set ...", the shell tried to interpret
the HTML as a script and errored with:
/usr/local/bin/mc: 1: cannot open a: No such file
(dash reading "<a" as an input redirect from a file named "a").
The image is rebuilt per CI run from cached layers; the broken mc layer
was cached and reused. Adding -fsSL fixes the download and the textual
change busts the cache so the next build pulls a real ELF binary.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
When a disk entry has no bus set (the schema default), the previous
template rendered `disk:` with no value — YAML parses that as
`disk: null` and KubeVirt's virtualmachines-mutator webhook denies the
request with: "spec.template.spec.domain.devices.disks.disk in body
must be of type object: null".
Restructure the disk loop so the disk/cdrom value is always a valid
object: `disk: {}` when bus is empty, `disk: {bus: <x>}` when set.
Mirrors how `cloudinitdisk` already does it in the same template.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 0f3d2fa3a6)
Flux v2.8 / helm-controller v1.5.0 embeds Helm v4, which renamed --force
to --force-replace and rejects combining it with SSA's --force-conflicts.
Existing HelmReleases with spec.upgrade.force: true fail every upgrade with
"cannot use force conflicts and force replace together", looping
upgrade -> rollback -> upgrade and blocking dependants. Observed on PR #2619
CI for tenant-root/{ingress-nginx-system,seaweedfs-db}, tenant-test/info,
and others.
Per Flux docs, upgrade.force has been a no-op since v2.8 anyway, so removing
it is safe on green clusters and unblocks broken ones.
Also fix packages/apps/kafka/templates/workloadmonitor.yaml: it referenced
.Values.replicas (no such field; values.yaml has kafka.replicas and
zookeeper.replicas). Flux v2.7 silently dropped the null; v2.8 SSA forwards
it, and the WorkloadMonitor CRD rejects "spec.replicas: null" as not
type:integer, which blocks every kafka-test install.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Distroless images (cert-manager and friends) ship no shell or /bin/sleep,
so the previous `command: ["sleep", "infinity"]` failed with exec ENOENT.
Stage busybox:musl (statically linked) into a shared emptyDir from an
initContainer; every prepull container then execs /shared/sleep regardless
of what the image itself ships. The glibc busybox tag is unusable here —
it needs /lib64/ld-linux-x86-64.so.2 which distroless images lack.
Re-adds cert-manager to the prepull set in the install bats so its 4
images are cached on every node before the HelmRelease installs.
Verified locally in a single-node kind v1.32: all 4 cert-manager-* images
plus busybox pulled and held Ready under this DaemonSet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The prepull DaemonSet starts containers with `command: ["sleep",
"infinity"]`, which assumes the image ships /bin/sleep. cert-manager
images (cainjector, controller, webhook, startupapicheck) are
distroless and have only the cert-manager binary — the container
fails to exec with "sleep: executable file not found".
Verified locally on a kind v1.33.1 cluster: the sidecar-init
pattern (initContainer with restartPolicy: Always) doesn't help
either, because crashing distroless containers never reach a
persistent `started=true` state, so kubelet never progresses past
the first sidecar and the pod stays Initialized=False.
Revert to the original two-chart kubeovn + linstor prepull (both
non-distroless, both have sleep). The PackageSource dependsOn chain
added in 9dae97e is what actually prevents the cert-manager cascade —
prepull was only meant as a speed-up. Cert-manager bootstraps in a
couple extra minutes on cold install, but consumers wait correctly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
cozytest.sh runs @test bodies under /bin/sh, not bash. The prepull-loop
refactor in b68ac346 used bash array syntax (charts=( ... ),
rendered+=("$f"), "${rendered[@]}") which dash rejects at parse time
with "Syntax error: '(' unexpected", killing the entire Install
Cozystack step in <1s.
Replace with a space-separated string and word-split it on the for-loop,
which works under POSIX shell. The original two-package version was
POSIX-safe because it didn't use arrays.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Cold helm install of cert-manager pulls four quay.io/jetstack images
across three nodes from inside the QEMU sandbox. Image-pull contention
delays cert-manager controller readiness, which in turn keeps the
webhook Secret unissued and the downstream operator pods stalled in
ContainerCreating long enough to trip progressDeadlineSeconds. The
PackageSource dependency fix sequences cert-manager before its
consumers; pre-pulling shortens the cert-manager pod-start window so
the dependency wait doesn't itself blow the install budget.
Also refactors the prepull setup to iterate over a chart list rather
than duplicating mktemp/render/cat per package.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Five PackageSources whose charts apply cert-manager Certificate CRs and
mount the resulting Secret on a webhook Deployment lacked an explicit
dependsOn: cozystack.cert-manager. Under Flux v2.8 helm-controller v1.5.0
kstatus, the dependent Deployment trips progressDeadlineSeconds (10m
default) when cert-manager controller isn't running yet, and the HR is
failed early. Repeatedly observed on PR #2619 with capi-operator,
kamaji, rabbitmq-operator, cozystack-api (cozystack-engine), and
ingress-nginx all cascading to InstallFailed at ~12-15m into a fresh
install.
The mariadb-operator, kubeovn-webhook, snapshot-controller, etcd-operator,
postgres-operator, mongodb-operator, victoria-metrics-operator, linstor,
and linstor-scheduler PackageSources already had the dependency, which
is why they weren't in the failure set.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two timing bugs unmasked by Flux v2.8 kstatus, with the same root shape:
the bats steps assumed a side-effect ordering ("by the time X is true,
the parent HR's helm install is done") that held under v2.7's faster
dispatch but breaks under v2.8 where helm-controller walks kstatus on
every applied resource before flipping the HR Ready.
run-kubernetes.sh:
After every addon HR (cilium, coredns, csi, vsnap-crd, ingress-nginx)
reaches Ready, the remediation guard reads `.status.history` on the
parent kubernetes-${test_name} HR. The parent's helm install can still
be Reconciling at that point because kstatus is finishing its full
sweep; `.status.history` is empty until install completes, so the
guard exits 1 with "empty .status.history on a Ready HR". kubernetes-
latest masked this because the ouroboros assertions add ~10m of
buffer; kubernetes-previous (no ouroboros) hit the guard with the
parent still installing. Add an explicit
kubectl wait hr/kubernetes-${test_name} --for=condition=ready
before the guard.
vminstance.bats:
The test waited 120s for `kubectl get vmi vm-instance-$name` to
return a VMI with an assigned IP, then waited for the parent HR Ready
separately afterwards. Under v2.7 the parent HR's helm install
applied the VirtualMachine CR fast enough that kubevirt had created
the VMI before the 120s budget expired. Under v2.8 the dispatch is
slower and the VMI poll returned NotFound for its entire budget.
Reorder: wait for HR Ready first (which implies the VirtualMachine
CR is applied + kstatus-ready), then poll for the VMI with IP.
Loosen the VMI poll budget from 120s to 5m for nested-virt slow paths.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Same class of bug as the foundationdb misplaced-field fixes. The kafka
chart had:
entityOperator:
template:
pod:
metadata: { labels: { ... } }
spec: ← child of template, not of pod
enableServiceLinks: false
The Strimzi Kafka CRD's `entityOperator.template` has no `spec` key.
`enableServiceLinks` is a direct field of `template.pod` (alongside
metadata, affinity, tolerations, etc.). Flux v2.7's permissive
client-side patch silently dropped the misplaced field; v2.8's strict
Server-Side Apply rejects:
failed to create typed patch object (Kind=Kafka):
.spec.entityOperator.template.spec: field not declared in schema
Move `enableServiceLinks` under `template.pod` where the operator
actually reads it, and fix the inconsistent metadata.labels indentation
(was 9/11 spaces instead of 10/12) while we're here.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Follow-up to 2a764683: the chart had three more fields misplaced under
`automationOptions` that Flux v2.8 Server-Side Apply rejects with
"field not declared in schema":
.spec.automationOptions.imageType
.spec.automationOptions.labels
.spec.automationOptions.minimumUptimeSecondsForBounce
Per the FoundationDBCluster CRD schema all three are direct children of
`spec` (siblings of automationOptions). Flux v2.7's permissive client-
side patch silently dropped them; v2.8's strict SSA fails the install.
Move all three out alongside the previously-relocated faultDomain.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The standalone hack/e2e-apps/ouroboros.bats provisioned a full Kamaji-
backed tenant Kubernetes cluster (control plane + 2 CAPI workers + base
addons) just to flip `addons.ouroboros.enabled=true` on top, then ran
~15m of hairpin-NAT reconciliation assertions. Cluster shape was
identical to what hack/e2e-apps/run-kubernetes.sh already provisions for
kubernetes-latest, so the second Kamaji bringup was pure duplication —
roughly 25-45m of CI wall-time per run.
Add an `enable_ouroboros` 4th parameter to `run_kubernetes_test()`. When
"true", inject the ouroboros addon block into the Kubernetes CR's
addons map and run the post-cluster assertions inline:
- Wait for the ouroboros HR Ready in the host namespace.
- Verify coredns-custom ConfigMap exists in tenant kube-system.
- Wait for the ouroboros controller pod Ready.
- Apply an Ingress with a TLS host in tenant default namespace.
- Poll coredns coredns-custom for the rewrite line (5m budget).
- Resolve the hairpin host from inside the tenant via a netshoot pod
and assert it points at the ouroboros-proxy ClusterIP.
- Clean up dnscheck pod + Ingress.
kubernetes-latest passes "true" (gets ouroboros coverage); kubernetes-
previous leaves it empty (control-plane-only smoke remains).
(The standalone ouroboros.bats was removed in the preceding commit
alongside an unrelated foundationdb chart fix — the git rm staging had
already happened when the foundationdb commit landed.)
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Flux helm-controller v1.5.0 (shipped in Flux v2.8) replaced the legacy
Helm readiness logic with kstatus-based health checking, which polls
Deployment/StatefulSet/Service status and waits for kstatus to assess
rollout-ready. This typically adds 10-30s of latency between resource
apply and HR condition Ready=True compared to v2.7.
Our per-app bats tests had `kubectl wait hr/... --for=condition=ready
--timeout=` budgets of 20s, 30s, 60s, 100s — sized for v2.7's faster
readiness flip and previously masked by the 3x retry on Run E2E tests
(dropped in #2558). Under v2.8 with no retry, every app that runs after
the first few hit its HR wait timeout exactly, surfacing as 13 of 16
test failures on PR #2619.
Standardise on 5m for HR Ready across all app tests. Tight enough to
catch genuine bring-up failures within the per-app job budget, loose
enough to absorb kstatus dispatch jitter on a management cluster that
is concurrently reconciling multiple tenant-test HRs.
Post-HR-Ready checks for app-specific resources (StatefulSet/Service/
PVC/operator CRs) remain — for CR-based apps kstatus only observes
that the chart applied, not that the operator finished. Audit of
which of those are actually redundant under v2.8 kstatus is queued
as a follow-up.
Also bumps the 10s "kamajicontrolplane appears" wait in
run-kubernetes.sh to 2m, and the kubernetes-addon HR Ready loop from
1m to 5m for the same reason.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The chart placed `faultDomain` nested inside `automationOptions`, but the
FoundationDBCluster CRD schema defines `faultDomain` as a direct child of
`spec`. Flux v2.7 (helm-controller v1.4.3) used permissive client-side
patch and silently dropped the unknown field; v2.8 (helm-controller
v1.5.0) does strict Server-Side Apply and rejects:
failed to create typed patch object: .spec.automationOptions.faultDomain:
field not declared in schema
This was a latent bug from day one — `automationOptions.faultDomain`
never actually configured anything. Moving the field one level up applies
the value to where the operator looks for it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The bats step had two waits for tenant-root child HRs:
kubectl wait hr/etcd hr/ingress hr/tenant-root hr/seaweedfs --timeout=4m
kubectl wait hr/monitoring hr/seaweedfs-system --timeout=2m
The second line was the residue of a flux-reconcile-force workaround
removed in 7d292eb1; the mechanical cleanup kept the same HRs on a
separate 2m wait without revisiting whether the line still made sense.
After PR #2509 fixed the underlying timing bug:
- monitoring is a top-level tenant-root child, identical shape to etcd
and ingress; its Ready is already implied by hr/tenant-root Ready on
the first line.
- seaweedfs-system is an internal HR of the seaweedfs umbrella chart;
its Ready is already implied by hr/seaweedfs Ready on the first line.
So the second wait was always tautologically true by the time it ran,
and the naming inconsistency (top-level monitoring next to internal
seaweedfs-system) was an accident of the historical workaround. Promote
monitoring to the first line where it belongs and drop the redundant
second wait.
Also bump the wait from 4m to 10m. seaweedfs now installs as a serial
chain seaweedfs-db (CNPG bootstrap) -> seaweedfs-system (raft quorum)
-> seaweedfs wrapper, so the parent tenant-root HR's Ready flip lags
~5-6 min. The previous 4m budget was sized for the old single-HR
seaweedfs and is the immediate cause of the e2e failure on PR #2619.
The 10m budget stays inside tenant-root HR.spec.timeout (15m).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Mirror the diagnostics group already on `Run E2E tests` to the
`Install Cozystack into sandbox` step. Without this, install-step
failures (e.g. an HR not reaching Ready inside the bats `kubectl wait`
window) surface only as "error: timed out waiting for the condition
on helmreleases/<name>" with no cluster snapshot — re-running is the
only way to triage, which is the exact pattern the parent commit
removed.
On failure, dump three groups:
- `kubectl get hr -A -o wide` (last 100 lines): which HRs exist and
their Ready/Status columns.
- `kubectl describe hr` for each not-Ready HR (tail -50): the
Conditions block surfaces helm install/upgrade errors, missing
CRDs, and chart-render failures — the actual cause.
- `kubectl get events -A --sort-by=.lastTimestamp` (last 50): cluster
events that often pinpoint the failing pod/PVC/job.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on
.github/workflows/pull-requests.yaml:235: $attempt is incremented only on
failure, so first-try success printed "completed after 0 attempts". Use
$((attempt + 1)) so the message reflects the total number of runs.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Audit of 30 successful PR runs found that across 5 sampled failure
attempts, **25/25 retries** on `Run E2E tests` failed — the retry
loop never recovered a flake, only stretched deterministic failures
and tripled diagnostic wall-time. Same data shape on `Install
Cozystack`. Drop both.
`Prepare environment` keeps its 3x retry — that step is pure
infrastructure (Talos image download, sandbox VM boot, network)
where transient runner hiccups warrant a retry.
On failure, the test step now captures `kubectl get hr -A -o wide`
and `kubectl get events -A` under a collapsible group so triage
starts with the actual broken-state snapshot.
Depends on:
- #2508 — installer namespace bootstrap (Helm namespace-ownership conflict)
- #2509 — operator HelmRelease config knobs (`seaweedfs-system` 2-min wait race)
- #2528 — harbor bucket-secret + BucketInfo gating (harbor ValuesError)
- #2529 — objectstorage-controller BucketAccess conflict retry
- the daniil/split-vminstance PR (vminstance disk race + VM IP/ready timeouts)
- the daniil/split-event-driven PR (existence backstops surfacing real errors)
Until those land, dropping the retry will fail CI for unrelated PRs
that hit the seaweedfs / harbor / installer / vminstance races.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
PR #2619 split seaweedfs into a serialized chain
(seaweedfs-db -> seaweedfs-system -> seaweedfs) that needs ~5 min of
wall-clock to install: CNPG bootstrap, master quorum, then the wrapper.
With the prior 5m wait budget on the parent tenant HR, helm-controller
would time out before the children reach Ready, rollback would delete
the child HRs, and the upgrade would retry in a loop. The e2e bats step
that patches the Tenant CR and waits 60s for child HRs to exist landed
in the rollback gap and timed out.
Two complementary changes:
- tenant-root is created statically by cozystack-basics, so its
HelmRelease spec.timeout is bumped 5m -> 15m directly.
- Sub-tenants (Tenant CRs) get their HR built by cozystack-api from
the tenant ApplicationDefinition. Add the
release.cozystack.io/helm-install-timeout=15m annotation that
kubernetes-rd already uses to opt the kind into the longer budget.
Keeps both values in lockstep and documents the coupling inline.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous healthCheckExprs flagged the seaweedfs-db HR as failed within
seconds of creating the Cluster CR. CNPG sets Ready=False with
reason=ClusterIsNotReady as the initial bootstrap condition before postgres
starts, so the generic `failed: ... Ready=False` predicate matches the
normal startup state and aborts the install early. This cascades: tenant-root
times out waiting for seaweedfs, rolls back (deleting child HRs), retries,
and the e2e "Configure Tenant" step races the rollback gap.
Use Flux's three-predicate form: route ClusterIsNotReady to `inProgress`
so the HR keeps polling, keep `failed` for any other future Ready=False
reasons, and rely on the HR `timeout: 10m` as the real backstop for a
genuinely stuck cluster.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Each of the 19 child HR templates in packages/apps/kubernetes/templates/
helmreleases/ added the parent kubernetes-application HR as a dependsOn
entry, guarded by a `lookup "helm.toolkit.fluxcd.io/v2" "HelmRelease"`
on the parent so it only stamped after the parent already existed.
Under Flux 2.7 (helm-controller v1.4.x), the parent HR's Ready=True flips
after helm install returns, regardless of any child HRs it just applied,
so the guard was a no-op for first install and a soft ordering hint for
subsequent reconciles.
Under Flux 2.8 (helm-controller v1.5.0), the default waitStrategy is
`poller` -- the parent HR now waits via kstatus for every applied
resource, including the 19 child HRs, to be Ready=True before flipping
its own Ready. The child HRs in turn cannot become Ready until they have
a working `super-admin.svc` kubeConfig and the lookup-stamped dependsOn
back-edge resolves -- which only happens after the parent HR is Ready.
Deadlock: parent waits on child, child waits on parent.
Removing the parent dependsOn from each child breaks the cycle. The
children start reconciling immediately; helm-controller's existing retry
loop covers the brief window before the parent's helm install lands the
admin-kubeconfig secret. The cross-child ordering (cilium -> cert-manager,
prometheus-operator-crds -> metrics-server, etc.) stays intact via the
unconditional sibling entries.
Five files (the *-crds and vsnap-crd HRs) had only the parent reference
under dependsOn; remove \`dependsOn:\` entirely in those. Cilium had one
conditional gateway-api-crds entry under it; pull the \`dependsOn:\` key
inside that conditional so it disappears when gateway-API is off rather
than rendering as a null array.
This is option B from issue #2642 task 1.
Verified:
- \`helm template\` clean with default values-ci.yaml and with every addon
enabled (cert-manager, ingress-nginx, monitoring-agents, velero, fluxcd,
gateway-api, gpu-operator, hami, ouroboros).
- No remaining \`- name: {{ .Release.Name }}\` dependsOn entries in
packages/apps/kubernetes/templates/helmreleases/.
- \`helm unittest\` passes (123/123).
Refs: #2642
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The packages/system/vertical-pod-autoscaler chart's vpa-for-vpa.yaml template
emits an HR in cozy-vpa-for-vpa that the parent helm install applies and waits
on. That nested HR had `dependsOn: monitoring-agents`, which never resolves:
parent VPA helm install
waits for nested vpa-for-vpa HR Ready
waits for monitoring-agents HR Ready
waits for cozystack.vertical-pod-autoscaler PackageSource Ready
waits for parent VPA HR Ready <-- back to start
Symptom observed on PR #2619 e2e (cozyreport):
Helm upgrade failed for release cozy-vertical-pod-autoscaler/
vertical-pod-autoscaler with chart cozy-vertical-pod-autoscaler@...:
timeout waiting for: [HelmRelease/cozy-vpa-for-vpa/vpa-for-vpa
status: 'InProgress']
The recommender's /health-check readiness probe is served by the binary
itself and doesn't require Prometheus. Removing the dependsOn lets the
nested HR become Ready quickly; the recommender will log connection
errors against vmselect until monitoring-agents installs later in the
boot sequence, then start producing recommendations normally.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main has a conditional dependsOn:[ingress-nginx-system] in the application
HR (added after #2601 branched). PR #2601 added an unconditional
dependsOn:[seaweedfs-db]. The merge produced two `dependsOn:` keys in the
same HelmRelease spec, which the Helm post-render rejected:
Helm install failed for release tenant-root/seaweedfs ...:
error while running post render on files: map[string]interface {}(nil):
yaml: unmarshal errors
Collapse into one list: the db dependency is unconditional; the ingress
dependency stays inside the existing `if eq $ingress .Release.Namespace`
guard so sub-tenants that inherit ingress from a parent namespace don't
deadlock on a non-existent local HR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The pre-pull test captured each rendered chart into a shell variable
(`kubeovn_manifests=$(helm template ...)`) so a render failure would
trip `set -e` (bats has no `pipefail`, so piping helm into yq would mask
a helm failure with yq's exit code).
cozytest.sh runs every bats test under `set -x`, and bash expands
captured command output into the trace as `+ var=<entire stdout>`.
Two full helm renders (kubeovn + linstor) plus a `printf` re-expansion
were ballooning the install step to ~40k log lines per CI run.
Stage each render to a tmp file instead. The same exit-code semantics
apply (a failed `helm template ... > file` still trips `set -e`), but
the trace now only shows the commands themselves, not the rendered YAML.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`make generate` in packages/apps/vm-instance/ rebuilds the instanceProfile
enum from packages/system/kubevirt-instancetypes/templates/preferences.yaml.
The pre-commit job picked up the drift after the upstream refresh in the
previous commit. No source changes — only regenerated outputs.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`make update` stripped `persistent: true` from upstream's
`preferredTPM: { persistent: true }`, leaving the parent key with a null value.
KubeVirt v1.6's CRD tolerated it; v1.8 (the operator version we bumped to in
6af0bf71) enforces structural schema and rejects null:
VirtualMachineClusterPreference "windows.11" is invalid:
spec.devices.preferredTPM: Invalid value: "null": spec.devices.preferredTPM
in body must be of type object: "null"
Stop stripping the line and regenerate the templates from upstream
common-instancetypes. Persistent TPM state falls back to the cluster default
StorageClass with RWO access mode when `vmStateStorageClass` is unset on the
KubeVirt CR, per pkg/storage/backend-storage/backend-storage.go.
Drive-by from the regen:
- Removed (upstream EOL): centos.7*, centos.stream8*
- Added: debian, oraclelinux, legacy, linux*, fedora.s390x, rhel.{9,10}.s390x,
windows.{xp,7,7.virtio,2k3,2k8,2k8.virtio,2k12,2k12.virtio}, plus new
cx1.*1gi and d1 instancetype variants.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Splits the seaweedfs-system HelmRelease in two so the CNPG
Cluster/seaweedfs-db lives in its own HR (seaweedfs-db), and the
application HR (seaweedfs-system) dependsOn it. The new HR uses Flux v2
HelmRelease.spec.healthCheckExprs with a CEL expression on
Cluster.status.conditions[type=Ready] plus waitStrategy.name: poller, so
its Ready=True only flips after the postgres primary is actually serving
connections — not just after helm install applied the Cluster CR.
This eliminates the seaweedfs-filer CrashLoopBackOff race on a fresh
tenant install. With Cilium kubeProxyReplacement: true, socket-LB
returns EPERM from connect(2) to ClusterIPs with no Ready endpoints.
Pre-split, the filer StatefulSet scheduled concurrently with the CNPG
bootstrap (~55–70s of unavailable postgres), each connect() failed
EPERM, kubelet exponential restart backoff pushed past the e2e bats
'kubectl wait hr/seaweedfs-system --timeout=2m' window, and the
'Configure Tenant and wait for applications' test failed.
Migration 39 adopts existing Cluster/seaweedfs-db resources into the new
release on upgrade by rewriting meta.helm.sh/release-name and stamping
helm.sh/resource-policy: keep.
Requires Flux 2.8.x (helm-controller v1.5.0+). Cozystack currently pins
Flux to 2.7.x; a separate PR bumps the pin.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Re-render internal/fluxinstall/manifests/fluxcd.yaml via
`make update` in packages/core/flux-aio (timoni bundle build against
oci://ghcr.io/stefanprodan/modules/flux-aio at the latest tag) so
the cozystack-operator binary embeds Flux 2.8.0 instead of 2.7.3.
Auto-syncs the helm-controller image into fluxcd-tenants.yaml.
Embedded versions now match the chart bump in the prior commit:
flux: v2.7.3 -> v2.8.0
helm-controller: v1.4.3 -> v1.5.0 (enables healthCheckExprs)
source-controller: v1.7.3 -> v1.8.0
kustomize-controller: v1.7.2 -> v1.8.0
notification-controller:v1.7.4 -> v1.8.0
source-watcher: v2.0.2 -> v2.1.0
Verified by rebuilding cmd/cozystack-operator and running `strings`
on the binary -- the expected v1.5.0 / v1.8.0 / v2.1.0 image strings
are now embedded in place of the v1.4.x / v1.7.x ones.
Structure preserved:
- single AIO Deployment named "flux" in namespace cozy-fluxcd
- all 5 controllers + source-watcher as containers in one pod
- custom tolerations from flux-aio.cue intact
- Namespace pod-security label still "privileged" via the
Makefile yq post-processing step
Net diff is -3664 lines (mostly CRD schema simplification in 2.8).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Bump flux-operator and flux-instance vendored charts from v0.33.0 to
v0.48.0, moving the Flux distribution from 2.7.x to 2.8.x. This brings
helm-controller v1.5.x, which exposes spec.install.healthCheckExprs on
HelmRelease — a prerequisite for adding real readiness gating to
dependency-target packages (postgres, mongodb, kafka, etc.).
The new flux-operator chart defaults web.enabled=true; override it to
false in packages/system/fluxcd-operator/values.yaml so the Flux Status
Web UI stays opt-in. Operators wanting it can set web.enabled=true and
configure web.config / web.configSecretName / web.ingress per the
upstream chart values.
Patches reapplied:
- kubernetesEnvs.diff: matched cleanly (offset shift only)
- networkPolicy.diff: extended to also restrict port 9080 (web UI port);
hunk count updated 20 -> 22
Pre-flight: no references to the v0.39-removed --disable-wait-interruption
flag anywhere in packages/ or internal/.
Verified locally:
- helm template renders both packages without errors
- web.enabled=false (default): no web Deployment/Service/NetworkPolicy
- web.enabled=true: renders web NetworkPolicy on port 9080
- CiliumClusterwideNetworkPolicy lists 8080/8081/9080
- FluxInstance distribution.version: "2.8.x", all 7 components present
- Cozystack kustomize patches (concurrent, requeue, storage-adv-addr,
events-addr) intact
Notable upstream changes between v0.33 and v0.48:
- Flux 2.8.x with HelmRelease inventory format change (auto-migrated)
- Reliability fixes for stuck FluxInstance/ResourceSet on health-check
cancellation (v0.39, v0.46)
- New CLI: flux-operator migrate, diff, patch instance, distro mirror
- ResourceSetInputProvider: GitLab Environments, GitHub App, Gitea/Forgejo
- CVE-2026-23990 (Web UI OIDC impersonation bypass) — N/A while web is off
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This PR adds the changelog for release `v1.3.4`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.3.4.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* OpenSearch is now integrated into the 1.3 PaaS bundle so operator and
dashboard components deploy when PaaS is enabled.
* Storage class input now renders as a dropdown including OpenSearch as
an option.
* **Bug Fixes**
* Fixed Harbor reconciliation regression related to bucket secret
handling and render-path coverage.
* **Documentation**
* v1.3.4 release notes published, expanded docs, summary of related
project changes, contributor list and compare link.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2676?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Adds the missing entry for #2649 (OpenSearch wired into PaaS bundle, plus
StorageClass dropdown override for the legacy 1.3 openapi-ui dashboard),
corrects six website-PR author attributions against gh pr view, picks up
two website docs PRs merged just before the v1.3.4 cut (cozystack/website#536
and #538), updates the contributors list, and refreshes the date to today.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This PR adds the changelog for release `v1.4.0`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.4.0.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Released v1.4.0 changelog describing a redesigned schema-driven
dashboard, persistent tenant storage, unified instance-type presets,
declarative multi-database backups, optional fractional GPU sharing, and
PROXY-protocol support
* Includes upgrade guidance, operator timing knobs, kubelet/autoscaler
behavior changes, security fixes, component version bumps, and full
changelog link
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2675?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
OpenSearch has shipped as a complete package set in the repo for several
releases — `packages/apps/opensearch/`,
`packages/system/opensearch-operator/`,
`packages/system/opensearch-rd/`, and `PackageSource` files under
`packages/core/platform/sources/` — but the PaaS bundle template was
never
updated to include the two `cozystack.opensearch-*` PackageSources.
Result on any cluster with `bundles.paas.enabled=true`:
- `opensearch-operator` is not deployed → no `OpenSearchCluster` CRD on
cluster
- `opensearch-rd` release is not deployed → no
`ApplicationDefinition/opensearch` → the dashboard catalog has no
OpenSearch entry and tenants cannot create
`opensearches.apps.cozystack.io`
This PR adds the missing includes alongside every other DB
operator/application in
`packages/core/platform/templates/bundles/paas.yaml`.
### Release note
```release-note
Fix: OpenSearch is now installed by the PaaS bundle (`bundles.paas.enabled=true`); previously the operator and ApplicationDefinition were defined but never referenced from the bundle, so OpenSearch did not appear in the dashboard catalog.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* OpenSearch operator and application packages are now included in the
PaaS bundle when enabled, providing additional search and analytics
capabilities for supported deployments.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2648)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Reverts the `publishing.exposure=loadBalancer` mode for ingress-nginx
added in #2468.
The mode rendered the ingress-nginx Service as `type: LoadBalancer` and
a `CiliumLoadBalancerIPPool` over `publishing.externalIPs`, as a
migration path off `Service.spec.externalIPs` (deprecated upstream in
Kubernetes v1.36 per KEP-5707). In default Cozystack the implementation
produces two failure modes:
- `publishing.externalIPs` is the operator-supplied list of node public
IPs (addresses the cluster nodes already hold and listen on, see [docs
section "4b. Node Public IP
Setup"](https://cozystack.io/docs/v1.3/install/cozystack/platform/#4b-node-public-ip-setup)),
not a virtual-VIP pool. Treating it as one and DNATing those IPs via
Cilium LB-IPAM collides with the existing ingress-nginx DNAT on the same
IPs (`Service.spec.externalIPs` path), so traffic is forwarded
non-deterministically.
- Cilium in default Cozystack runs with `ipam.mode: kubernetes` and no
L2/BGP announcer, so a pool over addresses outside the node range is
allocated but never advertised — the Service stays unreachable.
The KEP-5707 deprecation is real, but the right migration is a
mechanism-agnostic tenant API knob translated by a Cozystack-side
controller into whichever backend the cluster admin has configured
(MetalLB pool, Cilium LB-IPAM, robotlb, externalIPs pin) — not a
chart-side `IPPool` baked into the tenant chart. That follow-up belongs
in its own PR.
### Removed
- `publishing.exposure` value (and the inline comment) from
`core/platform/values.yaml`.
- `expose-mode` plumbing line in `core/platform/templates/apps.yaml`.
- `CiliumLoadBalancerIPPool` template at
`apps/tenant/templates/cilium-lb-pool.yaml`.
- `type: LoadBalancer + externalTrafficPolicy: Local` branch in
`extra/ingress/templates/nginx-ingress.yaml`. The Service now renders as
`ClusterIP + spec.externalIPs` when externalIPs are set, falling through
to `type: LoadBalancer` when they are not — pre-#2468 shape.
- Exposure mode section from `extra/ingress/README.md`.
- `exposure_test.yaml` fixtures in both charts.
### Compatibility
Operators who explicitly set `publishing.exposure=loadBalancer` will see
their ingress Service revert to `ClusterIP + spec.externalIPs` on
upgrade. Anyone on the default value (`externalIPs`) is unaffected.
### Release note
```release-note
revert(ingress): remove the publishing.exposure=loadBalancer mode added in #2468. The default externalIPs path is unchanged; operators who explicitly opted into loadBalancer will have their ingress Service revert to ClusterIP + spec.externalIPs on upgrade. Migration off Service.spec.externalIPs (deprecated in KEP-5707) will be reintroduced via a mechanism-agnostic API knob in a follow-up.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Refactor**
* Removed conditional generation of the Cilium load‑balancer IP pool and
simplified service exposure to rely on explicit external IP values.
* Streamlined configuration for service exposure keys.
* **Documentation**
* Removed the exposure-mode guidance from ingress documentation.
* **Tests**
* Deleted the exposure-mode-related test suites covering various
external IP and loadBalancer scenarios.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2603)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Marks the `generate-changelog` job in `.github/workflows/tags.yaml` with
`continue-on-error: true`, and changes the `update-website-docs` gate to
`always() && needs.prepare-release.result == 'success' &&
needs.prepare-release.outputs.skip != 'true'`.
Result: a failure inside `generate-changelog` (AI step timing out, PR
creation step erroring, etc.) no longer blocks `update-website-docs`.
`prepare-release` remains the hard gate — if it fails or signals
`skip=true`, downstream jobs still don't run.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
This pull request contains internal infrastructure changes only and has
no impact on end-users or the product.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2670?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from gemini-code-assist on
docs/changelogs/v1.4.0.md:326: the in-prose mention of dislogical was
rendered as inline code rather than as a contributor link. Switch to
the [**@user**](https://github.com/user) format used elsewhere in the
changelog.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on
docs/changelogs/v1.4.0.md:102 and coderabbitai on :103: the AMD
SEV-SNP feature gate in upstream KubeVirt source is WorkloadEncryptionSEV
(see pkg/virt-config/featuregate/active.go), not WorkloadEncruptionSEV
as quoted from the v1.7.0 release notes.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Mark the generate-changelog job with continue-on-error and switch the
update-website-docs gate to always() so a failed or skipped changelog
generation no longer blocks the website docs update. prepare-release
remains the hard gate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix regression of broken auth by token to the cozy-dashboard
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed a regression in JWK cache initialization where custom HTTP
client configurations for cluster authentication were not being properly
applied, ensuring secure cluster communication and service-account token
handling work as intended.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2661)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Adds `docs/changelogs/v1.2.4.md` so the `Update Release Notes` workflow
can publish the existing `v1.2.4` draft release.
`v1.2.4` is a hotfix patch backporting only the duplicate-`template:`
key removal from #2279 — the breaking service-naming changes in that PR
were intentionally omitted as not appropriate for a patch (see #2652 for
the minimal-backport details).
The `Generate Changelog` step on the tag-push workflow failed twice in a
row (yesterday's cron run and today's manual dispatch) due to `402 You
have no quota` on the Copilot API, so this changelog is hand-written
following `docs/changelogs/patch-template.md` and matches the v1.2.3
style.
## Test plan
- [x] file follows `patch-template.md` structure
- [x] author attribution verified against `gh pr view` for #2279 and
#2652
- [ ] CI passes
- [ ] after merge, `Update Release Notes` picks up the file and
publishes draft release `v1.2.4`
Hand-written patch changelog for v1.2.4 (Copilot quota exhausted on the
automatic Generate Changelog step). Single fix: minimal backport of the
MariaDB backup CronJob duplicate-key removal from #2279 via #2652.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The opensearch chart, operator chart, ApplicationDefinition and
PackageSources have existed since the original OpenSearch series, but the
PaaS bundle template never references the opensearch-operator or
opensearch-application PackageSources. On isp-* variants with
bundles.paas.enabled=true, neither the operator nor the ApplicationDefinition
is ever installed, so the dashboard catalog has no OpenSearch entry and
opensearches.apps.cozystack.io resources cannot be created.
Add both includes to packages/core/platform/templates/bundles/paas.yaml,
following the same pattern used for every other DB application.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## Summary
Replaces the old openapi-ui with the new cozystack-ui build from the
cozystack-ui repository.
## Key Changes
### New UI
- **Image**: `999669/cozystack-ui:latest@sha256:ea4e832c2...`
- **Architecture**: `linux/amd64`
- **Repository**: https://github.com/cozystack/cozystack-ui
(feat/ui-overhaul branch)
### Removed Components
- Removed BFF (Backend-For-Frontend) container - new UI communicates
directly with Kubernetes API
- Removed old openapi-ui and openapi-ui-k8s-bff build targets from
Makefile
- Removed old UI Dockerfiles and patch files
### RBAC Updates
- Added read permissions for `cozystack.io/applicationdefinitions` to
fix 403 errors
- New UI requires access to applicationdefinitions for marketplace and
catalog functionality
### Bug Fixes
- Fixed VNC WebSocket connections to use dynamic URLs instead of
hardcoded localhost:8001
- VNC now works properly in-cluster deployments
## Testing
Tested on dev3.infra.aenix.org cluster:
- ✅ Dashboard loads correctly
- ✅ Application definitions accessible
- ✅ VNC connections working
- ✅ Backup pages functional
- ✅ All TypeScript errors resolved
## Migration Notes
The new UI is a complete rewrite with:
- Modern React 19 + TypeScript stack
- Improved backup management pages
- Better type safety and error handling
- Direct Kubernetes API communication (no BFF needed)
