A rebase re-applied the namespaced CiliumNetworkPolicy that the move to
source-side clusterwide policies had deleted, so two policies now guard
the same webhook port and the older one is the worse of the two.
It is gated only on admissionWebhookPolicy.enabled, which defaults to
true, and never checks that the Cilium API exists. On a cluster without
Cilium every tenant ingress release therefore fails to install with
`no matches for kind "CiliumNetworkPolicy"`.
On a Cilium cluster it repeats the target-side deny the clusterwide
policy already carries. It also renders for child tenants, whose
controller runs with the admission webhook disabled and for which the
clusterwide policy renders nothing, so there it denies a port nothing
listens on and is the only thing that puts a cluster-spanning peer onto
the controller endpoints. That peer is enumerated per pod identity, the
coupling to namespace churn the clusterwide policy was written to
contain.
The clusterwide policy and the admissionWebhookPolicy value it reads are
left as they are. The existing capability suite selects only the
clusterwide template, so it could not have noticed this file coming
back; a suite with no template filter now fails if any Cilium kind
renders while the Cilium API is absent.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 8c1d2dbee2)
helm-controller waits on every object of a release with the kstatus
poller on install and on every upgrade. kstatus reads a TenantGateway at
Ready=False/ListenersNotReady as InProgress, and a TenantGateway sits
there whenever one listener certificate is pending, which depends on
ACME or DNS and, under HTTP-01, on routes that other releases own. After
a platform upgrade the gateway release therefore times out, retries
without end under RetryOnFailure, holds a helm-controller worker
meanwhile, and fails the tenant release that waits on it. It used to
pass because health is judged only inside a Helm action: the release
was installed while the TenantGateway had no status yet or was still
Ready, and the listeners whose certificates do not issue came later
with other releases' routes.
An always-true expression replaces kstatus for TenantGateway and keeps
the wait on. Flux checks status.observedGeneration against the
generation before it evaluates any expression, so the release still
waits until the controller has observed the new spec, but not for
listener or certificate health. The poller strategy is set explicitly
because expressions are only evaluated under it.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5265ee662c)
Removing on-connect auto-registration made several user-facing and doc texts
wrong: they promised that tapping registers a repository's apps in the catalog
and runs its charts. Tapping now installs nothing, so the texts told operators to
wait for apps that never appear.
Reword the `cozypkg tap`/`untap` help, the tap success output (now printing a
`cozypkg add <name>` line per materialized PackageSource), the `cozypkg init`
scaffold README, the packages-index trust model, the dashboard Repositories
banner and per-repository empty state, and the stale re-registration code
comments. No behaviour changes beyond the added `cozypkg add` hint in the tap
output.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 2f39e304f0)
Image overrides let a tenant run an arbitrary container in the
management cluster, including helpers with access to the Talos CA
secrets and the tenant cluster's API credentials. Drop those overrides
and run the images the platform ships.
The Kubernetes and KubernetesNodes helpers (the wait-for-kubeconfig
and Proxmox cloud-config init containers, the bootstrap-token and OIDC
RBAC Jobs, the talos-csr-signer sidecar, the Talos reconcile and unpin
Jobs) take the bundled image references through cozy-lib.image, so
they still follow the configured registry. OpenSearch no longer sets
an image and leaves it to the operator, which follows the version the
chart's version map selects; an explicit version: null now fails the
render instead of falling back to v2. The Talos installer settings for
tenant worker VMs stay.
FoundationDB replaces the free-form cluster.version with version, v7.3
or v7.4, mapped to 7.3.63 and 7.4.1; the 7.1 line is no longer
offered. The chart refuses a render where cluster.version and version
name different release lines, which also covers variants without
platform migrations. Migration 60 carries an existing cluster.version
over to its release line and leaves the legacy value in place while
the old chart can still render it. It records what needs attention in
the cozystack.io/migration-60-needs-attention annotation on the
cozystack-version ConfigMap: a FoundationDB release it cannot carry
over or whose patch changes, a release that still sets one of the
removed image overrides, an OpenSearch release storing version: null.
A dry run reports them without patching, annotating or advancing the
stamp. Migration 56 stops carrying the removed kubectl override into
adopted node pools.
OpenSearch and FoundationDB get version-map update scripts that keep
the current default while it is still supported.
Stored image overrides are still accepted and the charts ignore them.
On the first reconcile OpenSearch nodes move to the image of their
selected version and Kubernetes components whose image changed
restart. Air-gapped installations need docker.io and ghcr.io mirrored
on the management-cluster nodes.
Assisted-by: LLM
(cherry picked from commit 749d401899)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A Velero restore of a VMDisk or a VMInstance deletes the disk's
DataVolume and restores the PVC under its name, with no owner. The
restored release is in sync, so nothing recreates the DataVolume until
the next Helm upgrade, typically a platform update. That upgrade is
refused by CDI's admission webhook ("Destination PVC ... already
exists") and the release stays failed, followed by the VMInstance
release that looks the disk up.
CDI admits a DataVolume over an existing claim only when the
DataVolume carries cdi.kubevirt.io/allowClaimAdoption or the
DataVolumeClaimAdoption feature gate is on. The restore controller
puts the annotation on the restored PVC, where CDI never reads it, and
the CDI CR does not enable the gate. Setting it on the DataVolume
keeps the scope to vm-disk instead of every DataVolume in the cluster,
and does not depend on a gate CDI intends to remove. A claim CDI
created for this DataVolume carries its uid in createdForDataVolume
and is never treated as adoptable, so a fresh import is unaffected.
The VMDisk backup round-trip now forces an upgrade after the restore,
which is what reaches the failure, and checks that the DataVolume
adopted the restored PVC instead of replacing it.
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
(cherry picked from commit b4f6fb67db)
Backport of #4819 to `release-1.7`. The five commits are cherry-picked
in order from 1913104bd, d2c6b85ca, 5592a3f65, ecdc043e6 and e8635f6e4.
One conflict, in `.github/renovate.json`: the rule #4819 adds sits next
to two e2e sandbox rules from #4867, which is not on this branch. I kept
this branch's rules and added only the seaweedfs rule. The diff is the
same nine files, +281/-2, as #4819.
The bug is the same here. This branch runs the upstream
`chrislusf/seaweedfs:4.45` image, and its S3 gateway is OOM-killed while
serving a large object (#3782).
This needs to land before rc.2 is cut, otherwise rc.2 and v1.7.0 ship
the upstream image.
Valkey runs on the same spotahome RedisFailover operator as Redis and
speaks the same replication protocol and RDB format, so it needs no
strategy kind of its own. The only difference the backup Job sees is the
chart's resource-name prefix: valkey-<app> instead of redis-<app> for the
sentinel Service and the auth Secret.
The Redis driver now accepts applicationRef kind Valkey as well as Redis
and hands the template .Release.Prefix, derived from the app's kind, which
cozy-default-redis uses for every name it builds. A kind with no known
prefix fails the render instead of producing an empty one. cozy-default
routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must
target an app of the same kind as the backup's source: replaying a Redis
dump into Valkey, or the reverse, is not something the strategy can
promise across engine releases.
The shipped template reads the prefix as `.Release.Prefix | default
"redis-"`: during an upgrade the previous controller, which has no Prefix
in its render context and serves only Redis, renders the new CR, and a
bare reference would print "<no value>" into the names.
The redis:8 image already loads RDB v11, which both shipped Valkey
versions (7.2, 8.1) write.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
(cherry picked from commit 0c2f131192)
A restarted capi-operator v0.19.0 drops the create events for the
provider Deployments it finds already running
(kubernetes-sigs/cluster-api-operator#1031). A provider whose Ready
condition went False while the operator was down therefore keeps it
until that Deployment changes or the informer resyncs, and the first
resync after a restart lands two periods later because the handler
registers after the informer has listed. With the 10m default that is
about 20 minutes, during which capi-providers-core times out on the
CoreProvider and everything depending on it waits; every further
operator restart starts the clock again.
With a one-minute period the same repro recovered 2m02s after the
operator took its lease, against no recovery within 4m26s with the
default on the same cluster. A resync replays the cache and, with
providers in sync, ends in "No changes detected" and an unchanged
condition, so it writes nothing to the API server; the operator logged
about 15 lines a minute and used 2m CPU and 26Mi of memory.
The upstream chart has no value for the flag, so a package patch that
make update reapplies adds one, and a bats test holds the rendered
argument.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit c2dab12a2e)
backupstrategy-controller renders its strategy CRDs from templates, so
Helm stores each of them twice in every release revision, and a
revision is one Secret capped at 1 MiB. The Altinity, Job and
FoundationDB CRDs each inline the full upstream PodTemplateSpec schema,
about 600 KB apiece; once the default Strategy CRs render as well, the
stored release no longer fits and the forced upgrade that creates them
fails with "Secret ... is invalid: data: Too long".
The Kafka, RabbitMQ and Redis strategies already carry their pod
template schemaless for this reason. Give the three older fields the
same markers. The Go types stay typed, the drivers decode the template
as before, and the apiserver still validates the pod when the driver
creates the workload; every description is kept, and the largest
revision drops to about a quarter of the limit. The rationale moves
to the package doc, so the six field docs only tell the reader what
schemaless means for them.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 2f6c6a27b8)
When CDI clones a golden image into a StorageClass other than the one
the golden PVC lives in, its csi-clone path, and its snapshot path once
it gets a VolumeSnapshotClass of the provisioner, check the two classes
for a common driver, a matching volume mode and room to expand, and
never compare the class names. A target class of the same provisioner
therefore gets no host-assisted fallback and the clone is left to the
CSI driver. LINSTOR creates that clone next to the source data, so on a
target class that does not allow remote volume access from every node,
such as a node-local one, the temporary clone PVC stays unbound with
nothing surfaced to the user.
Refuse such a disk at render time, mirroring the planner's inputs: the
CDI override or the target StorageProfile's strategy, the snapshot class
CDI would pick, the golden's driver as its PV records it, the target's
provisioner, the volume modes and the target's expansion setting, and
refuse only a LINSTOR target whose remote access policy, parsed the way
linstor-csi parses it, is not plainly "from anywhere". A snapshot class
both StorageProfiles name but that does not exist is refused as well,
because CDI then fails to plan the clone instead of falling back. An
empty storageClass is resolved the way CDI resolves it for a kubevirt
DataVolume. A disk whose DataVolume or PVC already exists is exempt,
since CDI can garbage-collect a completed DataVolume. Combinations that
fall back to a host-assisted copy, targets reachable from any node, and
anything else the lookups cannot find still render.
This changes behaviour for a new disk in a LINSTOR class without remote
access from every node, and for one whose StorageProfiles name a
snapshot class that does not exist: both rendered before and are refused
now. The way out is the golden image's class, or the copy clone
strategy.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit d28834231c)
Changing the variant of the cozystack.cozystack-platform Package on a
running cluster changes bundles.system.variant and can re-render the
cozystack.networking Package with another variant. The operator then
deletes every HelmRelease of that Package the new variant does not
install, so isp-full -> isp-hosted removes Cilium and Kube-OVN and the
cluster loses its CNI; isp-full -> isp-slim removes Kube-OVN.
The platform render now looks up the live Package and fails when its
variant differs from the one about to be rendered, unless
networking.acknowledgeVariantChange names the new variant. The
acknowledgement is a variant name rather than a boolean so that a value
left behind after one switch does not cover a later one. The check is
inert where lookup returns nothing (helm template, client-side dry-run,
first install) and skipped when cozystack.networking is in
bundles.disabledPackages, since the platform then does not render it.
A live variant set by hand, such as cilium-kilo, is pointed at
bundles.disabledPackages rather than at the acknowledgement: the
acknowledged render would write the bundle's variant over it.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 1245b32a32)
The builder, the compile flags and the version stamp changed, so the
image built from them has a new digest, and values.yaml has to name it
for the components to run that build.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit e8635f6e4b)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The in-tree build pinned its source, base image and toolchain by hand,
and nothing moved them when `make update` vendored a new chart, so a
bump would build one release's weed into another release's image, or
keep patching a release that already carries the fix.
Pin the source archive by checksum and the upstream image by digest as
build arguments, compile with the toolchain upstream's own builder uses
for the tag, and let `make update` re-pin all of them from the vendored
chart and fail when the patches no longer apply. The tests of the
patched packages now run once rather than once per target platform,
and `weed version` names the build.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 5592a3f65d)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
With the image keys unset the vendored chart renders chrislusf/seaweedfs
at its appVersion, so the patched build from images/seaweedfs would be
built and never run, and the S3 gateway would keep being OOM-killed on
large GETs. Point every component at the in-tree image, pinned by
digest like every first-party reference, and hold that in a unit test
so a values change cannot quietly fall back to the upstream binary.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit d2c6b85cac)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The S3 gateway is OOM-killed while serving a large object. SeaweedFS
starts read-ahead downloads after every read of a sequential stream and
counts only the downloads it newly starts toward its prefetch window,
so a GET reading 256 KiB at a time out of 4 MiB chunks pulls up to
about a gigabyte of the object into memory, past the 512Mi limit of
the S3 deployment. No SeaweedFS release carries a fix yet, so there is
no release to move to instead.
Build weed from the 4.45 sources with a patch that keeps the window at
the next prefetchCount chunks, and lay it over the upstream image of
the same tag so the rest of the image stays upstream's own. The image
build runs the tests of the packages the patch touches, including the
regression test the patch adds, because go build never compiles them.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 1913104bd3)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
linstor-csi restores a snapshot on one node and autoplaces the other
replicas. In LINSTOR v1.33.3 the restored volume definition has no
DRBD_INITIALIZED flag, so the initial UpToDate race is still open, and
Linstor/Drbd/InitialUptoDateOn, copied from the source volume through
the snapshot, names the node that won it for the source. When a
replica is autoplaced on that node, the empty new volume wins the
race: it gets a random current UUID and a fresh filesystem, and goes
StandAlone against the restored replica. A pod attached there sees an
empty volume.
Backport the upstream fix, released in LINSTOR v1.34.0, which sets the
flag on a restore unless the resource definition asks for new metadata
through ForceInitialSyncPermanent, as a backup does. Nothing wins the
race any more, and an autoplaced replica resyncs from the restored one
like any replica added to an existing volume.
Link: https://github.com/LINBIT/linstor-server/commit/9d8cdf5118338fc72a34bb99799e452e031ea8f6Closes: #4862
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit e5689e2a63)
The pre-install/pre-upgrade hook applies the node pools and the drain
KafkaRebalance with kubectl, and its container has no resources of its
own, so in a tenant with resource quotas it runs under the LimitRange
default of 128Mi. When it applies an object that already exists,
kubectl downloads the cluster's whole OpenAPI v2 document only to
decide that a custom resource takes a JSON merge patch: with the 374
CRDs of an e2e cluster that peaks at about 300Mi, so every upgrade of a
Kafka in such a tenant OOM-kills the hook and fails with "pre-upgrade
hooks failed".
--openapi-patch=false skips that lookup and builds the same three-way
merge patch from the last-applied annotation, peaking at about 35Mi on
the same cluster. A server-side apply would be as small but changes
what gets removed: the "kafka" pool is created with "kubectl create",
whose manager keeps co-owning every field it set, so a later
server-side apply leaves those fields in place when the chart drops
them. A larger memory limit would charge every upgrade against the
tenant quota instead.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 6ba440f7a3)
Strimzi sizes a broker heap by asking the image's JDK 17 what heap it
would pick at MaxRAMPercentage=50, capped at 5 GiB.
On kernels built without CONFIG_MEMCG_V1 that JDK cannot see the limit
and reports host RAM, so every broker and controller starts with up to
a 5 GiB heap inside a 1Gi limit and is OOM-killed. The topic and user
operators run the same JDK with percentage-based defaults, and the
drain-time Cruise Control takes 75% of host RAM for its heap.
Each of these containers now gets JAVA_TOOL_OPTIONS with -XX:MaxRAM set
to its memory limit, plus -XX:ActiveProcessorCount where the chart sets
a CPU limit. The entity operators and Cruise Control have none in the
chart, so a LimitRange default still leaves them counting host CPUs:
Strimzi template env takes no resourceFieldRef. Every java in the
container reads JAVA_TOOL_OPTIONS, Strimzi's heap probe included, so
Strimzi's own heap rules and Kafka's G1 defaults apply to the real
memory limit. jvmOptions would not do: Strimzi turns -XX options into
KAFKA_JVM_PERFORMANCE_OPTS, and kafka-run-class.sh skips its G1 settings
when that is set. Each JVM prints a "Picked up JAVA_TOOL_OPTIONS" line
on start. Existing clusters roll once on upgrade, because the pod
environment changes.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 479d17abc8)
The JDK 17 in the Strimzi 0.45.1-rc1 operator image decides whether to
read the container limits by looking at /proc/cgroups. Kernels 6.12 and
newer built without CONFIG_MEMCG_V1 no longer list the memory controller
there, so the JVM turns container support off and takes host RAM and
host CPUs as its budget: on a 128 GiB node it picks G1 with a 25 GiB max
heap and a 2 GiB initial heap inside a 512Mi limit, and the kernel
OOM-kills the operator under load. A restart at the wrong moment leaves
a freshly created StrimziPodSet without pods for up to the five-minute
informer resync, which is how a KRaft controller pool ends up never
starting within an e2e timeout.
The limits now reach the JVM through the downward API as -XX:MaxRAM and
-XX:ActiveProcessorCount, so they follow any later change to the
container resources. In a test cluster the operator then picks Serial
GC and a heap of about 100 MiB. The upstream JDK fix is JDK-8347811,
backported to 17.0.20 and 21.0.10.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit b255dc4dd8)
The multus chart takes its resources from values, but the platform
renders the cozystack.multus Package with stageCniPlugins alone, and an
edit to that child Package is reverted by the platform reconcile. A
running cluster therefore had no way to raise the multus limits.
networking.multus.resources is passed to the Package only when set, so
an unset key leaves the chart default in charge. A value that is not a
map stops the render here; the multus chart keeps refusing a render
without a memory limit or request.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit ddd00f8639)
kube-multus serves the CNI call for every pod sandbox on its node, and
its 100m CPU limit was hardcoded in the patched upstream template.
After a CNI outage a node had ~600 dead sandboxes and 220 pods waiting:
multus was throttled in every CFS period, calls outlived kubelet's
two-minute CRI deadline, each retry grew the queue, and the node never
recovered. With the CPU limit lifted the 1Gi memory limit OOM-looped;
2 CPU and 4Gi cleared the backlog in about three minutes.
Resources now come from values, defaulting to the same requests, no
CPU limit and a 4Gi memory limit. The memory limit stays so the pod
cgroup keeps memory.max and remains outside the Talos OOM handler's
candidate set.
Since an explicit null in a values override could drop either memory
key, the render now fails without limits.memory or requests.memory.
Without the request it defaults to the limit, and a node with less
allocatable memory than that is left without multus.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit dca2e4ff0b)
The kubevirt-csi node plugin mounts the NFS export root into a temp
dir to create /data and migrate old files. That mount was hard, so
with the server gone the mkdir, readdir and rename calls on it never
returned. kubelet gives up after 2m and retries, and every retry
blocked the same way, piling up goroutines and nfs-init-* dirs.
Mount the temp root soft, so its RPCs fail once the connection to the
server is declared dead instead of blocking forever; it only does
metadata work that a retried publish redoes. retry=0 keeps mount.nfs
from retrying a failed temp mount for 2m. timeo and retrans keep their
defaults, because the first mount of a server sets them for the
connection the pods' mounts of it share. The volume mount the pod gets
stays hard, since a soft mount can report a failed write the server
applied.
Over NFSv4 that soft failure takes minutes, and a stat of a target
that is already hard-mounted can block for good, so a publish for a
target path that already has one running now returns Aborted instead
of starting another.
A soft stat can fail with EIO on a file /data already holds, and a
rename issued once the server is back would replace it, so the
migration now fails the publish on any stat error other than a
missing destination. It checks the destination with lstat, so a
user's symlink in /data that loops or dangles counts as present
instead of failing every publish or being replaced by the rename.
The skip it makes for a destination already in /data leaves the root
entry where the pod cannot see it, so it is now logged as a warning.
/data is created with a plain mkdir, since with the server gone
MkdirAll fails its stat on the mount point and then reports the local
temp dir as already existing instead of the EIO. The temp mount
errors now name the volume, like the migration errors do.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit f76dedf9ca)
A disk whose HTTP source answers 404 or does not resolve never leaves
ImportInProgress: CDI fails an import only on a fatal error and restarts
the importer pod otherwise, so the condition only ever said the disk was
still importing. The reason it will not finish was visible only in the
importer logs and on the DataVolume, whose Running condition carries the
error through the back-off between restarts.
Appending that message lets the user see why the disk is stuck from the
application status and the console, which already shows the
WorkloadsReady message for a disk.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 0b9a801a2b)
## What this PR does
Backport of #4889 to `release-1.7`. The branch was cut on October 12,
two days before #4889 merged, so its vmalert still sends to the shared
Service and an upgrade from 1.6.5 to 1.7.0 would bring the bug back. The
commit is cherry-picked from 9d4221b89 without conflicts, and after it
the monitoring chart here is identical to main.
The monitoring chart tests pass on this branch, 24 suites and 304 tests,
the new suite included. Without the fix the new suite fails, 2 of 3
tests.
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Same diff as #4889, two templates, the helpers and a test file of
`packages/system/monitoring`, so no trigger map entry applies.
### Release note
```release-note
fix(monitoring): vmalert now sends alerts to every Alertmanager replica instead of the shared Service, so the Alertmanager cluster deduplicates them and a long-running alert no longer produces repeated firing and resolved notifications.
```
The pre-delete hook read a release's state once, so a single failed
read was taken as the release being handled: its fence came off before
helm-controller's uninstall, and the report counted it as released
cleanly. A state read is now retried within the release's share of the
window, a release whose state still cannot be read keeps its fence for
the wait like one helm-controller has not handled, and the end of the
wait reads each again before naming what the release was left to.
A release still suspended after the sweep is named as suspended but not
by flux-plunger, which also holds when no manager is recorded for the
field at all.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 7011510456)
The README gave the recovery steps in the order flux-plunger used before
the processed-version annotation moved ahead of the Secret delete. An
operator who finds the annotation on a release whose revision Secret is
still there would read that state as impossible, when it is what a
failed delete leaves and the next pass retries.
Two boundaries were left unsaid. The pre-delete hook stops, waits for and
lists again only for the flux-plunger of its own release; another one,
under a second release name or with changed labels or out of cluster,
keeps the fences it sets and gives them up itself. And a backup restore
that starts while flux-plunger holds its own suspension does not take
the field over, since the restore steps write it only when it differs,
so the end of the recovery unsuspends the release under the restore.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 568e6f5e18)
The pre-delete hook's retry budget was one window spent in listing
order, so a release that could never be read or unfenced, listed
first, took all of it and every release after it got a single attempt.
The same transient failure then released a fence or not depending on
where its release sorted. Each release now gets its own share of the
window, capped at the window's end, and the steps that cover every
release keep the whole of it.
Two comments said more than holds. The retry window is measured from
the script's start, not the Job's, so scheduling and the image pull come
out of its one-minute reserve rather than being covered by it. And the
five-minute conflict retry inside kubectl scale never runs here, since
without a precondition it sends a merge patch with no resourceVersion
to conflict on.
helm-controller's finalizer, which the hook reads to tell whether a
release being deleted is still held, is now tied to the constant the
controller uses, as the fence already was.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 699555c7cb)
The pre-delete hook lifted flux-plunger's suspension of a release being
deleted and removed its fence in the same step. helm-controller runs the
uninstall of a release being deleted once it is no longer suspended, but
when its own finalizer is already gone the fence is all that keeps the
object, so removing the fence at once deleted the release before
helm-controller could reconcile it, without an uninstall. The hook now
keeps such a fence while helm-controller handles the generation the
lifted suspension wrote, until shortly before its retry window closes,
and names a release it still had to let go of as before.
A release being deleted under a suspension another manager also holds
was reported as cleanly released, although helm-controller skips the
uninstall of a suspended release; it is now named as deleted without
one and counted with the others.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit af50c4a6b4)
flux-plunger raised its SuspensionNotOwned and TerminatingWhileSuspended
Warnings on a suspension it does not own when the release carried its
processed-version annotation. That annotation is never removed, so
every later suspension of such a release, by a migration, a backup
restore or an operator, drew a Warning: one telling the reader to
resume it, which mid-restore is the one thing nobody wants, and one
calling a migration's deliberate suspend-then-delete an accident. Both
also missed the leftover they are for when the first recovery stopped
before writing the annotation.
A suspension an earlier flux-plunger left mid-recovery freezes the
release in the failure recovery started from, since helm-controller
rewrites no condition of a suspended release. Both Warnings now key on
that state instead, and the README's candidate query with them.
Two comments are qualified where they claimed more than holds: a stop
that overran the manager's grace period can still let a reconcile set
a fence after the drain's last listing, and a process stopped before
controller-runtime closes Elected holds the Lease yet does not drain.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 7b84919db3)
A drain deadline of zero or less is already spent when the drain
starts, so flux-plunger would exit holding every uninstall fence
without listing one. The chart could produce one: its 40-second floor
was a literal, so raising the manager's stop bound or the margin before
SIGKILL, which the drain subtraction is already tied to, would leave the
floor rendering a non-positive deadline. The binary now refuses such a
flag at startup, before it reaches for a cluster, and the chart's floor
and the timings the README and the values comment state are held to
the same constants.
A whole number of a million seconds or more printed in exponent form
and was refused as a fraction; the range is now checked first, so it
gets the range message.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 8bcb709ce8)
flux-plunger recovers a "has no deployed releases" HelmRelease by
briefly suspending it, deleting the stale storage Secret and
unsuspending. helm-controller v1.5.x skips the uninstall and drops its
finalizer when it handles a delete on a suspended HelmRelease, and it
wakes on the same delete event flux-plunger does, so a delete inside
that window orphaned the release's DataVolume, PVCs, RBAC and storage
Secret. Lifting the suspension on the delete event cannot win that race
reliably.
Hold the release with a flux-plunger finalizer while it is suspended.
The finalizer is applied together with the suspension in one server-side
apply. On a delete flux-plunger lifts its suspension and keeps the
finalizer, helm-controller reconciles the unsuspended generation and
runs the uninstall, and flux-plunger removes the finalizer once
helm-controller's own is gone and that generation is observed. This
holds whichever controller handles the delete first. A completed
recovery lifts the suspension and the finalizer together, and a
terminating release never begins recovery.
Decide ownership of a suspension by which field manager owns
spec.suspend in metadata.managedFields rather than by an annotation. The
apiserver transfers the field away the moment an operator, cozyhr resume
or a backup controller writes a different value to it, so a later
suspension is never misread as flux-plunger's, and one set through the
shared flux-client-side-apply manager (platform migrations, the tenant
delete hook, a manual kubectl apply) is never claimed. A release deleted
under another manager's suspension is left to that manager's decision.
Acquire applies without forcing first and, on a conflict, re-fetches and
forces only if spec.suspend is still false: a backup controller owns the
field at false through a dynamic-client Update, which a non-forced apply
can never take, while true is an active suspension to leave alone. Every
apply is preconditioned on the resourceVersion and uid of the read that
decided on it. Without the resourceVersion, the unforced apply of true
over a foreign true is accepted as shared ownership, a window the
optimistic-locked merge patch this replaces had closed, and lifting the
suspension and the finalizer from a release read as live would delete it
without an uninstall if helm-controller had meanwhile handled a delete.
Without the uid, an apply to a release that has gone would create an
empty one. The apply carries only spec.suspend and the finalizer, so it
cannot leak the required spec.interval.
Read the release, every re-check and the storage Secret list from the
apiserver rather than the informer cache. A lagging cache would let the
processed-version guard delete the revision below the one just
reclaimed, and the ClusterRole grants secrets get and list but not the
watch an informer needs.
Emit a Warning when flux-plunger holds a suspension it failed to clear,
and Normal events when it leaves an external suspension alone or defers
a recovery. An external suspension on a release flux-plunger has
recovered before is a Warning instead: it is most likely one an earlier
flux-plunger left under the shared manager, which nothing lifts unless
an operator does. A release deleted under another manager's suspension
is a Normal event when that was a migration's choice, and a Warning when
the release carries flux-plunger's processed-version annotation, since
then the skipped uninstall is an accident.
Keep a single writer with leader election. A second process able to act
under the same field manager could clear a suspension the first is still
recovering, and no rollout strategy prevents one: the next pod starts as
soon as the old one is terminating, and an eviction or a node drain
replaces a pod the same way. The old pod keeps its Lease, unreleased,
through its stop and drain, and the Lease outlasts both, so the next pod
acts only after the old one is done; the price is that a new pod waits
for the Lease to run out. The strategy sets maxSurge 0 with
maxUnavailable 1 and stays RollingUpdate because existing installs carry
the apiserver-defaulted rollingUpdate block and helm-controller applies
server-side, where type Recreate over it is rejected on upgrade.
Give the finalizer a way out when flux-plunger itself goes. Once the
manager has stopped, within a bounded time, and with it every reconcile,
flux-plunger drains its finalizers under a deadline that, with the stop,
fits inside the pod's termination grace period. A stop that overran its
bound is drained too and still reported, and only a pod that led drains.
The drain writes its events straight to the apiserver, since the
manager's recorder stops with the manager. A recovery in progress is
aborted, the suspension and the finalizer lifted together for the next
flux-plunger to start again, and a release being deleted has the
suspension lifted and keeps the finalizer until helm-controller has run
the uninstall. A release helm-controller does not handle in time keeps
it. The waiting lives in the controller, where the envtest suites hold
it against a real apiserver's generations and finalizers, rather than in
a shell script against a model of them.
A pre-delete hook on the chart is the backstop for what the drain could
not finish, so uninstalling the component never leaves a release that
tenant teardown waits on forever. It stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and finalizer without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, with a JSON patch preconditioned on the resourceVersion and
uid it read: kubectl apply --server-side on a release last written by a
client-side apply moves that apply's fields to the applying manager and
strips every one the body leaves out. The Job's deadline sits under the
300s helm-controller gives a pre-delete hook by default. A Deployment
that cannot be stopped or a listing that never succeeds ends the hook
before any finalizer is touched, and a finalizer it cannot remove ends
it after the sweep; either way it exits non-zero: a failed pre-delete
hook leaves the release untouched and helm-controller retries the
uninstall, hook included, from a state it can still converge from, where
exiting 0 would remove the component and strand the finalizer. A
flux-plunger pod still present after its grace period, typically one
stuck Terminating on a NotReady node, is only reported, since failing on
it would fail every retry until the node returns; because it may still
be setting finalizers, they are listed again after the sweep and any
found fails the hook. The Job runs once, since helm-controller's retry
is the one that starts afresh, and is kept after success so that its
report stays readable; helm.sh/hook-output-log-policy would make
helm-controller call a nil output writer (fluxcd/helm-controller#1562).
A delete still held after ten minutes raises a Warning, and the held
state is re-checked on a timer instead of only on helm-controller's next
status write. The chart README documents the finalizer and how to clear
it.
Run the envtest suites on every pull request through a new make target
that fetches a kube-apiserver and etcd with a pinned setup-envtest and
takes the HelmRelease CRD from the in-tree Flux manifests: the unit
tests drive a fake client, and everything the finalizer rests on (apply
preconditions, the finalizer set-merge, real managedFields) only holds
against a real apiserver. CI caches those binaries, and the target uses
them without asking the release index, so a pull request does not depend
on their sources being reachable; the control plane gets a longer start
timeout because it starts beside the other test goals.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 42a27567ce)
vmalert had a single notifier, the headless vmalertmanager-alertmanager
Service. It resolves to all three pods, but each alert reached one replica
at a time and moved to another when vmalert reconnected. On clusters running
1.6.4 all active alerts sat on one replica while the other two had none, and
on one of them the alert count of each replica changed about 170 times a
day. The Alertmanager cluster deduplicates notifications only when every
replica receives every alert. The replica an alert left let it expire and
sent a resolved notification, and the replica it reached sent a new firing
one.
vmalert now lists each Alertmanager pod as its own notifier, using the
<name>-<i>.<service>.<namespace>.svc address the operator itself gives
StatefulSet pods. The replica count comes from one helper that also sets
spec.replicaCount of the VMAlertmanager, so the two cannot drift.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 9d4221b890)
The CRD gate has now been missed twice: MongoDB shipped without it, and
Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only
checks the templates a case lists, so a new strategy-*-default.yaml
that forgets the gate stays green until an upgrade wedges.
Render the chart with the bucket gate open and no CRD served, and fail
on any strategy.backups.cozystack.io object from any template. Also
check that every rendered Strategy CR gates on the CRD this chart ships
for its kind, because a gate naming the wrong CRD passes the first
check and keeps the CR out of every live render.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit f47bab7280)
The chart ships its strategy CRDs as ordinary templates, and Helm maps
every document of a release through the RESTMapper before it applies
any. A default Strategy CR gated only on the bucket name therefore
fails Build() with "no matches for kind" whenever the bucket name
resolves before the CRD is served, the release applies nothing, the
CRDs are never created, and every retry renders the same manifest.
That happens on the upgrade that first ships a kind: the Kafka,
Rabbitmq and Redis CRs leave a v1.6 cluster's HelmRelease stuck in
UpgradeFailed on the way to v1.7. It also happens on the first render
of an external-S3 install (provisionBucket=false), where the bucket
name comes from values and every older default CR renders next to its
absent CRD.
MongoDB already gates on crdEstablished for this reason. Give every
default Strategy the same gate. On an installed cluster the CRDs are
established and the render is unchanged; otherwise the first revision
creates the CRDs and skips the CRs, and DefaultObjectsGate, which reads
the cozy-default BackupClass routes, forces the follow-up upgrade that
renders them once the kinds are served.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 7204992452)
## What this PR does
Adds two optional values to the Redis app, `minReplicasToWrite` and
`minReplicasMaxLag`, rendered as `min-replicas-to-write` and
`min-replicas-max-lag` in the RedisFailover `spec.redis.customConfig`.
Both are unset by default, so the rendered config and behaviour are
unchanged for existing instances.
Replication is asynchronous, so a master that is isolated from its
replicas keeps accepting writes until Sentinel fails over, and those
writes are lost. With `minReplicasToWrite: 1` the isolated master stops
accepting them after `minReplicasMaxLag` seconds, which bounds the loss.
Trade-off: with the default `replicas: 2` there is a single replica, so
the master also refuses writes whenever that replica is down (node
drain, restart, upgrade). Users who turn this on accept lower write
availability for a bounded loss window.
Includes helm-unittest cases and the regenerated schema, README, Go
types and ApplicationDefinition.
### Screenshots
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/68
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(redis): add optional minReplicasToWrite and minReplicasMaxLag values to bound writes lost on failover; unset by default
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added optional Redis settings to require a minimum number of replicas
to acknowledge writes and to set the maximum age of those
acknowledgements.
* Invalid settings are rejected: the required replica count must be
nonnegative and below the configured replica count, and the
acknowledgement age must be at least 1 when a replica requirement is
set. Replication remains asynchronous, so these settings do not
eliminate potential write loss during failover.
* **Documentation**
* Documented the settings, defaults, validation, and behavior in Redis
configuration guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
cozystack-ci[bot]