mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-10 06:24:09 +09:00
564ca281e4339d9557bbf640e1818cf2bfbab895
7590
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Andrey Kolkov
|
564ca281e4 |
test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.
Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
up via BackupClass + BackupJob, restores in place, and asserts the VM returns
Running off a disk PVC the restore created: a new uid, Velero's restore-name
label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
enough, because existingResourcePolicy: update also stamps it on a PVC Velero
only patched, which moves no data. A guest-side boot proof is unavailable: the
KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
place with the same three assertions. The controller prepares the target
(suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
class the Velero data mover waits for a Pod to select a node for the restored
PVC, which a detached disk never gets, so the script starts a pause Pod on the
new PVC, standing in for the VM that would attach it.
The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
(cherry picked from commit
|
||
|
Andrey Kolkov
|
da209397a2 |
fix(backups): prepare standalone VMDisk restore targets before the data mover runs
collectUnderlyingResources returned nothing for a non-VMInstance kind, so
prepareForRestore did nothing for a standalone VMDisk: no HelmRelease suspend and
no DataVolume delete. An in-place VMDisk restore then raced Flux/CDI — the
restored HelmRelease reinstalled and CDI recreated the DataVolume over the same
PVC name the data mover was repopulating, wedging the RestoreJob.
Report a standalone VMDisk's own disk as its underlying resource, so the existing
(VMInstance-proven) restore preparation suspends the vm-disk-<name> HelmRelease,
deletes its DataVolume and — with keepOriginalPVC — renames the live PVC before
the data mover recreates and repopulates it. No VM, so no IP/MAC is collected.
A VMInstance is halted before its disks are touched; a VMDisk target is not,
and every VMInstance disk is a VMDisk. Renaming the PVC of a disk a running VM
mounts leaves the original Terminating behind pvc-protection, Velero skips a
PVC that still exists, and the RestoreJob would report Succeeded having moved
nothing while the VM loses its disk on the next restart. An in-place VMDisk
restore now fails before changing anything when a Pod that has not terminated
mounts the disk's PVC, naming the Pod and telling the user to stop the VM or
restore the VMInstance instead.
A VMDisk Backup taken before the disk was recorded as its underlying
resource has none, so its disk is now derived from the application name
and such a restore is prepared and checked for a mounting Pod as well.
The per-disk orLabelSelectors stay a VMInstance concern. A VMDisk
strategy already selects its disk, and Velero rejects a Backup or
Restore that carries both labelSelector and orLabelSelectors, so
appending them for a VMDisk would break a strategy written with
labelSelector.
When an in-place restore fails after preparation, nothing is rolled
back: resuming the disk's HelmRelease while its PVC is renamed and its
DataVolume deleted would have CDI import a fresh disk under the original
name, which looks healthy and hides that the data sits in the -orig-
PVC. The failure message and a Warning event now name the suspended
HelmReleases and the -orig- PVCs so the user can recover them, read back
from the cluster, so a failure before a step ran does not name what it
would have left.
Preparation takes the target apart before Velero validates the Restore,
so it is refused up front when the Velero Backup is gone or not
Completed. That check now runs on every pass behind the same failure
path as the rest of preparation, and a FailedValidation Restore fails
the RestoreJob like a Failed one instead of leaving it Running. A
HelmRelease suspend or a VM halt that did not happen now stops
preparation instead of renaming and deleting under a running release or
VM, and a VMInstance's disks are renamed only once its launcher Pod has
let go of them. A VMDisk held only by a Pod already being deleted is
waited for rather than refused, for
up to 30 minutes after the RestoreJob started.
Between renaming a kept disk's PVC and deleting its DataVolume, CDI can
recreate an empty PVC under the original name. Velero then skips the
PVC as already present, the RestoreJob reports Succeeded and the VM
boots a blank disk. Once the DataVolume is gone, preparation now
deletes a PVC found under the original name, read uncached, before the
Velero Restore is created.
Preparation spans several reconciles, so the RestoreJob records a
PreparationStarted condition before it first changes the target, and
retries without changing anything when recording it fails. A
refusal on a later pass (the Velero Backup expired, a VM started on the
disk) then names what preparation left behind instead of claiming that
nothing changed, and a preparation still waiting 30 minutes after it
started fails the same way instead of polling forever.
Now that a VMDisk target is prepared, keepOriginalPVC=false reaches it
too: its DataVolume is deleted with orphan propagation and the bound PVC
is deleted before the Restore, as for a VMInstance disk. Left in place,
Velero would skip it and CDI would refuse the DataVolume the resumed
HelmRelease creates.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
2c3ff51a2a |
[Backport release-1.7] feat(backups): back up Valkey with the Redis strategy (#5032)
# Description Backport of #4898 to `release-1.7`. |
||
|
myasnikovdaniil
|
7c7d60611c |
[Backport release-1.7] fix(cluster-api): resync capi-operator every minute (#5031)
# Description Backport of #4939 to `release-1.7`. |
||
|
Andrey Kolkov
|
e2e0684b19 |
test(e2e): run the Redis backup round-trip against Valkey
The Redis backup demo now takes APP_KIND=Valkey, which switches the app
kind and the chart prefix the helpers derive sentinel and Secret names
from. The demo BackupClass routes Valkey to the same demo strategy.
The valkey suite gains valkey-backup-roundtrip, which drives that demo
end to end: seed a marker, back it up, restore it in place and into a
copy. It writes to tenant-root's seaweedfs through seaweedfs-s3, so a
seaweedfs change now selects the valkey suite too, and since the
round-trip runs examples/backups/redis as its harness, an edit there
selects both the redis and the valkey suites. The reduced E2E install
keeps the backup stack for the valkey suite, as it does for the other
backup round-trip suites.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
(cherry picked from commit
|
||
|
Andrey Kolkov
|
82af1a457b |
feat(backups): back up Valkey with the Redis strategy
Valkey runs on the same spotahome RedisFailover operator as Redis and
speaks the same replication protocol and RDB format, so it needs no
strategy kind of its own. The only difference the backup Job sees is the
chart's resource-name prefix: valkey-<app> instead of redis-<app> for the
sentinel Service and the auth Secret.
The Redis driver now accepts applicationRef kind Valkey as well as Redis
and hands the template .Release.Prefix, derived from the app's kind, which
cozy-default-redis uses for every name it builds. A kind with no known
prefix fails the render instead of producing an empty one. cozy-default
routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must
target an app of the same kind as the backup's source: replaying a Redis
dump into Valkey, or the reverse, is not something the strategy can
promise across engine releases.
The shipped template reads the prefix as `.Release.Prefix | default
"redis-"`: during an upgrade the previous controller, which has no Prefix
in its render context and serves only Redis, renders the new CR, and a
bare reference would print "<no value>" into the names.
The redis:8 image already loads RDB v11, which both shipped Valkey
versions (7.2, 8.1) write.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
cc8d6a4f05 |
fix(cluster-api): resync capi-operator every minute
A restarted capi-operator v0.19.0 drops the create events for the
provider Deployments it finds already running
(kubernetes-sigs/cluster-api-operator#1031). A provider whose Ready
condition went False while the operator was down therefore keeps it
until that Deployment changes or the informer resyncs, and the first
resync after a restart lands two periods later because the handler
registers after the informer has listed. With the 10m default that is
about 20 minutes, during which capi-providers-core times out on the
CoreProvider and everything depending on it waits; every further
operator restart starts the clock again.
With a one-minute period the same repro recovered 2m02s after the
operator took its lease, against no recovery within 4m26s with the
default on the same cluster. A resync replays the cache and, with
providers in sync, ends in "No changes detected" and an unchanged
condition, so it writes nothing to the API server; the operator logged
about 15 lines a minute and used 2m CPU and 26Mi of memory.
The upstream chart has no value for the flag, so a package patch that
make update reapplies adds one, and a bats test holds the rendered
argument.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
c873a99704 |
[Backport release-1.7] fix(vm-disk)!: refuse a golden-image clone CDI would leave unbound (#5022)
# Description Backport of #4796 to `release-1.7`. |
||
|
myasnikovdaniil
|
89076c5f03 |
[Backport release-1.7] fix(platform)!: refuse a networking variant change on a live cluster (#5021)
# Description Backport of #4790 to `release-1.7`. |
||
|
myasnikovdaniil
|
f39982a30f |
[Backport release-1.7] fix(backups): keep the backupstrategy-controller Helm release under the Secret size limit (#5023)
# Description Backport of #4941 to `release-1.7`. |
||
|
myasnikovdaniil
|
a115355f45 |
[Backport release-1.7] fix(linstor): mark volumes restored from a snapshot DRBD-initialized (#5020)
# Description Backport of #4866 to `release-1.7`. |
||
|
myasnikovdaniil
|
2ccca66791 |
[Backport release-1.7] fix(kafka): pass the container limits to the Kafka JVMs (#5019)
# Description Backport of #4865 to `release-1.7`. |
||
|
myasnikovdaniil
|
54b3c78f76 |
[Backport release-1.7] fix(multus): drop the CPU limit and make resources configurable (#5018)
# Description Backport of #4830 to `release-1.7`. |
||
|
Myasnikov Daniil
|
b6ed4bb0cd |
test(hack): keep every chart's stored Helm release within budget
A chart can grow past the 1 MiB Helm release Secret without any test
noticing: it renders, lints and installs, and only the upgrade whose
manifest is large enough fails, on a live cluster, with a release that
stays failed. backupstrategy-controller sat under 1 KB from the limit
before it crossed it.
Estimate the stored size the way Helm encodes it (release JSON with the
chart's templates and files, gzip, base64) and fail any chart under
packages/ above 90 percent of the limit, plus the largest revision of
backupstrategy-controller with every default Strategy rendered. The
estimate counts files a .helmignore names, because the production path
stores them.
A size measured without its manifest is a lower bound that passes for
the wrong reason, so every chart must render offline except an explicit
list of charts that need cluster state, and the worst-case render must
succeed and contain the default strategies before it is measured.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
22d7f4d04c |
test(backups): fail on a strategy CRD that inlines a pod schema
The release-size budget only reports that the backupstrategy-controller
release grew too large, not which field did it, and three strategies
once inlined the upstream pod schema before anyone noticed. Check the
generated strategy CRDs for an inlined PodSpec and name the property
path, with the markers that fix it.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
6c393452a1 |
fix(backups): store the Altinity, Job and FoundationDB pod templates schemaless
backupstrategy-controller renders its strategy CRDs from templates, so
Helm stores each of them twice in every release revision, and a
revision is one Secret capped at 1 MiB. The Altinity, Job and
FoundationDB CRDs each inline the full upstream PodTemplateSpec schema,
about 600 KB apiece; once the default Strategy CRs render as well, the
stored release no longer fits and the forced upgrade that creates them
fails with "Secret ... is invalid: data: Too long".
The Kafka, RabbitMQ and Redis strategies already carry their pod
template schemaless for this reason. Give the three older fields the
same markers. The Go types stay typed, the drivers decode the template
as before, and the apiserver still validates the pod when the driver
creates the workload; every description is kept, and the largest
revision drops to about a quarter of the limit. The rationale moves
to the package doc, so the six field docs only tell the reader what
schemaless means for them.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
2c620a7f44 |
[Backport release-1.7] fix(kubernetes): keep NFS publish from hanging on a dead server (#5017)
# Description Backport of #4791 to `release-1.7`. |
||
|
myasnikovdaniil
|
c2bd2227bc |
[Backport release-1.7] fix(vm-disk): name the importer error in the disk's WorkloadsReady message (#5016)
# Description Backport of #4815 to `release-1.7`. |
||
|
Aleksei Sviridkin
|
c4cc45da4a |
fix(vm-disk)!: refuse a golden-image clone CDI would leave unbound
When CDI clones a golden image into a StorageClass other than the one
the golden PVC lives in, its csi-clone path, and its snapshot path once
it gets a VolumeSnapshotClass of the provisioner, check the two classes
for a common driver, a matching volume mode and room to expand, and
never compare the class names. A target class of the same provisioner
therefore gets no host-assisted fallback and the clone is left to the
CSI driver. LINSTOR creates that clone next to the source data, so on a
target class that does not allow remote volume access from every node,
such as a node-local one, the temporary clone PVC stays unbound with
nothing surfaced to the user.
Refuse such a disk at render time, mirroring the planner's inputs: the
CDI override or the target StorageProfile's strategy, the snapshot class
CDI would pick, the golden's driver as its PV records it, the target's
provisioner, the volume modes and the target's expansion setting, and
refuse only a LINSTOR target whose remote access policy, parsed the way
linstor-csi parses it, is not plainly "from anywhere". A snapshot class
both StorageProfiles name but that does not exist is refused as well,
because CDI then fails to plan the clone instead of falling back. An
empty storageClass is resolved the way CDI resolves it for a kubevirt
DataVolume. A disk whose DataVolume or PVC already exists is exempt,
since CDI can garbage-collect a completed DataVolume. Combinations that
fall back to a host-assisted copy, targets reachable from any node, and
anything else the lookups cannot find still render.
This changes behaviour for a new disk in a LINSTOR class without remote
access from every node, and for one whose StorageProfiles name a
snapshot class that does not exist: both rendered before and are refused
now. The way out is the golden image's class, or the copy clone
strategy.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
f0a2855dc6 |
fix(platform)!: refuse a networking variant change on a live cluster
Changing the variant of the cozystack.cozystack-platform Package on a
running cluster changes bundles.system.variant and can re-render the
cozystack.networking Package with another variant. The operator then
deletes every HelmRelease of that Package the new variant does not
install, so isp-full -> isp-hosted removes Cilium and Kube-OVN and the
cluster loses its CNI; isp-full -> isp-slim removes Kube-OVN.
The platform render now looks up the live Package and fails when its
variant differs from the one about to be rendered, unless
networking.acknowledgeVariantChange names the new variant. The
acknowledgement is a variant name rather than a boolean so that a value
left behind after one switch does not cover a later one. The check is
inert where lookup returns nothing (helm template, client-side dry-run,
first install) and skipped when cozystack.networking is in
bundles.disabledPackages, since the platform then does not render it.
A live variant set by hand, such as cilium-kilo, is pointed at
bundles.disabledPackages rather than at the acknowledgement: the
acknowledged render would write the bundle's variant over it.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
f9550b71cc |
fix(linstor): mark volumes restored from a snapshot DRBD-initialized
linstor-csi restores a snapshot on one node and autoplaces the other
replicas. In LINSTOR v1.33.3 the restored volume definition has no
DRBD_INITIALIZED flag, so the initial UpToDate race is still open, and
Linstor/Drbd/InitialUptoDateOn, copied from the source volume through
the snapshot, names the node that won it for the source. When a
replica is autoplaced on that node, the empty new volume wins the
race: it gets a random current UUID and a fresh filesystem, and goes
StandAlone against the restored replica. A pod attached there sees an
empty volume.
Backport the upstream fix, released in LINSTOR v1.34.0, which sets the
flag on a restore unless the resource definition asks for new metadata
through ForceInitialSyncPermanent, as a backup does. Nothing wins the
race any more, and an autoplaced replica resyncs from the restored one
like any replica added to an existing volume.
Link: https://github.com/LINBIT/linstor-server/commit/9d8cdf5118338fc72a34bb99799e452e031ea8f6
Closes: #4862
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
fc38a540c0 |
fix(kafka): skip the OpenAPI patch lookup in the migration hook
The pre-install/pre-upgrade hook applies the node pools and the drain
KafkaRebalance with kubectl, and its container has no resources of its
own, so in a tenant with resource quotas it runs under the LimitRange
default of 128Mi. When it applies an object that already exists,
kubectl downloads the cluster's whole OpenAPI v2 document only to
decide that a custom resource takes a JSON merge patch: with the 374
CRDs of an e2e cluster that peaks at about 300Mi, so every upgrade of a
Kafka in such a tenant OOM-kills the hook and fails with "pre-upgrade
hooks failed".
--openapi-patch=false skips that lookup and builds the same three-way
merge patch from the last-applied annotation, peaking at about 35Mi on
the same cluster. A server-side apply would be as small but changes
what gets removed: the "kafka" pool is created with "kubectl create",
whose manager keeps co-owning every field it set, so a later
server-side apply leaves those fields in place when the chart drops
them. A larger memory limit would charge every upgrade against the
tenant quota instead.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
f6f5c7ab75 |
fix(kafka): pass the container limits to the Strimzi-managed JVMs
Strimzi sizes a broker heap by asking the image's JDK 17 what heap it
would pick at MaxRAMPercentage=50, capped at 5 GiB.
On kernels built without CONFIG_MEMCG_V1 that JDK cannot see the limit
and reports host RAM, so every broker and controller starts with up to
a 5 GiB heap inside a 1Gi limit and is OOM-killed. The topic and user
operators run the same JDK with percentage-based defaults, and the
drain-time Cruise Control takes 75% of host RAM for its heap.
Each of these containers now gets JAVA_TOOL_OPTIONS with -XX:MaxRAM set
to its memory limit, plus -XX:ActiveProcessorCount where the chart sets
a CPU limit. The entity operators and Cruise Control have none in the
chart, so a LimitRange default still leaves them counting host CPUs:
Strimzi template env takes no resourceFieldRef. Every java in the
container reads JAVA_TOOL_OPTIONS, Strimzi's heap probe included, so
Strimzi's own heap rules and Kafka's G1 defaults apply to the real
memory limit. jvmOptions would not do: Strimzi turns -XX options into
KAFKA_JVM_PERFORMANCE_OPTS, and kafka-run-class.sh skips its G1 settings
when that is set. Each JVM prints a "Picked up JAVA_TOOL_OPTIONS" line
on start. Existing clusters roll once on upgrade, because the pod
environment changes.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
df20607516 |
fix(kafka-operator): size the operator JVM from its container limits
The JDK 17 in the Strimzi 0.45.1-rc1 operator image decides whether to
read the container limits by looking at /proc/cgroups. Kernels 6.12 and
newer built without CONFIG_MEMCG_V1 no longer list the memory controller
there, so the JVM turns container support off and takes host RAM and
host CPUs as its budget: on a 128 GiB node it picks G1 with a 25 GiB max
heap and a 2 GiB initial heap inside a 512Mi limit, and the kernel
OOM-kills the operator under load. A restart at the wrong moment leaves
a freshly created StrimziPodSet without pods for up to the five-minute
informer resync, which is how a KRaft controller pool ends up never
starting within an e2e timeout.
The limits now reach the JVM through the downward API as -XX:MaxRAM and
-XX:ActiveProcessorCount, so they follow any later change to the
container resources. In a test cluster the operator then picks Serial
GC and a heap of about 100 MiB. The upstream JDK fix is JDK-8347811,
backported to 17.0.20 and 21.0.10.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
363c8426ba |
feat(platform): pass multus resources from networking.multus.resources
The multus chart takes its resources from values, but the platform
renders the cozystack.multus Package with stageCniPlugins alone, and an
edit to that child Package is reverted by the platform reconcile. A
running cluster therefore had no way to raise the multus limits.
networking.multus.resources is passed to the Package only when set, so
an unset key leaves the chart default in charge. A value that is not a
map stops the render here; the multus chart keeps refusing a render
without a memory limit or request.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
7c0c6f7e71 |
fix(multus): drop the CPU limit and make resources configurable
kube-multus serves the CNI call for every pod sandbox on its node, and
its 100m CPU limit was hardcoded in the patched upstream template.
After a CNI outage a node had ~600 dead sandboxes and 220 pods waiting:
multus was throttled in every CFS period, calls outlived kubelet's
two-minute CRI deadline, each retry grew the queue, and the node never
recovered. With the CPU limit lifted the 1Gi memory limit OOM-looped;
2 CPU and 4Gi cleared the backlog in about three minutes.
Resources now come from values, defaulting to the same requests, no
CPU limit and a 4Gi memory limit. The memory limit stays so the pod
cgroup keeps memory.max and remains outside the Talos OOM handler's
candidate set.
Since an explicit null in a values override could drop either memory
key, the render now fails without limits.memory or requests.memory.
Without the request it defaults to the limit, and a node with less
allocatable memory than that is left without multus.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
fa88bd1c49 |
fix(kubernetes): keep NFS publish from hanging on a dead server
The kubevirt-csi node plugin mounts the NFS export root into a temp
dir to create /data and migrate old files. That mount was hard, so
with the server gone the mkdir, readdir and rename calls on it never
returned. kubelet gives up after 2m and retries, and every retry
blocked the same way, piling up goroutines and nfs-init-* dirs.
Mount the temp root soft, so its RPCs fail once the connection to the
server is declared dead instead of blocking forever; it only does
metadata work that a retried publish redoes. retry=0 keeps mount.nfs
from retrying a failed temp mount for 2m. timeo and retrans keep their
defaults, because the first mount of a server sets them for the
connection the pods' mounts of it share. The volume mount the pod gets
stays hard, since a soft mount can report a failed write the server
applied.
Over NFSv4 that soft failure takes minutes, and a stat of a target
that is already hard-mounted can block for good, so a publish for a
target path that already has one running now returns Aborted instead
of starting another.
A soft stat can fail with EIO on a file /data already holds, and a
rename issued once the server is back would replace it, so the
migration now fails the publish on any stat error other than a
missing destination. It checks the destination with lstat, so a
user's symlink in /data that loops or dangles counts as present
instead of failing every publish or being replaced by the rename.
The skip it makes for a destination already in /data leaves the root
entry where the pod cannot see it, so it is now logged as a warning.
/data is created with a plain mkdir, since with the server gone
MkdirAll fails its stat on the mount point and then reports the local
temp dir as already existing instead of the EIO. The temp mount
errors now name the volume, like the migration errors do.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
4323363870 |
test(vm-disk): pin the importer error against a real stuck DataVolume
The table test builds its Running conditions by hand, so it can only
confirm the shape it already assumes. A DataVolume read off a cluster
where the importer had failed on a 404 for days carries a Bound
condition with the same error, which the message must not pick up, and
proves the error sits where the controller looks for it.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
095d9a189f |
fix(vm-disk): name the importer error in the disk's WorkloadsReady message
A disk whose HTTP source answers 404 or does not resolve never leaves
ImportInProgress: CDI fails an import only on a fatal error and restarts
the importer pod otherwise, so the condition only ever said the disk was
still importing. The reason it will not finish was visible only in the
importer logs and on the DataVolume, whose Running condition carries the
error through the back-off between restarts.
Appending that message lets the user see why the disk is stuck from the
application status and the console, which already shows the
WorkloadsReady message for a disk.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
37c82f301c |
[Backport release-1.7] fix(fluxcd): close the flux-plunger review follow-ups for the hook, drain and Warnings (#4996)
# Description Backport of #4914 to `release-1.7`. |
||
|
myasnikovdaniil
|
a739d04d2e |
[Backport release-1.7] fix(monitoring): send alerts to every Alertmanager replica (#4943)
## What this PR does
Backport of #4889 to `release-1.7`. The branch was cut on October 12,
two days before #4889 merged, so its vmalert still sends to the shared
Service and an upgrade from 1.6.5 to 1.7.0 would bring the bug back. The
commit is cherry-picked from
|
||
|
myasnikovdaniil
|
3e4295b4a6 |
[Backport release-1.7] fix(backups): gate every default Strategy CR on its CRD (#4935)
# Description Backport of #4929 to `release-1.7`. |
||
|
Alexey Artamonov
|
a8e491646a |
fix(fluxcd): keep the hook waiting on a release it cannot read
The pre-delete hook read a release's state once, so a single failed
read was taken as the release being handled: its fence came off before
helm-controller's uninstall, and the report counted it as released
cleanly. A state read is now retried within the release's share of the
window, a release whose state still cannot be read keeps its fence for
the wait like one helm-controller has not handled, and the end of the
wait reads each again before naming what the release was left to.
A release still suspended after the sweep is named as suspended but not
by flux-plunger, which also holds when no manager is recorded for the
field at all.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
ccfc9d752d |
docs(fluxcd): correct flux-plunger's recovery order and name its limits
The README gave the recovery steps in the order flux-plunger used before
the processed-version annotation moved ahead of the Secret delete. An
operator who finds the annotation on a release whose revision Secret is
still there would read that state as impossible, when it is what a
failed delete leaves and the next pass retries.
Two boundaries were left unsaid. The pre-delete hook stops, waits for and
lists again only for the flux-plunger of its own release; another one,
under a second release name or with changed labels or out of cluster,
keeps the fences it sets and gives them up itself. And a backup restore
that starts while flux-plunger holds its own suspension does not take
the field over, since the restore steps write it only when it differs,
so the end of the recovery unsuspends the release under the restore.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
683e80ba0c |
fix(fluxcd): give each release its own share of the hook's retries
The pre-delete hook's retry budget was one window spent in listing
order, so a release that could never be read or unfenced, listed
first, took all of it and every release after it got a single attempt.
The same transient failure then released a fence or not depending on
where its release sorted. Each release now gets its own share of the
window, capped at the window's end, and the steps that cover every
release keep the whole of it.
Two comments said more than holds. The retry window is measured from
the script's start, not the Job's, so scheduling and the image pull come
out of its one-minute reserve rather than being covered by it. And the
five-minute conflict retry inside kubectl scale never runs here, since
without a precondition it sends a merge patch with no resourceVersion
to conflict on.
helm-controller's finalizer, which the hook reads to tell whether a
release being deleted is still held, is now tied to the constant the
controller uses, as the fence already was.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
8416aea08b |
fix(fluxcd): let helm-controller uninstall before the hook unfences
The pre-delete hook lifted flux-plunger's suspension of a release being
deleted and removed its fence in the same step. helm-controller runs the
uninstall of a release being deleted once it is no longer suspended, but
when its own finalizer is already gone the fence is all that keeps the
object, so removing the fence at once deleted the release before
helm-controller could reconcile it, without an uninstall. The hook now
keeps such a fence while helm-controller handles the generation the
lifted suspension wrote, until shortly before its retry window closes,
and names a release it still had to let go of as before.
A release being deleted under a suspension another manager also holds
was reported as cleanly released, although helm-controller skips the
uninstall of a suspended release; it is now named as deleted without
one and counted with the others.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
18c5acae7b |
fix(fluxcd): warn about a foreign suspension only on a stuck release
flux-plunger raised its SuspensionNotOwned and TerminatingWhileSuspended
Warnings on a suspension it does not own when the release carried its
processed-version annotation. That annotation is never removed, so
every later suspension of such a release, by a migration, a backup
restore or an operator, drew a Warning: one telling the reader to
resume it, which mid-restore is the one thing nobody wants, and one
calling a migration's deliberate suspend-then-delete an accident. Both
also missed the leftover they are for when the first recovery stopped
before writing the annotation.
A suspension an earlier flux-plunger left mid-recovery freezes the
release in the failure recovery started from, since helm-controller
rewrites no condition of a suspended release. Both Warnings now key on
that state instead, and the README's candidate query with them.
Two comments are qualified where they claimed more than holds: a stop
that overran the manager's grace period can still let a reconcile set
a fence after the drain's last listing, and a process stopped before
controller-runtime closes Elected holds the Lease yet does not drain.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
1fce03594a |
fix(fluxcd): make flux-plunger's shutdown drain report what it left
The drain's only report was an Info line, so an alert on errors never
saw fences left behind; it named no release, and printed held: -1 when
no listing had answered. A flux-plunger that never led logged that it
had released every fence next to the line saying it drained nothing.
Fences left behind are now an error naming each release, an unanswered
listing says so, and a process that never led says only that.
The drain writes its Events itself, since the manager's recorder stops
with the manager, but it created one Event object per pass for a
release it revisits every second, and each write had a deadline of its
own outside the drain's. Repeats now fold into one counted Event, as the
manager's recorder folds them, and every write is bounded by the drain.
Tests now hold the outcome line, the retry of a release whose fence
cannot be given up yet, and the pause between passes, none of which any
suite noticed when removed.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
56dd6c2e41 |
fix(fluxcd): refuse a flux-plunger drain deadline that leaves no time
A drain deadline of zero or less is already spent when the drain
starts, so flux-plunger would exit holding every uninstall fence
without listing one. The chart could produce one: its 40-second floor
was a literal, so raising the manager's stop bound or the margin before
SIGKILL, which the drain subtraction is already tied to, would leave the
floor rendering a non-positive deadline. The binary now refuses such a
flag at startup, before it reaches for a cluster, and the chart's floor
and the timings the README and the values comment state are held to
the same constants.
A whole number of a million seconds or more printed in exponent form
and was refused as a fraction; the range is now checked first, so it
gets the range message.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
587ec5913f |
[Backport release-1.7] fix(fluxcd): keep Helm uninstall working during flux-plunger recovery (#4995)
# Description Backport of #4293 to `release-1.7`. |
||
|
Alexey Artamonov
|
021040795b |
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
flux-plunger recovers a "has no deployed releases" HelmRelease by
briefly suspending it, deleting the stale storage Secret and
unsuspending. helm-controller v1.5.x skips the uninstall and drops its
finalizer when it handles a delete on a suspended HelmRelease, and it
wakes on the same delete event flux-plunger does, so a delete inside
that window orphaned the release's DataVolume, PVCs, RBAC and storage
Secret. Lifting the suspension on the delete event cannot win that race
reliably.
Hold the release with a flux-plunger finalizer while it is suspended.
The finalizer is applied together with the suspension in one server-side
apply. On a delete flux-plunger lifts its suspension and keeps the
finalizer, helm-controller reconciles the unsuspended generation and
runs the uninstall, and flux-plunger removes the finalizer once
helm-controller's own is gone and that generation is observed. This
holds whichever controller handles the delete first. A completed
recovery lifts the suspension and the finalizer together, and a
terminating release never begins recovery.
Decide ownership of a suspension by which field manager owns
spec.suspend in metadata.managedFields rather than by an annotation. The
apiserver transfers the field away the moment an operator, cozyhr resume
or a backup controller writes a different value to it, so a later
suspension is never misread as flux-plunger's, and one set through the
shared flux-client-side-apply manager (platform migrations, the tenant
delete hook, a manual kubectl apply) is never claimed. A release deleted
under another manager's suspension is left to that manager's decision.
Acquire applies without forcing first and, on a conflict, re-fetches and
forces only if spec.suspend is still false: a backup controller owns the
field at false through a dynamic-client Update, which a non-forced apply
can never take, while true is an active suspension to leave alone. Every
apply is preconditioned on the resourceVersion and uid of the read that
decided on it. Without the resourceVersion, the unforced apply of true
over a foreign true is accepted as shared ownership, a window the
optimistic-locked merge patch this replaces had closed, and lifting the
suspension and the finalizer from a release read as live would delete it
without an uninstall if helm-controller had meanwhile handled a delete.
Without the uid, an apply to a release that has gone would create an
empty one. The apply carries only spec.suspend and the finalizer, so it
cannot leak the required spec.interval.
Read the release, every re-check and the storage Secret list from the
apiserver rather than the informer cache. A lagging cache would let the
processed-version guard delete the revision below the one just
reclaimed, and the ClusterRole grants secrets get and list but not the
watch an informer needs.
Emit a Warning when flux-plunger holds a suspension it failed to clear,
and Normal events when it leaves an external suspension alone or defers
a recovery. An external suspension on a release flux-plunger has
recovered before is a Warning instead: it is most likely one an earlier
flux-plunger left under the shared manager, which nothing lifts unless
an operator does. A release deleted under another manager's suspension
is a Normal event when that was a migration's choice, and a Warning when
the release carries flux-plunger's processed-version annotation, since
then the skipped uninstall is an accident.
Keep a single writer with leader election. A second process able to act
under the same field manager could clear a suspension the first is still
recovering, and no rollout strategy prevents one: the next pod starts as
soon as the old one is terminating, and an eviction or a node drain
replaces a pod the same way. The old pod keeps its Lease, unreleased,
through its stop and drain, and the Lease outlasts both, so the next pod
acts only after the old one is done; the price is that a new pod waits
for the Lease to run out. The strategy sets maxSurge 0 with
maxUnavailable 1 and stays RollingUpdate because existing installs carry
the apiserver-defaulted rollingUpdate block and helm-controller applies
server-side, where type Recreate over it is rejected on upgrade.
Give the finalizer a way out when flux-plunger itself goes. Once the
manager has stopped, within a bounded time, and with it every reconcile,
flux-plunger drains its finalizers under a deadline that, with the stop,
fits inside the pod's termination grace period. A stop that overran its
bound is drained too and still reported, and only a pod that led drains.
The drain writes its events straight to the apiserver, since the
manager's recorder stops with the manager. A recovery in progress is
aborted, the suspension and the finalizer lifted together for the next
flux-plunger to start again, and a release being deleted has the
suspension lifted and keeps the finalizer until helm-controller has run
the uninstall. A release helm-controller does not handle in time keeps
it. The waiting lives in the controller, where the envtest suites hold
it against a real apiserver's generations and finalizers, rather than in
a shell script against a model of them.
A pre-delete hook on the chart is the backstop for what the drain could
not finish, so uninstalling the component never leaves a release that
tenant teardown waits on forever. It stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and finalizer without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, with a JSON patch preconditioned on the resourceVersion and
uid it read: kubectl apply --server-side on a release last written by a
client-side apply moves that apply's fields to the applying manager and
strips every one the body leaves out. The Job's deadline sits under the
300s helm-controller gives a pre-delete hook by default. A Deployment
that cannot be stopped or a listing that never succeeds ends the hook
before any finalizer is touched, and a finalizer it cannot remove ends
it after the sweep; either way it exits non-zero: a failed pre-delete
hook leaves the release untouched and helm-controller retries the
uninstall, hook included, from a state it can still converge from, where
exiting 0 would remove the component and strand the finalizer. A
flux-plunger pod still present after its grace period, typically one
stuck Terminating on a NotReady node, is only reported, since failing on
it would fail every retry until the node returns; because it may still
be setting finalizers, they are listed again after the sweep and any
found fails the hook. The Job runs once, since helm-controller's retry
is the one that starts afresh, and is kept after success so that its
report stays readable; helm.sh/hook-output-log-policy would make
helm-controller call a nil output writer (fluxcd/helm-controller#1562).
A delete still held after ten minutes raises a Warning, and the held
state is re-checked on a timer instead of only on helm-controller's next
status write. The chart README documents the finalizer and how to clear
it.
Run the envtest suites on every pull request through a new make target
that fetches a kube-apiserver and etcd with a pinned setup-envtest and
takes the HelmRelease CRD from the in-tree Flux manifests: the unit
tests drive a fake client, and everything the finalizer rests on (apply
preconditions, the finalizer set-merge, real managedFields) only holds
against a real apiserver. CI caches those binaries, and the target uses
them without asking the release index, so a pull request does not depend
on their sources being reachable; the control plane gets a longer start
timeout because it starts beside the other test goals.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Yan Bondarenko
|
5033d53410 |
fix(monitoring): send alerts to every Alertmanager replica
vmalert had a single notifier, the headless vmalertmanager-alertmanager
Service. It resolves to all three pods, but each alert reached one replica
at a time and moved to another when vmalert reconnected. On clusters running
1.6.4 all active alerts sat on one replica while the other two had none, and
on one of them the alert count of each replica changed about 170 times a
day. The Alertmanager cluster deduplicates notifications only when every
replica receives every alert. The replica an alert left let it expire and
sent a resolved notification, and the replica it reached sent a new firing
one.
vmalert now lists each Alertmanager pod as its own notifier, using the
<name>-<i>.<service>.<namespace>.svc address the operator itself gives
StatefulSet pods. The replica count comes from one helper that also sets
spec.replicaCount of the VMAlertmanager, so the two cannot drift.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
dee1eef559 |
test(backups): fail CI on any default Strategy CR rendered without its CRD gate
The CRD gate has now been missed twice: MongoDB shipped without it, and
Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only
checks the templates a case lists, so a new strategy-*-default.yaml
that forgets the gate stays green until an upgrade wedges.
Render the chart with the bucket gate open and no CRD served, and fail
on any strategy.backups.cozystack.io object from any template. Also
check that every rendered Strategy CR gates on the CRD this chart ships
for its kind, because a gate naming the wrong CRD passes the first
check and keeps the CR out of every live render.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
a29fd4814c |
fix(backups): gate every default Strategy CR on its CRD
The chart ships its strategy CRDs as ordinary templates, and Helm maps
every document of a release through the RESTMapper before it applies
any. A default Strategy CR gated only on the bucket name therefore
fails Build() with "no matches for kind" whenever the bucket name
resolves before the CRD is served, the release applies nothing, the
CRDs are never created, and every retry renders the same manifest.
That happens on the upgrade that first ships a kind: the Kafka,
Rabbitmq and Redis CRs leave a v1.6 cluster's HelmRelease stuck in
UpgradeFailed on the way to v1.7. It also happens on the first render
of an external-S3 install (provisionBucket=false), where the bucket
name comes from values and every older default CR renders next to its
absent CRD.
MongoDB already gates on crdEstablished for this reason. Give every
default Strategy the same gate. On an installed cluster the CRDs are
established and the render is unchanged; otherwise the first revision
creates the CRDs and skips the CRs, and DefaultObjectsGate, which reads
the cozy-default BackupClass routes, forces the follow-up upgrade that
renders them once the kinds are served.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
38d6ac7efb |
feat(redis): expose min-replicas-to-write and min-replicas-max-lag (#4848)
## What this PR does Adds two optional values to the Redis app, `minReplicasToWrite` and `minReplicasMaxLag`, rendered as `min-replicas-to-write` and `min-replicas-max-lag` in the RedisFailover `spec.redis.customConfig`. Both are unset by default, so the rendered config and behaviour are unchanged for existing instances. Replication is asynchronous, so a master that is isolated from its replicas keeps accepting writes until Sentinel fails over, and those writes are lost. With `minReplicasToWrite: 1` the isolated master stops accepting them after `minReplicasMaxLag` seconds, which bounds the loss. Trade-off: with the default `replicas: 2` there is a single replica, so the master also refuses writes whenever that replica is down (node drain, restart, upgrade). Users who turn this on accept lower write availability for a bounded loss window. Includes helm-unittest cases and the regenerated schema, README, Go types and ApplicationDefinition. ### Screenshots ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/68 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(redis): add optional minReplicasToWrite and minReplicasMaxLag values to bound writes lost on failover; unset by default ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional Redis settings to require a minimum number of replicas to acknowledge writes and to set the maximum age of those acknowledgements. * Invalid settings are rejected: the required replica count must be nonnegative and below the configured replica count, and the acknowledgement age must be at least 1 when a replica requirement is set. Replication remains asynchronous, so these settings do not eliminate potential write loss during failover. * **Documentation** * Documented the settings, defaults, validation, and behavior in Redis configuration guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai -->v1.7.0-rc.1 |
||
|
myasnikovdaniil
|
ebceef4105 |
ci(retention): sweep packages named by the published nightly tree (#4826)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The Artifact Retention workflow has failed every day since mid-August, and it never pruned anything. It lists the org's container packages with `GET /orgs/cozystack/packages?package_type=container`, and that endpoint answers `400 Invalid argument` to the app installation token the job uses. Others hit the same in actions/github-script#449 and cli/cli#9606. An old `|| true` used to hide it. This is item A19 in #3278. The per-package endpoints accept the same token: a probe run confirmed it, and the dry-run below read every package with it. So the job no longer lists the org. The new `hack/retention-packages.sh` takes the package names from the published `cozystack-packages:nightly` tree, reading refs through the shared `hack/lib/image-refs.sh`. It also adds `cozy-installer` and `cozystack-nocloud`, which the nightly publishes outside the tree. The committed tree can't be the source: it misses images whose digest only the build pins, and those still get dated nightly tags every day. Failures now turn the run red instead of passing silently: - if the name list is empty, the nightly sweep is skipped and the run fails, but abandoned promotion candidates are still handled, because their protection doesn't depend on that list - if a package's versions can't be read, that package is skipped and the run fails at the end - if a DELETE is refused, `hack/retention-delete.sh` logs an error with the HTTP status, goes on with the remaining versions and fails the run (before, this was only a warning) - if names resolve but none of the packages is linked to this repository, nothing gets swept, so the run fails too, after the promotion sweep has run (that sweep itself is skipped only when candidate protection can't be resolved) Promotion-candidate protection itself is unchanged. The contract test that pinned the old org listing now pins the new behaviour, and both new scripts have bats tests. Known gap: a component that disappears from the nightly tree drops off the list, so its old dated nightlies are never pruned. Today none of the packages outside the list has more dated nightlies than the keep window. Dry-run on this branch is green: https://github.com/cozystack/cozystack/actions/runs/38075159900. It resolves 59 packages, skips `cozystack/cozystack-ui` with the warning, and would delete 418 old nightly versions plus one abandoned promotion candidate. Deleting needs a different token. The app token lists versions, but a one-version apply run (https://github.com/cozystack/cozystack/actions/runs/37939905343) got `404 Package not found` on DELETE for a version it had just listed. The same DELETE with this repository's `GITHUB_TOKEN` and the job's `packages: write` worked in a second run (https://github.com/cozystack/cozystack/actions/runs/37941084456), with no change to package settings. So `hack/retention-delete.sh` now deletes with `GITHUB_TOKEN`, and listing stays on the app token. That delete was shown on a package linked to this repository. One of the swept packages, `cozystack/cozystack-ui`, is linked to another repository, and I didn't try deleting there. Such a package is skipped before its versions are read and named in a warning, so it can't turn every apply run red. The follow-up for it is in #3278. `RETENTION_APPLY` stays off, so the scheduled run is a green dry-run until someone turns it on. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> N/A ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ci(retention): the GHCR retention workflow takes package names from the published nightly tree instead of the org package listing, which rejects the app token, and fails the run whenever names, versions or deletions cannot be resolved. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Nightly package retention now targets packages referenced by the published nightly tree and keeps the 14 newest eligible versions per package. * Promotion candidates older than 30 days are eligible for pruning only when their digests are not protected by qualifying release pull requests or branch tips. * **Bug Fixes** * Packages linked to another repository are skipped, and sweep failures are reported without preventing other eligible packages from being processed. * **Documentation** * Clarified which nightly versions and promotion candidates qualify for retention. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
34488db766 |
ci(e2e): never tear down the Proxmox tenant on the runner's default context (#4807)
## What this PR does The Proxmox e2e teardown could delete from the wrong cluster. It runs under `always()` and got its kubeconfig only from `GITHUB_ENV`, which the "Stage credentials" step writes. If a step before that fails, for example a missing stand variable, `KUBECONFIG` is never set. The three `kubectl delete` calls then go to whatever cluster the self-hosted runner's own kubeconfig selects. Every teardown call now passes the staged file with `--kubeconfig`. Stage fails unless that file's current context resolves to a server, and prints kubectl's own error when it does not. Before, an empty `COZY_KUBECONFIG` secret let client-go fall back to its defaults, in-cluster credentials among them. The suite step gets the staged path in its own `env`. The `GITHUB_ENV` export is gone, so no step in this workflow picks its cluster up implicitly. Tear down decides whether to skip from how Stage ended, not from the file. If Stage did not succeed, the suite never ran and there is no tenant to remove, so the step prints a notice and exits 0. If Stage succeeded and the file no longer resolves, the step fails with kubectl's error, because the tenant and its VMs may still be there. A side effect: a run whose Stage fails no longer cleans up a tenant left by an earlier `keep_tenant` run. The next run that passes Stage without `keep_tenant` removes it. The Stage and Tear down scripts now have unit tests in `hack/proxmox-teardown-contract.bats`, run against a stub `kubectl`. I also ran both against a real kubectl v1.37.1 for every Stage outcome. The files were missing, empty, garbage YAML, without a current context, with a dangling context, without a server, and valid. One more case was a runner without kubectl. The deletes also no longer stop at the first failure. Under `set -e`, a failed first delete skipped the tenant delete, and the tenant's VMs stayed on the hypervisor. Now each delete runs, and the step fails at the end if any of them failed. With a recording `kubectl` that fails one chosen delete, all three ran in every case and the step exited 1. With the change reverted, a failing first delete was the only one that ran. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note ci(e2e): the Proxmox e2e teardown always uses the staged kubeconfig, so it can no longer delete from the runner's default cluster. It skips only when Stage did not succeed and fails when the kubeconfig breaks after Stage. A failed delete no longer skips the tenant delete. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Improved reliability of automated Proxmox test setup and cleanup. Tests now stop setup when Kubernetes configuration is unavailable, skip cleanup when setup fails, and attempt all resource deletions before reporting cleanup failures. * **Tests** * Added checks for configuration validation, cleanup behavior, and handling of failed deletion attempts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
2db18a0c86 |
test(e2e): record ZFS datasets and LINSTOR volumes in cozyreport (#4657)
## What this PR does The kubernetes suites' LINSTOR cleanup check now compares dataset names per node, and prints the leftover names when it fails (#4651). This PR puts the rest of what a triager needs into `cozyreport.tgz` on every run, green or red: the size of each dataset, its `origin` and `creation`, the snapshots, the pool's pending frees, and the LINSTOR volume list. What `hack/cozyreport.sh` now collects: - `linstor/zfs-pools.txt`: the zpool behind each ZFS storage pool, per node, taken from `StorDriver/StorPoolName` in `linstor --machine-readable sp l`. - `linstor/zfs/<node>.txt`: that node's satellite runs `zpool get -p freeing,leaked,allocated,free` and `zfs list -p -t all -o name,used,avail,refer,origin,creation -r` over the node's pools. - `linstor/volumes.txt`: `linstor v l`, next to the existing `resources.txt`. Sizes are exact bytes, so the sizes of leftover datasets can be summed against the pool root's `avail`. That `avail` can differ from `storage-pools.txt` without anything leaking: on a node with only thick pools, `linstor sp l` serves the free capacity cached at the satellite's last report. A dataset LINSTOR could not destroy because snapshots depend on it is renamed to `_deleted_<name>` in the same pool, so a search for leftovers has to look for that prefix as well as `pvc-`. A pool that `zpool get` or `zfs list` cannot open does not cost the listing of the other pools. `zpool get`'s message and exit status go into the listing, and `zfs list` lists the rest. A node LINSTOR names no ZFS pool for gets a note instead of a listing, because `zfs list -r` with no pool argument lists every pool on the machine. A satellite whose `spec.nodeName` cannot be read gets a note that says so. All reads go through the bounded exec reader, like the other LINSTOR reads, and the time budget in the script header counts them. The second commit changes how that reader marks a read that wrote output and then exited non-zero without hitting the timeout. It used to append `TRUNCATED ... part way through, not because the command finished`. But `kubectl exec` passes the remote command's status through, and `zfs list` over one missing pool prints a complete listing of the others and exits 1. The marker now names the exit status and says the status can't tell an error after complete output from a cut stream. This changes the marker for every LINSTOR exec read. The timeout marker stays as it was. Tested with `hack/cozytest.sh hack/cozyreport.bats`. The new tests run the controller and satellite shells against stub `linstor`, `zpool` and `zfs`, with the real `jq`. They cover a failed and an unparsable storage-pool read, a pool row with a missing, empty or blank backing name, a non-ZFS pool, a pool name with a space, a pool that `zpool` and `zfs` cannot open, a failing `zfs`, and an unreadable node name. A test goes red if any of these break: the filter that keeps non-ZFS pools out, the filter that drops a missing, empty or blank pool name, the jq exit status, the `zpool` fallthrough, the node-name note, or the reader's new wording. Refs #4653 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(e2e): cozyreport records the ZFS datasets of each node's LINSTOR pools in exact bytes, and the LINSTOR volume list, on every e2e run. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Collection reports now include LINSTOR volumes, ZFS pool capacity and usage, and per-node ZFS datasets and snapshots with exact-byte details. Dataset listings are scoped to the ZFS pools LINSTOR reports for each node. * Reports include notes or warnings when node, pool, or dataset information is missing or cannot be collected. * **Documentation** * Clarified storage report contents, artifact markers, bounded reads, and how unsupported data, incomplete results, collection limits, and per-object warnings are reported. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
95c2d85b5b |
fix(etcd): leave compaction to etcd and pass the real quota to defrag (#4873)
## What this PR does The hourly defrag CronJob ran `etcd-defrag` with its defaults, and two of them do not fit the cluster the chart creates. - `--compaction` (default `true`) compacts to the latest revision before every defrag. The EtcdCluster already compacts itself (`autoCompactionMode: periodic`, `autoCompactionRetention: 5m`), so the extra compaction either fails with `mvcc: required revision has been compacted` when nothing was written since the last auto-compaction, which is what every defrag Job log shows on our clusters, or, on a busy cluster, drops the history the retention keeps for watchers to resume from. - `--etcd-storage-quota-bytes` defaults to 2 GiB, so `dbQuotaUsage` in the defrag rule is measured against 2 GiB rather than the `quotaBackendBytes` the chart sets (95% of `size`, about 3.8 GiB for the default `4Gi`). The CronJob now passes `--compaction=false` and the chart's own quota through the existing `calculateQuotaBackendBytes` helper. The new unit tests check both arguments and that the defrag quota equals the EtcdCluster's `quotaBackendBytes` for a non-default size. Without the template change all three fail. The same templates are on `release-1.6` and the change applies there cleanly. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff is two template arguments and a unit test in `packages/extra/etcd`, with no change to values, schema, defaults or package names, so nothing in the trigger map applies. ### Release note ```release-note fix(etcd): the defrag job no longer compacts on top of etcd's periodic auto-compaction, and it evaluates its 80% quota threshold against the cluster's real quotaBackendBytes instead of 2 GiB. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * The etcd defragmentation job now uses the cluster’s configured backend storage quota and skips compaction before defragmenting. This keeps the defragmentation operation aligned with the cluster’s storage settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Yasar Rahman
|
8be7fb2e66 |
feat(redis): expose min-replicas-to-write and min-replicas-max-lag
Lets an operator bound the writes lost when an isolated Redis master is failed over: the master refuses writes while fewer replicas than configured are connected and recent. Both values are unset by default, so the rendered Redis config is unchanged. Redis treats 0 in either directive as disabling the check, so the chart refuses minReplicasMaxLag 0 next to a replica requirement instead of silently turning the protection off. Negative values are refused by the schema (minimum 0), since Redis rejects them at CONFIG SET and the operator would stop healing the Sentinels of that instance. A requirement of replicas or more is refused at render: a master with replicas pods has replicas-1 replicas, so it would refuse every write for good. Assisted-by: LLM Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com> |