Three package Makefiles spelled out their docker buildx flags instead
of using $(BUILDX_ARGS) from hack/common-envs.mk: flux-plunger,
kubeovn-plunger and both dashboard images. kubeovn-plunger landed just
before the change that moved the other Makefiles onto BUILDX_ARGS,
flux-plunger was modelled on it, and dashboard kept its own flags when
that change touched it.
Those targets ignored PLATFORM, SBOM and BUILDX_EXTRA_ARGS. On an arm64
workstation the plungers came out as arm64 images, which fail on amd64
nodes with "exec format error"; CI builds on amd64 and never saw it.
Dashboard and metallb hardcoded --platform=linux/amd64 instead; for
metallb that sat next to $(BUILDX_ARGS), so setting PLATFORM gave buildx
two platforms and a multi-platform build.
All of them now build through $(BUILDX_ARGS) alone. CI sets only
BUILDER, which these targets already honoured, and runs on amd64, so
its builds are unchanged. Local builds of dashboard and metallb now
follow the host architecture like every other package: on an arm64
machine build them for an amd64 cluster with PLATFORM=linux/amd64.
A bats test dry-runs every package image target with a probe platform
and a marker in BUILDX_EXTRA_ARGS, and requires each buildx call to
carry the marker, so $(BUILDX_ARGS) is really passed, and that
platform and no other.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Talos userspace OOM handler (v1.12+) ranks cgroups with
memory_max.hasValue() ? 0.0 :
{Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current
and discards every cgroup scoring zero, so only a memory limit takes a
pod out of the victim set. Selection is decoupled from the trigger: a
tenant workload thrashing against its own limit drives node memory PSI,
and a limit-free system DaemonSet is killed for it. metallb-speaker and
linstor-satellite were the observed victims.
Set requests and limits on the node DaemonSets that had none. Requests
come from 14d peak working set measured on a 3-node dev cluster; limits
are deliberately loose, sized as a runaway backstop rather than a fitted
ceiling, because several of these scale with cluster size that a dev
cluster understates and because undersizing converts a rare
pressure-driven kill into a deterministic one. A limit costs nothing
until it is hit -- only the request reserves.
memory.max lands on the POD cgroup only when EVERY container in the pod
carries a limit, so sidecars and init containers are covered too; one
limit-free container returns the whole pod to the victim set.
Two DaemonSets are not chart-rendered and needed their own seam:
linstor-satellite via the LinstorSatelliteConfiguration podTemplate, and
virt-handler via KubeVirt customizeComponents.patches, since virt-operator
owns that DaemonSet. Note linstor-controller stays deliberately limit-free
-- it is a JVM that must not be OOM-killed mid-operation, and cluster_test
pins that.
Two things the multus manifest needed beyond the values change.
customize-deployment.patch is the source for that template, so the
limits are carried there rather than only in the generated file, which
`make update` would discard. And the reason kube-multus had no memory
limit was that this patch DELETED upstream's 50Mi rather than raising
it -- 50Mi being well under the 230Mi measured peak -- which is what
removed memory.max from the pod cgroup in the first place.
Not covered, because upstream hardcodes the containers with no resources
knob: the four frr-k8s cp-* init containers, the kube-ovn hostpath-init
and install-cni init containers, and cozy-proxy, whose vendored chart
exposes no resources value at all. hami and kilo are left alone as
optional components with no usage measurements behind them. All of these
still get a limit from the default LimitRange the operator maintains in
every system namespace.
Every limit is pinned by a helm-unittest assertion, and every assertion
was mutation-tested: changing the rendered value turns its suite red.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Bump vendored MetalLB chart from v0.15.2 to v0.16.1 and rebuild the
cozystack-built `metallb-controller` / `metallb-speaker` images at the
new version. Adopt the upstream-default BGP backend: FRR-K8s replaces
the deprecated FRR mode. Rotate the cilium-networkpolicy clusterwide
host-network port denylist to match the v0.16 listener set.
The chart introduces a new probe contract (`/healthz` and `/readyz` on a
dedicated host-local port 17472, distinct from the metrics port), so the
cozystack-built controller and speaker images must move in lockstep with
the chart. Leaving the digests on v0.15.2 would CrashLoopBackOff every
pod on main between merge and the next release-prep commit.
Upstream highlights between 0.15.2 and 0.16.1:
- 0.16.0: FRR-K8s is the new default BGP backend; legacy FRR is
deprecated.
- 0.16.0: kube-rbac-proxy replaced by native TLS; metrics are HTTPS-only
on the formerly-rbac-proxy port 9120. Default TLS minimum is now 1.3
(was 1.2); operators with TLS 1.2 scrapers override via
`metallb.tls.minVersion: VersionTLS12`.
- 0.16.0: new `tls.*` and `networkpolicies.*` value blocks (opt-in).
- 0.16.1: chart-only patch — liveness/readiness probe port is now a
configurable value (default unchanged at 17472), controller probes drop
the hardcoded `host: 127.0.0.1`, pods/ServiceMonitor/PodMonitor gain
`prometheus.io/scheme: https`, the frr-k8s subchart gains an opt-in
ServiceMonitor toggle (default off), and one CRD numeric field widens
from int32 to int64.
- 0.15.3: `ConfigurationState` CRD exposes reconcile errors per
component; chart gained NetworkPolicy support and tightened pod
SecurityContext.
The `packages/system/metallb/values.yaml` header documents the changes
operators see on upgrade, including the partial nature of the legacy-FRR
escape hatch (Helm evaluates dep `condition:` against the declaring
chart's values context, so a grandparent override from the umbrella
cannot suppress the frr-k8s subchart through that condition; the
subchart stays installed but idle when the speaker is flipped to FRR
mode).
New helm-unittest coverage pins the post-bump invariants and would have
caught the original chart/binary skew before merge: controller/speaker
image tags exactly match the chart `appVersion`, speaker is the
single-container DaemonSet with `BGP_TYPE=frr-k8s`, frr-k8s DaemonSet
and CRDs are installed, NetworkPolicies stay off by default. The
cilium-networkpolicy chart gains its first unittest suite, enumerating
every metallb v0.16 host-network port plus the unrelated etcd
kube-rbac-proxy port that shares port number 9443 (pod-IP vs
host-network).
### Release note
```release-note
chore(metallb): bump MetalLB to v0.16.1; adopt upstream-default FRR-K8s BGP backend in place of the deprecated FRR mode. Metrics endpoints move from HTTP on 7472 to HTTPS on 9120 with native TLS 1.3 by default (operators on TLS 1.2 scrapers must set `metallb.tls.minVersion: VersionTLS12`). The cilium-networkpolicy clusterwide host-network port denylist is rotated to cover the new metallb listeners (7946 memberlist, 9120, 9121, 9140, 9141, 19443). The legacy FRR mode remains available via `metallb.frrk8s.enabled=false` + `metallb.speaker.frr.enabled=true`, with the caveat that the frr-k8s subchart stays installed (idle) under that escape hatch.
```
Upgrade vendored MetalLB chart from 0.15.2 to 0.16.1 and rebuild the
cozystack-built controller / speaker images at the new version (pinned to
linux/amd64 in the Makefile to match the prior baseline and the e2e Talos
sandbox arch — without the pin, a contributor building on an arm64 host
like Apple Silicon ships single-arch arm64 images that crashloop on amd64
nodes with "exec format error"). Adopt the upstream-default BGP backend:
FRR-K8s replaces the deprecated FRR mode. Rotate the cilium-networkpolicy
clusterwide host-network port denylist to match the new metallb endpoints.
The Makefile now vendors the frr-k8s subchart instead of dropping it, and
pins the chart version so future make update invocations are deterministic.
Upstream highlights between 0.15.2 and 0.16.1:
- 0.16.0: FRR-K8s is the new default BGP backend; legacy FRR is deprecated.
- 0.16.0: new probe contract — metallb controller and speaker serve
/healthz and /readyz on a dedicated host-local health port 17472
(distinct from the metrics port). The frr-k8s controller uses its own
host-local probe port 7572. The cozystack-built image must move in
lockstep with the chart so probes resolve against the new endpoints.
- 0.16.0: kube-rbac-proxy is replaced by native TLS; metrics endpoints are
HTTPS-only, served on the formerly-rbac-proxy port 9120. Default TLS
minimum is now 1.3 (was 1.2); operators with TLS 1.2 scrapers override
via metallb.tls.minVersion: VersionTLS12.
- 0.16.0: new tls.* and networkpolicies.* value blocks (opt-in defaults).
- 0.16.1: chart-only patch on top of 0.16.0 — the liveness/readiness probe
port is now a configurable value (default unchanged at 17472) and the
controller probes drop the hardcoded host: 127.0.0.1; pods, ServiceMonitor
and PodMonitor gain prometheus.io/scheme: https for the HTTPS metrics
endpoint; the frr-k8s subchart gains an opt-in ServiceMonitor toggle
(default off); one CRD numeric field widens from int32 to int64.
- 0.15.3: ConfigurationState CRD exposes reconcile errors per component;
Helm chart gained NetworkPolicy support and tightened pod SecurityContext.
Adaptations:
- packages/system/metallb/Makefile: pin helm pull --version 0.16.1; remove
the rm -rf charts/metallb/charts/frr-k8s line so the new default BGP
backend ships with the chart; add a test: target running helm unittest;
hardcode --platform=linux/amd64 on the buildx invocations so the image
arch does not depend on the contributor's host.
- packages/system/metallb/charts/metallb/: regenerated via make update.
- packages/system/metallb/values.yaml: cozystack-built controller and
speaker image tags moved to v0.16.1 with fresh linux/amd64 digests
pushed to ghcr.io/cozystack/cozystack. The chart's new probe contract
requires the v0.16.x binary; leaving the digests on v0.15.2 would
CrashLoopBackOff every pod on main between merge and the next
release-prep commit. A header comment documents the backend default
switch, the metrics-port move (7472 HTTP → 9120 HTTPS), the new TLS 1.3
default, the new host-network ports (9140, 9141, 19443), and the partial
nature of the FRR-mode escape hatch — flipping
metallb.frrk8s.enabled=false plus metallb.speaker.frr.enabled=true
switches the speaker but does not suppress the idle frr-k8s subchart,
because Helm evaluates a dep's `condition:` against the declaring
chart's own values context and a grandparent override from the
umbrella cannot reach that resolution.
- packages/system/metallb/tests/metallb_test.yaml: new helm-unittest suite
pinning post-bump invariants — speaker is the single-container daemonset,
BGP_TYPE is frr-k8s, frr-k8s daemonset and CRDs are installed,
NetworkPolicies stay off by default, controller/speaker image tags
exactly match the chart appVersion (the test that would have caught the
chart / binary skew up-front), and the documented FRR-mode escape hatch
is pinned in both halves (speaker switches to four-container FRR layout,
frr-k8s subchart still renders idle — the latter test is the contract
that a future cleanup must update together with the values.yaml note).
- packages/system/cilium-networkpolicy/templates/networkpolicy.yaml: drop
the stale FRR-mode metrics port 7473; add every metallb v0.16
host-network listener that this policy file is responsible for —
7946 (speaker memberlist gossip, TCP/UDP; cluster peers still reach it
via the same policy's ingress fromEntities[cluster,host] clause),
9120 (speaker metrics), 9121 (legacy FRR metrics, kept defensively so
the documented escape hatch stays safe), 9140 (frr-k8s controller
metrics), 9141 (frr-k8s FRR metrics), 19443 (frr-k8s statuscleaner
webhook). Port 9443 stays in the deny list with a tightened comment:
the host-network bind belongs to the etcd kube-rbac-proxy DaemonSet in
cozy-monitoring; the metallb controller webhook collides on the same
port number but on pod-IP, out of scope for a world-deny.
- packages/system/cilium-networkpolicy/tests/networkpolicy_test.yaml: new
helm-unittest suite that enumerates every metallb v0.16 host-network
port in the deny list (including 7946 memberlist and the escape-hatch
9121), pins port 9443 to its etcd kube-rbac-proxy owner with the
pod-IP collision note, and asserts the legacy 7473 entry is gone.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
This patch implements name-based selectors for
`CozystackResourceDefinitions.spec.secrets`. Application developers may
now specify secrets that should or should not be visible to end users by
specifying a `resourceNames` field with a string slice of acceptable
names. This will, for instance, let developers exclude a secret like
`postgres-dbname-superuser` that has a predictable name even if it does
not have predictable labels. Simple templates are supported, so
`postgres-{{ .name }}-superuser` is also a valid entry under
`resourceNames`.
```release-note
[lineage, controller] Let application developers determine resource
visibility for end users by name, as well as by labels.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Many resources created as part of managed apps in cozystack (pods,
secrets, etc) do not carry predictable labels that unambiguously
indicate which app originally triggered their creation. Some resources
are managed by controllers and other custom resources and this
indirection can lead to loss of information. Other controllers sometimes
simply do not allow setting labels on controlled resources and the
latter do not inherit labels from the owner. This patch implements a
webhook that sidesteps this problem with a universal solution. On
creation of a pod/secret/PVC etc it walks through the owner references
until a HelmRelease is found that can be matched with a managed app
dynamically registered in the Cozystack API server. The pod is mutated
with labels identifying the managed app.
```release-note
[cozystack-controller] Add a mutating webhook to identify the Cozystack
managed app that ultimately owns low-level resources created in the
cluster and label these resources with a reference to said app.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Since it's taking a while for metallb/metallb#2726 to get released, the
binaries with the fix are recompiled in-tree. Workaround for #909.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Upgraded MetalLB application version to `v0.14.8`.
- Introduced a new `frr-k8s` dependency for enhanced BGP management.
- Added new configuration options for TLS settings and extra containers
in the controller.
- Implemented new Custom Resource Definitions (CRDs) for managing FRR
configurations and node states.
- **Bug Fixes**
- Improved validation logic for service account names to ensure
consistency.
- **Documentation**
- Updated README files for the MetalLB and `frr-k8s` charts to reflect
new features and configuration options.
- **Refactor**
- Enhanced RBAC configurations for better resource management and
security.
- Improved webhook configurations for better validation and consistency.
- **Chores**
- Updated various YAML configuration files to include namespace
specifications for clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This change is aimed at improving the development experience.
- The option `make delete` has been added.
- Added check for `NAME` and `NAMESPACE` variables
- Now, any package (not just system ones) can include options such as
make show, make diff, make apply.
- Applications from packages/extra require explicit specification of the
`NAMESPACE`.
- Applications from packages/apps require explicit specification of both
`NAME` and `NAMESPACE`.
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>