Replace deprecated single-node VLogs CR with VLCluster (cluster mode)
for reliability and horizontal scalability.
Changes:
- Replace VLogs (v1beta1) with VLCluster (v1) using vlinsert/vlselect/vlstorage
- Update fluent-bit outputs to vlinsert-generic:9481
- Update Grafana datasource to vlselect:9471
- Update ExternalName service from vlogs-generic to vlinsert-generic
- Add VPA for all VLCluster components
- Update WorkloadMonitors for three-component architecture
- Add migration 35 to preserve old VLogs resources with keep annotation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Cilium's MTU auto-detection does not account for VXLAN overhead when
running inside KubeVirt VMs. The VM interface inherits MTU 1400 from
the parent OVN/Geneve overlay, and Cilium sets all interfaces
(cilium_vxlan, lxc*, cilium_host/net) to 1400 without subtracting
the 50-byte VXLAN encapsulation overhead.
This causes intermittent packet drops for large packets (TLS
handshakes, HTTP responses with data), resulting in timeouts and
499 errors for services running in tenant clusters.
Set MTU to 1350 (1400 - 50 VXLAN overhead) explicitly in the default
Cilium values for tenant Kubernetes clusters.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
PR #2075 introduced `_cluster.cluster-domain` references in
monitoring-agents `values.yaml` for FQDN resolution in tenant clusters.
This broke the fluent-bit subchart because `_cluster` values are not
accessible from the Helm subchart context — only `global` values are
shared with subcharts.
This PR replaces `_cluster` references with a new `global.clusterDomain`
variable:
- Empty by default (management cluster uses short DNS names like
`service.namespace.svc`)
- Set to the management cluster domain (e.g. `cozy.local`) for tenant
clusters, enabling FQDN resolution for cross-cluster service discovery
Fixes#2084
### Release note
```release-note
[system] Fix monitoring-agents installation failure caused by inaccessible _cluster values in fluent-bit subchart context. Introduce global.clusterDomain for proper FQDN resolution in tenant workload clusters.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Monitoring agent configuration updated to support configurable cluster
domain names for greater flexibility.
* Remote write and log-forwarding endpoints adjusted to align with
cluster domain handling, improving compatibility when deploying across
different cluster DNS setups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
PR #2075 added _cluster.cluster-domain references to monitoring-agents
values.yaml for FQDN resolution. This broke fluent-bit because it is
a subchart where _cluster values are not accessible (only global
values are shared with subcharts), causing "index of untyped nil".
Revert values.yaml to short DNS names (.svc) for the management
cluster where they resolve locally, and add FQDN with cluster-domain
suffix in the tenant kubernetes HelmRelease where _cluster values are
available and cross-cluster DNS resolution is needed.
Closes#2084
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Kamaji auto-derives the konnectivity proxy image tag as v0.{minor}.0
from the Kubernetes version. For K8s v1.35, this produces v0.35.0,
but the kas-network-proxy/proxy-server:v0.35.0 image does not exist
in registry.k8s.io yet, causing ImagePullBackOff on new TCP pods.
Add konnectivity-versions.yaml mapping to explicitly override the
konnectivity version when the auto-derived tag is unavailable.
For v1.35, pin to v0.34.0 (latest available).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Update Kubernetes version matrix to match Talos 1.12 support range:
- Add v1.35.0 (latest) and v1.34.4
- Update existing patch versions (v1.33.8, v1.32.12)
- Drop EOL versions v1.28 and v1.29
- Set default version to v1.35
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Move nil check before req dereference in CreateVolume
- Scope CiliumNetworkPolicy endpointSelector to specific VMI
- Use vmNamespace from NodeId for VMI lookup instead of infraNamespace
- Log PVC lookup errors in ControllerExpandVolume
- Wrap CNP ownerReference updates in retry.RetryOnConflict
- Fix infraClusterLabels validation to check runControllerService flag
- Dereference nodeName pointer in error message
- Replace panic with klog.Fatal for consistent error handling
- Honor CSI readonly flag in NFS NodePublishVolume
- Log mount list errors in isNFSMount
- Reorder Dockerfile ENTRYPOINT after COPY for better layer caching
- Add cleanup on e2e test failure and --wait on pod deletion
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Remove separate NFS Application dependency from e2e test. The kubevirt
CSI driver wrapper now handles RWX Filesystem volumes natively - PVCs
with ReadWriteMany accessMode use the standard kubevirt StorageClass.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Implement a wrapper around upstream kubevirt-csi-driver that adds RWX
Filesystem volume support via LINSTOR NFS exports:
- CreateVolume: intercepts RWX+Filesystem requests, creates DataVolume
with explicit AccessMode=RWX and VolumeMode=Filesystem
- ControllerPublishVolume: waits for PVC bound, extracts NFS export URL
from infra PV, creates CiliumNetworkPolicy with VMI ownerReferences
- ControllerUnpublishVolume: manages CNP ownerReferences per-VMI
- ControllerExpandVolume: delegates to upstream, disables node expansion
for NFS volumes
- NodeStageVolume/NodePublishVolume: mounts NFS at target path
- NodeExpandVolume: no-op for NFS (LINSTOR handles resize)
Also includes:
- RBAC for CiliumNetworkPolicy management in infra cluster
- Explicit --run-node-service=false for controller deployment
- Explicit --run-controller-service=false for node DaemonSet
- nfs-utils in container image for NFS mount support
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
When the Gateway API addon is enabled, automatically configure
cert-manager with enableGatewayAPI: true. Uses the same default
values + mergeOverwrite pattern as Cilium for consistency.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
- Removes `hostNetwork` for Ingress NGINX Controller for Managed
kubernetes cluster
- Adds Service of type NodePort for load balancing
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[kubernetes] use NodePort service instead of hostNetwork for ingress-nginx
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Configuration**
* Updated ingress-nginx controller to use NodePort exposure method
(ports 30000 for HTTP, 30001 for HTTPS) when configured as "Proxied."
* Removed unnecessary hostNetwork configuration for the controller.
* Service port mappings updated to align with new NodePort settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Change the default resourcesPreset for kube-apiserver from "medium" to
"large" to prevent OOM kills during normal operation. The "medium"
preset provides insufficient memory for clusters with moderate workloads.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add dedicated flux-tenants controller with label selector
--watch-label-selector=sharding.fluxcd.io/key=tenants to handle
tenant workloads separately from platform components.
Update all kubernetes app HelmReleases to:
- Use chartRef with ExternalArtifact instead of OCIRepository sourceRef
- Add sharding.fluxcd.io/key=tenants label
- Add cozystack.io/target-cluster-name label
Update fluxinstall to parse multiple YAML manifest files and
use flux service for storage-adv-addr.
Add cozystack-basics package for core tenant/namespace setup.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Integrate fix
- https://github.com/kubevirt/cloud-provider-kubevirt/pull/379
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[kubernetes] Fix endpoints for cilium-gateway
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved handling of incomplete endpoint data by introducing fallback
detection mechanisms.
* Enhanced service discovery to gather endpoints from all available
resources when standard detection fails.
* Updated logging to provide better visibility into fallback operations
and current resource status.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace Helm lookup functions with FluxCD valuesFrom mechanism for
reading cluster and namespace configuration.
Changes:
- Create Secret cozystack-values in each namespace with values.yaml key
containing _cluster and _namespace configuration as nested YAML
- Configure HelmReleases to read from this Secret via valuesFrom
(valuesKey defaults to values.yaml, so it can be omitted)
- Update cozy-lib helpers to access config via .Values._cluster
- Add default values for required _cluster keys to ensure all fields exist
- Update Go code (cozystack-api and helm reconciler) to use new format
This eliminates the need for Helm lookup functions while maintaining
the same configuration interface for charts.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
More granular k8s app extensions dependencies.
As a part of the change, packages were updated:
- monitoring-agents: metrics-server moved to a separate package
- monitoring-agents: prometheus-operator-crds moved to a separate
package
Main Cozystack bundles were updated to reflect packages changes.
### Release note
```release-note
Granular k8s extensions dependencies.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a standalone metrics-server release and a
prometheus-operator-crds release.
* **Improvements**
* Expanded Prometheus-related CRDs with new configuration options and
status subresources.
* Adjusted monitoring orchestration so components (cilium, metrics,
CRDs) install in a clearer order.
* **Refactor**
* Split metrics-server out of the monitoring-agents bundle and moved
Prometheus CRDs into a dedicated package.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Similar to an earlier issue with DataVolumes remaining after deleting
the tenant k8s cluster using them, a similar problem is observed with
LoadBalancer services consuming external IPs. This patch adds another
step to the cleanup Helm hook to delete any such services.
### Release note
```release-note
[kubernetes] Add a cleanup hook to delete LoadBalancer services after
deleting the tenant Kubernetes cluster that they were servicing.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
The kubelet version of tenant k8s clusters is baked into the worker VM
image. Previously, selecting any version of tenant k8s had an impact
only on the controlplane, the workers were fixed at v1.33. This patch
modifies the KubeadmConfigTemplate to attempt to download the
user-selected versions of kubelet and kubeadm and replace the baked-in
versions with those. If failing, the bootstrap continues with the
baked-in versions.
### Release note
```release-note
[kubernetes] Make kubelet versions on tenant k8s clusters' worker nodes
user-configurable.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
When deleting a Kubernetes, some resources may linger post deletion
because of a race to remove HelmReleases deployed inside the tenant
cluster and the removal of the cluster and its controlplane itself. This
patch modifies the existing pre-delete hook to remove those helmreleases
instead of simply suspending them. Similarly, datavolumes may also
remain. These are now delete with a post-delete hook.
### Release note
```release-note
[kubernetes] Use Helm hooks to clean up HelmReleases deployed in tenant
clusters and DataVolumes backing the tenant clusters' PVCs when deleting
a tenant Kubernetes.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
When deleting a Kubernetes, some resources may linger post deletion
because of a race to remove HelmReleases deployed inside the tenant
cluster and the removal of the cluster and its controlplane itself. This
patch modifies the existing pre-delete hook to remove those helmreleases
instead of simply suspending them. Similarly, datavolumes may also
remain. These are now delete with a post-delete hook.
### Release note
```release-note
[kubernetes] Use Helm hooks to clean up HelmReleases deployed in tenant
clusters and DataVolumes backing the tenant clusters' PVCs when deleting
a tenant Kubernetes.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
The managed Kubernetes app accepts a .controPlane.replicas field, but
this value was never used, instead being hardcoded in the
KamajiControlPlane template to 2. This patch fixes this.
### Release note
```release-note
[kubernetes] Pass the .controlPlane.replicas field into the
KamajiControlPlane template, making the replica count of the
controlplane pods user-configurable.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
