Remove authorizationServicesEnabled as it's incompatible with public
clients (requires service account which public clients don't have).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Switch ClusterKeycloak to use internal service URL instead of external
ingress, and enable insecureSkipVerify for environments with self-signed
certificates.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Migrates usage of deprecated field `running` to `runStrategy`
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[vm] migrate usage of deprecated field `running` to `runStrategy`
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* VM running states now support Always, Halted, Manual, RerunOnFailure,
and Once strategies
* Configure CPU, memory, and socket allocation for virtual machines
* Add GPU support to virtual machines
* Define network subnets for virtual machines
* Configure storage classes and options for system disks
* Set SSH keys and cloud-init parameters for initialization
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- Remove `cozystack.cozystack-controller` from monitoring package
dependencies
- Allows monitoring to work without cozystack-engine being enabled
## Motivation
The monitoring package depends on `cozystack.cozystack-controller` which
is only installed as part of `cozystack-engine`. This prevents
monitoring from becoming ready when cozystack-engine is not enabled.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Simplified monitoring default: reduced installed sub-components and
top-level dependencies for a leaner deployment.
* Removed generation of several automated workload-monitoring manifests
to streamline resource set.
* **New Features**
* Added a curated list of monitoring dashboards for cluster, networking,
storage, databases, control-plane, ingress, observability tooling, and
key services.
* Added three ExternalName services to expose tenant monitoring
endpoints.
* **Improvements**
* Made monitoring agents namespace-aware by templating
tenant/multi-namespace references for metrics and logging endpoints.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Expose the --mtu flag in values.yaml to allow overriding the default
WireGuard interface MTU (1420). This is needed for environments where
the underlying network has a lower MTU, such as Azure VMs (eth0 MTU
1400), to avoid packet fragmentation in the WireGuard tunnel.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Add Qdrant vector database as a new managed application, following the
HelmRelease-based vendoring pattern (same as nats, ingress, seaweedfs).
Changes:
- `packages/system/qdrant/` — vendored upstream Qdrant Helm chart
- `packages/apps/qdrant/` — wrapper chart creating Flux HelmRelease CR
- `packages/system/qdrant-rd/` — ApplicationDefinition for
`apps.cozystack.io/v1alpha1/Qdrant` CRD
- `hack/e2e-apps/qdrant.bats` — E2E test
Features:
- Single replica or clustered mode (automatic based on replica count)
- Persistent storage with configurable size and storage class
- Resource presets (nano to 2xlarge)
- API key authentication (auto-generated)
- Optional external LoadBalancer access
- Dashboard integration with WorkloadMonitor and ServiceMonitor
### Release note
```release-note
[qdrant] Add Qdrant vector database as a managed application
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added Qdrant vector database as a managed service, enabling users to
deploy and manage Qdrant instances with configurable replicas, storage,
and resource presets.
* Introduced support for external access, persistent storage, API key
management, and metrics monitoring integration.
* Added end-to-end testing for Qdrant deployments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add PackageSource and PaaS bundle entry so the platform deploys
qdrant-rd, which provides the ApplicationDefinition that registers
the Qdrant Kind in the API server.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Previously a namespaced role was created per tenant and access level.
Since these roles are all identical, it's sufficient to create a cluster
role per access level and just create namespaced rolebindings to these
cluster roles per tenant. This will enable aggregation rules. E.g. if a
new API group is installed, such as backups.cozystack.io, a new
clusterrole can be created for managing this API group with a label like
rbac.cozystack.io/aggregate-to-admin. Smart use of aggregation rules
will enable automatic granting of access rights not just to admin, but
to super-admin too, and there will be no need to update every single
tenant.
### Release note
```release-note
[tenant,rbac] Use ClusterRoles with aggregationRules instead of roles
per every tenant.
```
## What this PR does
Previously a namespaced role was created per tenant and access level.
Since these roles are all identical, it's sufficient to create a cluster
role per access level and just create namespaced rolebindings to these
cluster roles per tenant. This will enable aggregation rules. E.g. if a
new API group is installed, such as backups.cozystack.io, a new
clusterrole can be created for managing this API group with a label like
rbac.cozystack.io/aggregate-to-admin. Smart use of aggregation rules
will enable automatic granting of access rights not just to admin, but
to super-admin too, and there will be no need to update every single
tenant.
### Release note
```release-note
[tenant,rbac] Use ClusterRoles with aggregationRules instead of roles
per every tenant.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Add topology.kubernetes.io/zone node label alongside kilo.squat.ai/location
for standard Kubernetes topology awareness across Hetzner and Azure zones.
Update Azure docs with correct Package resource format (spec.components),
replace hardcoded values with placeholders, and remove environment-specific
infrastructure references.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
This patch adds a conditional for running on a statically defined VM the
maintainers have SSH access to if the pull request has a `debug` label.
This is useful for debugging failing workflows when the diagnostic info
from the pipeline is insufficient.
### Release note
```release-note
[ci] Run builds on a static VM with SSH access if the PR has a debug
label.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
This release contains no user-facing changes. Internal workflow
adjustments have been made to optimize development infrastructure.
* **Chores**
* Updated CI/CD pipeline configuration for improved test execution
efficiency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
WorkloadMonitor resources are only needed when extra/monitoring is
deployed. Move them from system/monitoring to extra/monitoring package.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
This patch adds a conditional for running on a statically defined VM the
maintainers have SSH access to if the pull request has a `debug` label.
This is useful for debugging failing workflows when the diagnostic info
from the pipeline is insufficient.
### Release note
```release-note
[ci] Run builds on a static VM with SSH access if the PR has a debug
label.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
- Replace hardcoded tenant-root with {{ .Release.Namespace }} in vmagent
and fluent-bit configs
- Add ExternalName services in cozystack-basics to redirect monitoring
traffic from cozy-monitoring to tenant-root when engine is deployed
- Add missing components to monitoring PackageSource
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
When the Gateway API addon is enabled, automatically configure
cert-manager with enableGatewayAPI: true. Uses the same default
values + mergeOverwrite pattern as Cilium for consistency.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Kubelet kills bff and web containers on slow hardware because the
livenessProbe only allows 33 seconds for startup. Add startupProbe
with failureThreshold=30 and periodSeconds=2, giving containers up
to 60 seconds to start before livenessProbe kicks in.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add dashboards.list that defines which Grafana dashboards to create.
Without this file, no GrafanaDashboard resources were being generated.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Add optional node-lifecycle-controller that automatically deletes
unreachable NotReady nodes from the cluster. This solves the problem
of "zombie" node objects left behind when cluster autoscaler deletes
cloud instances.
Features:
- Monitors nodes matching a label selector
- Deletes nodes that are NotReady for configurable duration
- Verifies unreachability via ICMP ping before deletion
- Supports protected labels to prevent deletion of specific nodes
- Leader election for HA deployment
Disabled by default. Enable with:
nodeLifecycleController:
enabled: true
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Remove components that are already defined in separate PackageSources
to avoid race conditions and HelmRelease conflicts:
- vertical-pod-autoscaler
- postgres-operator
- grafana-operator
- victoria-metrics-operator
- prometheus-operator-crds
These components should be installed via their dedicated PackageSources.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Helm does not reliably install all CRDs from templates/ directory,
particularly when the crds.yaml file is large. This causes
linstorsatellites.piraeus.io CRD to be missing, breaking satellite pod
creation.
Changes:
- Create new piraeus-operator-crds package with all piraeus CRDs
- Add piraeus-operator-crds as dependency for piraeus-operator
- Set privileged: true for CRDs package to ensure namespace has correct
PodSecurity labels from the start
- Disable installCRDs in piraeus-operator since CRDs come from separate
package
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The monitoring package depends on cozystack.cozystack-controller which
is only installed as part of cozystack-engine. This prevents monitoring
from becoming ready when cozystack-engine is not enabled.
Remove this dependency to allow monitoring to work independently.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
Use GitHub Copilot CLI for automatic changelog generation on tagged
releases.
### How it works
When a new version tag is pushed, the `generate-changelog` job in
`tags.yaml`:
1. Checks out the `main` branch with full history and tags
2. Verifies that a changelog file doesn't already exist in
`docs/changelogs/`
3. Installs GitHub Copilot CLI (`@github/copilot` npm package)
4. Runs Copilot CLI in non-interactive mode (`-p` flag) with
`--allow-all-tools --allow-all-paths` to generate the changelog
following the instructions in `docs/agents/changelog.md`
5. Commits the generated file to a `changelog-vX.Y.Z` branch and opens a
PR to `main`
### Authentication
- `COPILOT_GITHUB_TOKEN` secret — fine-grained PAT with **"Copilot
Requests: Read"** account permission, used to authenticate Copilot CLI
- `GH_PAT` secret — used by `gh` CLI inside the Copilot session to query
PR authors via GitHub API
### Release note
```release-note
[ci] Replaced Gemini with GitHub Copilot CLI for automatic changelog generation on release tags
```
## Summary
- Fix invalid PromQL expression in `CNPGClusterOffline` alert rule that
causes vmalert pods to crash
- Fix PromQL logic so the alert fires correctly when cluster is offline
## Problem
The `CNPGClusterOffline` alert in
`packages/system/postgres-operator/alerts/cnpg-default-alerts.yaml` had
two issues:
1. **Syntax error** - Extra closing parenthesis causing vmalert pods to
crash:
```promql
sum by (namespace, pod) (cnpg_collector_up)) OR on() vector(0) == 0
^^-- extra )
```
2. **Logic error** - After removing the extra parenthesis, the
expression still had incorrect logic:
```promql
sum by (namespace, pod) (cnpg_collector_up) OR on() vector(0) == 0
```
This evaluates as `cnpg_collector_up OR (vector(0) == 0)` which fires
when the cluster is **online** (wrong behavior).
## Solution
Fix both issues by properly wrapping the OR expression in parentheses:
```promql
(sum by (namespace, pod) (cnpg_collector_up) OR on() vector(0)) == 0
```
This correctly fires when `cnpg_collector_up` is 0 or absent (cluster
actually offline).
## Test plan
- [ ] Verify vmalert pods start successfully after applying the fix
- [ ] Verify the alert rule is properly loaded
- [ ] Verify alert only fires when CNPG cluster is actually offline
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Change TLS certificate duration from 90 days to 10 years to prevent
certificate expiration issues. Also adjust renewBefore from 15 to 30 days.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
When OIDC is disabled, the dashboard's token-proxy now properly
validates bearer tokens against the k8s API's JWKS url.
### Release note
```release-note
[dashboard] Verify bearer tokens against the issuer's JWKS url.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented JWKS-based JWT token verification for enhanced security.
* Added conditional role-based access control for non-OIDC deployments.
* **Chores**
* Updated authentication dependencies and modernized token validation.
* Removed legacy external token validation mechanism.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
When OIDC is disabled, the dashboard's token-proxy now properly
validates bearer tokens against the k8s API's JWKS url.
### Release note
```release-note
[dashboard] Verify bearer tokens against the issuer's JWKS url.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Remove extra closing parenthesis in the CNPGClusterOffline alert expression
that causes vmalert pods to crash with "bad prometheus expr" error.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
- Add monitoring as a system chart
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Introduced comprehensive monitoring stack with Grafana, Victoria
Metrics operators, Prometheus operator, and vertical pod autoscaler for
enhanced system observability and performance metrics.
* **Chores**
* System upgraded to version 27 with automatic monitoring resource
reorganization for improved infrastructure management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add migration 26 that re-labels monitoring resources to be owned by
monitoring-system HelmRelease instead of monitoring. This allows
seamless transition from direct resource management in extra/monitoring
to delegated management via system/monitoring chart.
Migration steps:
- Find all monitoring HelmReleases in tenant namespaces
- Suspend HelmRelease to prevent reconciliation
- Delete helm secrets to orphan resources
- Relabel all resources to monitoring-system
- Delete suspended HelmRelease
Also updates:
- monitoring-application.yaml: add monitoring-system component
- helmrelease.yaml: reference monitoring-system ExternalArtifact
- values.yaml: bump targetVersion to 25
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Configure CoreDNS chart to create `kube-dns` ServiceAccount matching
Kubernetes bootstrap ClusterRoleBinding
- Fixes RBAC errors (`Failed to watch`) when CoreDNS pods restart
## Problem
Kubernetes bootstrap creates a `ClusterRoleBinding: system:kube-dns`
that references `ServiceAccount: kube-dns` in `kube-system`. However,
the coredns chart was using the `default` ServiceAccount because
`serviceAccount.create` was not enabled.
This caused CoreDNS pods to fail with `[ERROR] plugin/kubernetes: Failed
to watch` errors after restarts, as they lacked RBAC permissions to
watch the Kubernetes API. Old pods worked due to cached data, but new
pods failed after rollout.
## Solution
Add ServiceAccount configuration to
`packages/system/coredns/values.yaml`:
```yaml
serviceAccount:
create: true
name: kube-dns
```
## Test plan
- [x] Verify ServiceAccount `kube-dns` is created: `kubectl get sa -n
kube-system kube-dns`
- [x] Verify deployment uses correct ServiceAccount: `kubectl get
deployment -n kube-system coredns -o
jsonpath='{.spec.template.spec.serviceAccountName}'` → `kube-dns`
- [x] Restart CoreDNS pods and verify all pods are Ready
- [x] Check logs show no RBAC errors
- [x] Test DNS resolution works
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated DNS service configuration with service account settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- Add migration 25 for v1.0 upgrade cleanup:
- Remove legacy `cozystack` installer deployment
- Remove legacy `cozystack-assets` statefulset
- Remove `bootbox` and `bootbox-rd` HelmReleases (will be recreated by
new platform chart if enabled)
- Revert migration 23 to original state (only removes
cozystack-resource-definition-crd)
- Update targetVersion from 24 to 26
- Fix jq syntax error in `hack/migrate-to-version-1.0.sh`
Migration 23 was already executed on existing clusters at version 23+,
so adding cleanup there would never run. The new migration 25 ensures
proper cleanup during v1.0 upgrade.
## Test plan
- [ ] Upgrade from v0.4x to v1.0
- [ ] Verify old `cozystack` deployment is deleted
- [ ] Verify old `cozystack-assets` statefulset is deleted
- [ ] Verify old `bootbox` and `bootbox-rd` HelmReleases are deleted
- [ ] Verify platform installs successfully after migration
- [ ] Run `./hack/migrate-to-version-1.0.sh` without jq errors
Migration 23 was already executed on existing clusters, so the legacy
installer removal never ran. This moves the cleanup to a new migration 25
which will execute during upgrade.
Changes:
- Revert migration 23 to original state (only removes cozystack-resource-definition-crd)
- Add migration 25 for v1.0 upgrade cleanup:
- Remove legacy cozystack installer deployment
- Remove legacy cozystack-assets statefulset
- Remove bootbox and bootbox-rd HelmReleases
- Update targetVersion from 24 to 26
- Fix jq syntax error in migrate-to-version-1.0.sh
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Remove duplicate `bootbox-system` component from `bootbox-application`
PackageSource (was duplicating system/bootbox installation)
- Auto-create `cozystack.bootbox-application` Package when bootbox is
enabled in `bundles.enabledPackages`
- This ensures `bootbox-rd` (ApplicationDefinition) is properly
installed as a dependency before `cozystack.bootbox`
## Test plan
- [ ] Enable bootbox in `bundles.enabledPackages`
- [ ] Verify both `cozystack.bootbox-application` and
`cozystack.bootbox` Packages are created
- [ ] Verify `bootbox-rd` (ApplicationDefinition) is installed
- Fix docker build context for migrations image
- Fix yq flags for compatibility with Mike Farah's yq v4
- Add platform image to main build target
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Kirill Ilin