mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, security (a fix that comes through a security advisory), docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Fixes merged to `main` reach the release lines the backport batch runs for automatically, so do not open backport PRs by hand: https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md --> ## What this PR does This upgrades the Kafka operator to Strimzi 1.2.0. Kafka 4.3 is the application default, with 4.2 also available. Upgrades must pass through Cozystack v1.7 and finish the ZooKeeper-to-KRaft migration first. The upgrade now stops while any Kafka lacks KRaft evidence. Every Kafka must report a nonempty `kafkaMetadataVersion` and the `KRaft` metadata state. The API conversion deletes that state, so without it a Kafka needs `Ready=True` or the `cozystack.io/kraft-verified` annotation, which migration 61 writes on every Kafka it accepts before it stops the operators. Any other explicit state blocks, and the error gives the reason for each Kafka. On every variant, the platform chart refuses to switch package sources while Strimzi still serves `v1beta2` and a Kafka fails that check. On variants without the migration hook, it also refuses while any Strimzi CRD still stores `v1beta2`, so `/migrations/61` has to run first. Migration 61 checks the same evidence again, including on repeat runs, and blocks unfinished broker-pool drains. A manual change that bypasses the managed package source is outside this check. Existing Kafka 3.x applications move to v4.3. Kafka 4 no longer accepts clients older than Kafka 2.1. Kafka 4 also rejects the topic settings `message.format.version`, `message.downconversion.enable` and `message.timestamp.difference.max.ms`, and a `segment.bytes` below 1 MiB (1048576), which Kafka 3.x accepted down to 14 bytes. A Kafka 4 broker whose log holds such a topic does not start. The chart now refuses to render a topic that sets any of these. The platform upgrade refuses to start while a Kafka application sets one, and migration 61 also checks KafkaTopic resources, so topics created outside the chart are caught too. A topic created through the Kafka admin API without a KafkaTopic is not visible to these checks. Restoring a topic-metadata backup taken on Kafka 3.x skips the three removed settings and raises a smaller `segment.bytes` to 1 MiB, with a warning for each. In-place downgrades are unsupported. The default 10-minute Helm timeout can leave a long broker roll Failed until its retry succeeds. The migration does not run automatically on every platform variant. Only the `isp-*` variants run it automatically. On other variants, run `/migrations/61` from the matching migrations image before upgrading, as the Kafka README describes. This includes clusters that run no Kafka but have the kafka-operator package, from the `paas` bundle or `bundles.enabledPackages`. Their Strimzi CRDs still store `v1beta2` even with no objects. The migration converts Strimzi resources and stored Helm references to `v1`. It stamps platform version 62. Running it manually also skips earlier migrations on a later move to an `isp-*` variant. Before it stops the operators, migration 61 refuses objects that the Strimzi 0.51 conversion tool would reject and names each one, for example a Kafka with `.spec.kafka.resources` or with OAuth, Keycloak or OPA settings. A failed conversion can leave the old operators stopped until the migration succeeds and the operator release upgrades. The failed conversion Job stays for diagnosis, but its ClusterRole, binding and ServiceAccount are removed. A Job that is still running when the wait times out keeps them, because the retry waits for it. Known limitations. Migration 61 converts with the old operator stopped, while Strimzi's conversion guide expects a running operator of 0.49 or later. I stop it because 0.45 keeps writing the Kafka status through `v1beta2`, but I have not checked this on a live cluster. Objects Strimzi created keep `ownerReferences` to `v1beta2` until Strimzi 1.2 reconciles them. Deleting a Kafka before that can leave dependents behind, and the chart's pre-delete hook covers only StrimziPodSets. The conversion image is pulled from quay.io by digest with no values override, so an air-gapped cluster has to mirror it under the same name. The full offline unit run passed on Linux on `1171d63adfafb45be34214fb9a0f565723e255fd`: 2,576 cases, no failures, errors or skipped tests. Later changes were checked on `69c704499fa4` with the Bats tests for migration 61, the migration runner, the Kafka hooks and the Kafka backup scripts (102), the platform Helm suite (310 tests) and the Kafka Helm suite (91 tests). After the rebase onto `main` at `5a5d5cd3d85f`, `a48a2f259947` passes the same Bats tests (115) and the platform and Kafka Helm suites (326 and 103 tests). It also passes the new check against added `documentIndex` selectors. I have not validated this revision on a live cluster. Closes #4365 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/805 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The Terraform provider does not model the Kafka `version` field. The website issue covers the upgrade notes the `default` variant needs. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(kafka)!: upgrade to Strimzi 1.2.0 and Kafka 4.3 by default, with Kafka 4.2 also available. Platform migration 61 converts Strimzi resources to v1 and moves Kafka 3.x applications to v4.3. Every Kafka must have confirmed KRaft state before the upgrade, including on repeat runs. Upgrade through Cozystack v1.7 first. Kafka 4 drops clients older than Kafka 2.1 and the message.format.version, message.downconversion.enable and message.timestamp.difference.max.ms topic settings, and requires segment.bytes of at least 1 MiB; the chart and the upgrade refuse a topic that breaks either rule, and a topic-metadata backup taken on Kafka 3.x restores without the removed settings and with segment.bytes raised to 1 MiB. In-place downgrades are unsupported. ```