mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but only the `s3_lifecycle` job of its plugin worker applies it, and that job is scheduled by the admin server. Cozystack deployed neither, so a lifecycle rule returned 200 and no object ever expired, on the platform backup bucket and on tenant buckets alike. That also made the documented way to bound backup storage a no-op. This PR adds the admin server and a worker to Simple and MultiZone SeaweedFS instances, behind a new `lifecycle.enabled` value that is **off by default**. A Client instance is unchanged, because its rules are enforced where the instance it connects to runs. The worker registers for `s3_lifecycle` only, so vacuum, balancing and erasure coding are not started as a side effect. The admin keeps its job state on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default when that is empty; `volume.storageClass` is not reused, because in MultiZone it can name a class local to one zone or node. The admin UI is protected by a generated password, because a tenant namespace admits traffic from the whole cluster. The dashboard Role lists the two new WorkloadMonitors. Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart renders no `[grpc.s3]` section, so the s3 gateway served its internal gRPC in plaintext while the worker dials it with mTLS, and every lifecycle delete failed the TLS handshake. The section is backported from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart patch, applied with `--forward` so the next `make update` stops at the point where the patch must be dropped instead of silently reversing it. The s3 pods are restarted once so they load the section. This also repairs IAM cache propagation between s3 replicas, which failed in the same way, and it applies whether or not lifecycle is enabled. The data-plane release now retries a failed install or upgrade in place (`RetryOnFailure`) instead of rolling it back. With the previous default remediation and unlimited retries, an upgrade that waited out its timeout, for example on a tenant quota with no room for the admin volume, rolled back and retried forever, restarting the s3 pods twice per cycle. The worker's hostname anti-affinity is relaxed to a preference so a rollout finishes on a single-node cluster. #### Why lifecycle is off by default The daily walk that SeaweedFS uses to expire objects sends its deletes without an object identity, and the s3 server skips its compare-and-swap when no identity is given; it does not re-check the object's age either. A key overwritten between the walk's listing and the delete loses its new content. This path is unchanged from 4.45 through 4.52 and on upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`, `identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup buckets write unique keys and are barely exposed; a tenant bucket that rewrites a fixed key is. On top of that, turning enforcement on would start applying, on upgrade, every rule that was stored and ignored until now. Both are retention changes an operator should choose, so nothing changes for existing instances until `lifecycle.enabled` is set. Making it the default again is left to a separate PR that carries object identity on the walk path; upstream report: https://github.com/seaweedfs/seaweedfs/issues/12451. #### Turning lifecycle on and off When `lifecycle.enabled` is set, rules that have been stored but ignored start applying within minutes, and objects they have already expired are deleted on that first pass, so review existing bucket lifecycle rules first. Set `lifecycle.storageClass` before enabling, and prefer a replicated class: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. Moving the volume means disabling lifecycle, deleting the `admin-data-seaweedfs-admin-0` PVC, changing the class and enabling lifecycle again. Each instance gains two small pods and a 1Gi PVC; when the tenant quota has no room for them, the data-plane release keeps retrying in place until it has. When `lifecycle.enabled` is turned off again, the parent release removes the admin credentials Secret while the data-plane release removes the admin StatefulSet only on its own reconcile, shortly after. An admin pod that restarts in that window fails to start on the missing Secret until the StatefulSet is gone; nothing is lost, and the order is left as is. The admin PVC stays until the instance is deleted, and a later re-enable starts the admin with a new password. Verified on a test cluster with the 4.45 chart: without the backport, the worker found the expired objects and every delete failed with `tls: first record does not look like a TLS handshake`; with this branch and lifecycle enabled, a rule with a past expiration date removed the matching objects and left the others, both on a manual run and on the first scheduled pass. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up needed, not opened yet: `operations/services/backup-classes.md` (v1.7 and next) advises an S3 lifecycle policy, which the bundled SeaweedFS applies only once `lifecycle.enabled` is set; the SeaweedFS reference page gains `lifecycle.enabled` and `lifecycle.storageClass` from the regenerated README on the next release - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The terraform provider does not model the SeaweedFS application, so the new values do not reach it. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added optional, daily S3 bucket lifecycle-rule processing for supported SeaweedFS configurations. It is disabled by default; Client instances rely on lifecycle rules configured on the connected SeaweedFS. * Added configurable storage for lifecycle job state and dashboard visibility for lifecycle components. * Enabled the S3 gRPC server to use its internal TLS identity when security is configured. * **Bug Fixes** * Improved storage-class handling so numeric-looking names remain strings and unset classes are omitted where appropriate. * Clarified backup cleanup guidance, including when lifecycle rules apply and how to reclaim storage. <!-- end of auto-generated comment: release notes by coderabbit.ai -->