mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
IvanHunters
6818795588
## What this PR does Raises how many buckets a SeaweedFS instance can serve, by lowering the master's `volumeSizeLimitMB` from 1000 to 400 and its two-copy growth count from 6 to 3. The size limit is exposed on the app chart, where an operator can reach it. Existing instances stay at 1000 unless they already name a limit, so for them the upgrade changes only the growth count. A platform migration does that where migrations run; on the `default` variant it is a manual step before upgrading. A unit test pins each constant separately, and an e2e case creates several buckets and requires each to take a write. Late in a full e2e run every S3 `PUT` starts timing out and a backup suite fails its access preflight. It reads like a credentials or COSI problem and is not one: the instance has no volume slots left, so the master cannot grow a volume for a newly created bucket. The disk is still almost entirely free, which is what makes it look like a flake. The COSI objects say nothing either way, the claim is ready and the access is granted. ### The ceiling A volume server started with `-max 0` takes a slot count derived from disk rather than a fixed one, replication `001` spends a slot on each of the two servers per logical volume, and a collection claims `COPY_2` volumes each time it grows. So the ceiling is set by the product of the size limit and `COPY_2`, and moving either number alone moves it. The mechanism is upstream in `MaybeAdjustVolumeMax` (`weed/storage/store.go`) and `findVolumeCount` (`weed/topology/volume_growth.go`); what matters for choosing the numbers is only that upshot. Do not divide the slot figures out of the nominal PVC size, because what the server counts is the space the filesystem makes available and not the volume's advertised size. At the upstream 1000 and 6 a 10Gi volume PVC gives nine slots per server and two collections, which the platform's own `cozy-backups` bucket and a single tenant bucket already use. That same PVC was seen to give nineteen slots at 500, which puts 400 at twenty-three to twenty-four by the same division, or eight collections. The span is the observation's own: the free space behind nineteen at 500 fits more than one filesystem geometry, and those divide differently at 400. Eight either way, because a collection that cannot get a whole batch of `COPY_2` keeps the volumes it did get and writes into them, so the count rounds up rather than down. Every figure here is for the shipped `replicationFactor: 2`. The master picks which of its four growth counts to apply from the replication the filer asks for, and the filer takes that from `replicationFactor`, so at 1 or 3 copies it reads `COPY_1` or `COPY_3` and the `COPY_2` this PR sets does not apply. Lowering the size limit still raises the ceiling there, since it is the divisor whatever the replication is. I measured this instead of only deriving it, on the real `weed` 4.31 binary: one master, two volume servers with `-max 0`, each on its own 1 GB filesystem image, replication `001`, then one `assign` per fresh collection until the master refused. Scaled 10:1 against the 10Gi PVC, so the limits scale with it. | `volumeSizeLimitMB` | `COPY_2` | slots reported | collections served | | --- | --- | --- | --- | | 100 | 6 | 10 | 2 | | 100 | 3 | 10 | 4 | | 50 | 3 | 20 | 7 | | 40 | 3 | 25 | 9 | | 25 | 6 | 40 | 7 | The fourth row is the pair this PR ships. The refusal arrives as `No writable volumes and no free volumes left`, with `volumes == max` on both servers and the disk nearly empty. The reported `max` matched the space the filesystem made available over the limit in every run, which is the rule the PVC figures above are divided with. Those runs sit on a 1 GB image, so their counts are the rule's, not the PVC's: a 10Gi volume has its own filesystem geometry and divides to its own numbers. ### Why both constants move The last two rows are the argument: lowering the size limit and lowering the growth count reach the same ceiling. They do not cost the same. Open descriptors scale with stored bytes rather than with the PVC, so at 400 a 10 TiB server keeps roughly 52k of them against 21k upstream, where taking the whole reduction out of the size limit would have put it at 105k, well past a 65536 `nofile` limit where one is set. Two stay open per volume, a `.dat` and an `.idx`; the `.vif` beside them is written and closed. That cost lands hardest on large instances, which never had a ceiling problem in the first place: at any of these limits a multi-TiB PVC has slots to spare. Lowering the growth count costs no descriptors at all. It costs write spread, since a bucket then spreads its writes over three volumes rather than six, and that cost is flat, it does not grow with the instance. So the reduction is split across both. Neither number is a new bet: 3 is what upstream already uses for three-copy replication. One price is shared by both routes. A collection absorbs `volumeSizeLimitMB × COPY_2` before it needs another grow, so growth requests arrive five times as often as they did at 6000 MB. A collection also starts with half as many writable volumes at `replicationFactor: 2`, and a volume server takes the writes into each volume one at a time, so a heavily parallel writer spreads over fewer volumes until the collection grows again. That holds for instances kept at 1000 too. ### On upgrade On a variant that runs platform migrations, the upgrade does not change the limit on an existing instance unless its operator already set one. Platform migration 57 writes `master.volumeSizeLimitMB: 1000` into every existing SeaweedFS instance that has not chosen a limit, except one whose `master` is not an object (below), so 400 applies only to instances created after this release. The `default` variant runs no platform migrations. A cluster there that installed SeaweedFS by hand has to apply the same patch before upgrading, and the operations guide has the command. Both guide steps end with a `complete` line. `jq` prints it only after reading the whole list, and the pinning loop stops at the first failed patch before it, so a run without that line is incomplete, whatever else it printed. A chosen limit is a whole number from 1 to 30000 and is kept. Anything else counts as not chosen and becomes 1000: the previous chart ignored the key and ran 1000 in its place, and the new schema would refuse it. A kept value other than 1000 takes effect with the upgrade itself. A lower one can strand buckets and a higher one takes the room new buckets grow into, so the guide has a query that lists those instances before upgrading. An instance kept at 1000 still gains: at `replicationFactor: 2` a grow now claims three volumes instead of six, so it has room for about twice as many new buckets. Three shapes get their own line in the Job log, with what they hold. A non-object `spec.values` is pinned too. helm-controller v1.5.0 ignores values it cannot read as a map and renders the chart defaults, so that instance runs at 1000 today and would drop to 400. The merge patch replaces the unreadable value, and nothing that applies is lost. A non-object `master` is left alone. Every v1 schema already refuses it, so that release does not render today, and the log says to set the limit in the same edit that repairs it. A `valuesFrom` with entries besides the `cozystack-values` Secret is named and pinned anyway. helm-controller merges `spec.values` over those entries, so a limit set there does not apply, unless it is set through `targetPath`, which can still win depending on the entries before it. The guide steps print the same three. The value goes into the application's HelmRelease, whose `spec.values` is the application spec, so it covers an instance created through the API and one the tenant module renders alike, the platform's own in `tenant-root` included. The tenant module renders that release without `spec.values`, which a new tenant chart test pins, and a Helm upgrade leaves alone a key that neither the previous nor the new manifest carries: I measured that on a kind cluster with the real HelmRelease CRD and Helm 4.3.0, on client-side, server-side and forced server-side apply, and read `createPatch` at the v4.1.1 that helm-controller v1.5.0 runs. Where migrations run, the order comes from the platform release itself. The migration is its pre-upgrade hook, and the `cozystack-packages` source every application chart is built from is a manifest of that same release, applied only after the hooks finish, so no instance meets the new chart unstamped and a failed migration stops the platform upgrade before that source moves. That holds while the platform source is pinned by digest, as it is out of the box; a source that follows a tag, a semver range or a branch lets source-controller move `cozystack-packages` on its own schedule, and an instance there can run at 400 until the migration has run. Until `cozystack-packages` moves, each patched release still runs its old chart, which ignores the key: no v1 release reads it, and none closes the `master` object in its values schema. The migration's test runs the script under the migrations image's shell against a fake `kubectl` that refuses any selector but the one cozystack-api lists SeaweedFS with, because an empty fleet is a legitimate answer and a mistyped selector would otherwise pass while pinning nothing. The same test runs the two guide steps, found by a marker comment above each, against the same fake and checks that they select the same releases as the migration. Lowering an existing instance is left to its operator, because it is not free. Its volumes between the new limit and the old go oversized on the next heartbeat and leave the master's writable set, and each affected collection needs one successful grow before its next write lands. Nothing is lost and the volumes stay readable. Those grows compete for one pool, and the budget is per collection rather than per server. An oversized volume is charged no headroom while one below the limit is charged all of its unfilled room, so the slots open to a stranded collection are free space less that room, divided by the limit; a grow then claims a whole `COPY_2` batch and takes as much of it as is there, and the volumes it creates are empty and charge their headroom straight back. Measured on the rig: two oversized collections with two free slots between them, and the first to write took both, leaving the second refused afterwards with the disk still reporting a quarter of itself free. Collections that keep writing are not free either: they were grown six at a time, and each of their volumes below 400 MB can hold back a whole 400 MB. Measured at a scaled pair, a server with free space for one batch per collection refused the only grow it needed, because a second, nearly empty collection held the room, and the same free space without that collection let the grow through. The master also grows, on its own schedule and with no write, any collection with no writable volume at or below 90 percent of the limit, which at 400 is 360 MB; measured at the same scaled pair, a collection with every volume in that band got two new volumes per server within one pass while a roomy one was left alone. So the precondition on every volume server is a batch for each collection with no volume at or below that line plus 400 MB for each volume still below the limit, and no volume size exempts an instance from it, since a larger volume carries more buckets and the budget scales with them. The `minFreeSpacePercent` reserve only comes into it on a server whose writable volumes have all passed the limit, since nothing there is charged headroom. It is five percent of what the filesystem reports rather than of the PVC's nominal size, so `All >= 8000 MiB` is where it reaches one 400 MB limit; a 10Gi PVC measures 9979 MiB and clears that, and such a server then refuses a grow only once its disk is down to the reserve. Everywhere else the headroom charge decides, and a refusal can come with the disk far from full whatever the reserve. Deleting a collection gives back a slot for each of its volumes, however little it stored, since the headroom charged to them leaves with them; only a server already at its reserve, where every volume is read-only and charged nothing, gets back no more than the bytes. Deleted objects return room only through a vacuum. The room left inside those volumes for further writes is given up and does not return as capacity, since such a volume is charged no headroom while its own bytes still come out of the budget. Lowering the growth count takes nothing away by contrast: no path removes a volume from the writable set for being surplus to the batch size, so an existing collection keeps what it has and converges as those volumes fill. [`docs/operations/seaweedfs-lowering-the-volume-size-limit.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/seaweedfs-lowering-the-volume-size-limit.md) has the check for it, and the way back. The ceiling cannot come out lower at any supported `replicationFactor`, which is the part worth stating plainly. The divisor halves, and the space the slot count is derived from grows at the same time, because a volume past the new limit stops being charged headroom against it. `COPY_1` and `COPY_3` are untouched, so at 1 and 3 copies the limit alone still moves the ceiling upward. ### Reaching the value Neither constant was reachable by an operator. The data-plane chart is pulled by `ExternalArtifact`, and the component publishing it declares no `install` section, so the reconciler drops a `Package`-level values override for it before any merge happens. Editing the chart was the only route, on a value whose right setting depends on the PVC size the operator chose. `master.volumeSizeLimitMB` is on the app chart now, with a schema entry. Worth being precise about which file the two numbers end up living in, because exposing one of them moves it. The app chart's HelmRelease sends the size limit on every release, so `packages/extra/seaweedfs/values.yaml` holds the 400 installs run with, and `packages/system/seaweedfs/values.yaml` keeps upstream's 1000 as the fallback a render without it gets. That fallback is deliberate: a platform upgrade can re-render the data-plane release from its artifact before the parent has sent anything new, and a lower value there would then reach the master first and drop the limit ahead of an operator's own setting. Nothing sends the growth count, so the data-plane chart's value is the live one for that half, and it is the part that does land in that window, where a grow costs 3000 MB against the 6000 MB it cost before. The budget the pair produces is written out next to the growth count, with a pointer from the size limit to it, so the two do not drift apart. The field is bounded at both ends, because either end takes an instance down. Zero is not a smaller limit but an off switch: the volume server leaves its slot calculation untouched at zero, keeps reporting no capacity, and nothing is ever grown. Past 30000 the master does not start at all, exiting on `volumeSizeLimitMB should be smaller than 30000`. The bound lives in the values schema, through the generator's `@minimum` and `@maximum` annotations, so Helm refuses the value before any template runs. The apps API reads that schema only for defaults, so it accepts a value out of range, and the release then fails to render. A case pins each edge along with the value just outside it. An explicit null gets past both bounds, because Helm drops a null key before it validates, and would reach the master as an empty `-volumeSizeLimitMB=` that the shipped image exits on at start, so the template requires the value and a case pins that as well. There is no second bound against the disk, and not for want of trying: a limit at or above the volume's own size does not break an instance. `MaybeAdjustVolumeMax` floors the slot count at one, and a 255 MiB disk against a 1000 MB limit reported `Max 1`, took an assign for a fresh collection and served back what was written to it. Such a server is capped at a single slot rather than dead, so a render refusal would stop a working release on every reconcile. The field description says instead that a value close to or above `volume.size` leaves each volume server a single slot. The growth count is deliberately not exposed beside the limit. A field wired to `COPY_2` would do nothing at 1 or 3 copies, for the reason above, while still looking like a setting, and a knob that is silently inert for two of the three supported values is worse than no knob. Making it track the replication factor is a feature with its own matrix, and it would rest on a path where the master's own default replication is separately broken. The size limit has no such dependency, so it is the one that ships. ### Picking a value 400 is sized for the shipped `volume.size: 10Gi`, which is where the slot and collection figures above come from. Going up, it holds until the two descriptors per volume meet a 65536 `nofile` limit, roughly 12.5 TiB per volume server. Going down, an instance needs 1600 MB of filesystem per volume server to serve a second collection at all, since the first claims the opening three slots. ### What the tests cover The existing bucket case creates one bucket, which sits inside the ceiling at any sizing, so it could not tell a working ceiling from an exhausted one. The new e2e case creates three beside the platform's own backup bucket and requires each to accept an object, which is the only signal the exhaustion produces. Each `mc` call there carries its own timeout, since a hung PUT is exactly that signal and the step timeout alone would not end it. It was observed red at the old sizing in [run 34948279363](https://github.com/cozystack/cozystack/actions/runs/34948279363), which ran the case against the unchanged constants: the pre-existing single-bucket case passed in 18s beside it, and the new one failed on the write to the second bucket after the first succeeded. Three rather than two so the count clears the ceiling without depending on how many slots the unnamed default collection holds, which the suite cannot observe. The unit tests pin each constant on its own, because losing one is a silent capacity regression rather than something a render would catch. The app-chart test asserts an overridden size limit as well as the default, since asserting the default alone passes just as well against a template that ignores the value and writes the number as a literal. It also pins both ends of the accepted range and the first value outside each, the `-max 0` zero on all four render paths it is written to, and the 10Gi volume default the slot figures above are measured for, which nothing else guarded. The data-plane suite pins the five percent free-space reserve as well, since the 8000 MiB threshold above is computed from it and nothing else reads it. The capacity case also reads the master's slot accounting in its `catch`. A grow failure in the master log does not say whether the shipped limit is too coarse for the PVC or an earlier collection in the same run is still holding the budget, and the per-server `Max` against `Volumes` does. Closes #4239, closes #4245, closes #4248 ### Screenshots No UI change. ### Downstream repositories Walked against the diff. The new field lands on `packages/extra/seaweedfs`, and the provider models bucket, generic, qdrant, redis and tenant, with no SeaweedFS resource among them, so there is no schema or expand/flatten pair over there for it to drift from. No package is added, renamed or removed, so the website app lists are untouched. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(seaweedfs): raise how many buckets a new SeaweedFS instance can serve on the default 10Gi volume PVC from two to eight at `replicationFactor: 2`, by defaulting the master volume size limit to 400 MB instead of 1000 and lowering the two-copy growth count from 6 to 3, and expose the limit as `master.volumeSizeLimitMB` on the app chart. **Action required on the `default` variant only:** it runs no platform migrations, so before upgrading, set `master.volumeSizeLimitMB: 1000` on every SeaweedFS instance that has no valid value and whose `master` is an object, with the step in [`docs/operations/seaweedfs-lowering-the-volume-size-limit.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/seaweedfs-lowering-the-volume-size-limit.md). If this is skipped, every such instance drops to 400 on the upgrade, and a bucket whose volumes have all reached 400 MB takes no writes until it can grow. On the variants that run platform migrations, migration 57 does this: it sets `master.volumeSizeLimitMB: 1000` on every instance that has not chosen a value, except one whose `master` is not an object, which no v1 chart renders and which it names in its Job's log instead, so nothing changes for an instance unless it already names a different value, which the previous chart accepted and ignored and this one applies; the same guide lists such instances before the upgrade, and the migration names them in its Job's log. An existing value that is not a whole number from 1 to 30000 is replaced with 1000, which is what the previous chart applied in its place. Instances at 1000 still get roughly twice the room for new buckets from the lower growth count. The upgrade rolls every SeaweedFS master StatefulSet once, since the growth count is part of the master's environment, and an instance that names a value other than 1000 rolls a second time when that value arrives. The pin lives in the instance's spec, so a tool that replaces the whole resource from a manifest without `master.volumeSizeLimitMB` removes it and the instance takes 400; add the field to such manifests before upgrading. Lowering the limit on an instance that already holds data takes every volume at or above the new value out of the master's writable set, and raising it takes the room new buckets grow into, so the same guide describes the free-space check to run first and how to set it back. ```