mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-30 16:47:20 +09:00
11bac54f931682dd94de4081bb9e8a7eef60bef1
413
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Alexey Artamonov
|
80d32b3ab7 |
fix(keycloak): complete realm-name coverage and guard non-string names
The enable-update-email hook Job in keycloak-configure still pinned REALM=cozy, so on a custom realmName it waited on a realm the chart never provisions and the cold install timed out — the one seventh consumer left behind, and the only value nobody could set without breaking the install. A bare-integer or YAML-reserved realm name reached the printf %s that builds the derived keycloakInternalUrl as a Go int, rendering a "%!s(...)" format error into the URL; coerce the name to a string once at the source. Document that renaming the realm on a running cluster provisions a fresh empty realm and orphans the accounts in the old one, and note in the OIDC tenant and grafana guides that the realm name is now the configurable default. info/Makefile carried NAME=etcd, so cozyhr drove the wrong release from that package; correct it while touching the file. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
44a9a8e735 |
feat(platform): make the Keycloak realm name configurable
The realm name "cozy" was hardcoded in keycloak-configure (which owns the realm, its clients and the cozystack-cluster-admin roles), in the dashboard and linstor-gui gatekeepers, in the info kubeconfig, and in the Grafana and managed-Kubernetes apiserver OIDC issuers. Operators could not rename the realm without patching vendored templates, which `make update` reverts. The apiserver OIDC issuer is already configurable via talm oidcIssuerUrl, so pointing it at a custom realm left every SSO component on "cozy" and broke login with an issuer mismatch. authentication.oidc.realmName (default "cozy") now rides the _cluster channel so every consumer resolves the same realm, and the default keycloakInternalUrl is derived from it so the realm is named in one place. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
a2db2aeb8c |
fix(seaweedfs): raise the number of buckets a volume server can serve
A volume server started with -max 0 takes a slot count derived from disk rather than a fixed one, replication 001 spends a slot on each of the two servers per logical volume, and a collection claims COPY_2 volumes whenever it grows. So the bucket ceiling is set by the product of the size limit and COPY_2, and moving either number alone moves it. The mechanism is upstream in MaybeAdjustVolumeMax (weed/storage/store.go) and findVolumeCount (weed/topology/volume_growth.go). Do not divide these out of the nominal PVC size, because the count follows the space the filesystem makes available and not the volume's advertised size: the shipped 10Gi volume PVC was seen to give nine slots per server at the upstream 1000 and 19 at 500, which puts 400 at 23 to 24 by the same division, or eight collections against two. The span is the observation's own, since the free space behind 19 at 500 fits more than one filesystem geometry and those divide differently at 400; eight either way, because a collection that cannot get a whole batch of COPY_2 keeps the volumes it did get and writes into them. The division itself is measured: on a 1 GB image the master reports max as the filesystem's available space over the limit in every run, 25 slots and nine collections at the 40 and 3 that scale to this pair. Two is what the platform's own cozy-backups bucket and a single tenant bucket already use. Past that every S3 PUT to a new bucket times out while the disk still reads as nearly empty, and no COSI object reports anything wrong, because the claim and the access are both granted. 400 and 3 make the budget 1200 MB per collection. Splitting the reduction across both numbers rather than taking it out of the size limit alone keeps the cost off the curve that grows: open descriptors track stored bytes divided by the limit, so a 10 TiB server keeps roughly 52k of them at this limit against 21k upstream, where moving the size limit alone would have put it at 105k. A .dat and an .idx stay open per volume; the .vif beside them is written and closed. The ceiling reached is the same either way. Both figures describe the shipped replicationFactor of 2. The master picks its growth count from the replication the filer asks for, so at 1 and 3 copies it reads COPY_1 and COPY_3 instead and this COPY_2 does not apply; lowering the size limit still raises the ceiling there. The size limit was out of an operator's reach: it lived in the data-plane chart, which is pulled by ExternalArtifact from a component that declares no install section, so a Package-level values override for it is dropped before any merge. The app chart now declares master.volumeSizeLimitMB and sends it on every release, and 400 is set there. The data-plane chart keeps upstream's 1000 as its fallback on purpose. A platform upgrade can re-render that release from its artifact before the parent has sent anything new, and a lower fallback would then reach the master first and drop the limit ahead of an operator's own setting. COPY_2 is sent by nothing and lives in the data-plane chart, so it does land in that window, where a grow costs 3000 MB against the 6000 MB it cost before. The app chart also writes maxVolumes into dataDirs on every release, and because dataDirs is a list Helm replaces the data-plane chart's copy wholesale, so the zero that makes the slot count auto-derived at all comes from there. Losing it drops the server to the subchart's fixed seven without failing any render. The field is bounded at both ends, because either end takes an instance down. Zero is not a smaller limit but an off switch: the volume server leaves its slot calculation untouched at zero, keeps reporting no capacity, and nothing is ever grown. Past 30000 the master does not start at all. The bound sits in the values schema rather than in a template guard, so Helm refuses the value before any template runs. The apps API does not: it reads that schema only for defaults, so it accepts a value out of range, and the release then fails to render. An explicit null gets past both bounds, because Helm drops the key before it validates, and would render an empty -volumeSizeLimitMB= that the master exits on at start, measured on the shipped image, so the template requires the value. No bound against the disk goes with it, though the field invites one. A limit at or above the volume's own size does not break an instance: MaybeAdjustVolumeMax floors the slot count at one, measured on 4.31 with a 255 MiB disk against a 1000 MB limit, which reported Max 1 and served a write it then read back. Such a server is capped at a single slot rather than dead, so refusing it at render would be a capacity policy that stops a working release on every reconcile, not only on the upgrade. The field's description carries the relation instead, where the value is chosen. The growth count is deliberately not exposed beside the limit. The master picks which of its four growth counts to apply from the replication the filer requests, and the filer takes that from replicationFactor, so a field wired to COPY_2 would do nothing at 1 or 3 copies while still appearing to be a setting. A knob that is silently inert for two of the three supported values is worse than no knob. Two further prices. A collection absorbs the budget before it needs another grow, so growth requests arrive five times as often as at the upstream pair. A collection also starts with half as many writable volumes, and a volume server takes the writes into each volume one at a time (syncWrite holds that volume's dataFileAccessLock), so a heavily parallel writer spreads over fewer volumes until the collection grows again; that holds for instances kept at 1000 too. And lowering the limit on an instance that already holds data carries the other one: its volumes between the new limit and the old go oversized on the next heartbeat and leave the master's writable set, and each affected collection needs one successful grow before its next write lands. Nothing is lost and the volumes stay readable; the room left inside them for further writes is given up and does not return as capacity. Those grows compete for one pool, and the pool is smaller than it looks. An oversized volume is charged no headroom, while one below the limit is charged all of its unfilled room; a grow then claims a whole COPY_2 batch and takes as much of it as is there, and the volumes it creates are empty and charge their headroom straight back. Measured on a local SeaweedFS 4.31 setup: two oversized collections with two free slots between them, the first to write took both, and the second stayed refused with the disk still reporting a quarter of itself free. Collections that need no grow are not free either, since their volumes below the new limit were created six at a time and each can hold back a whole limit. Also measured there, with the limit and batch scaled down: a server with free space for one batch per collection refused the grow its one affected collection needed, because a second, nearly empty collection held the room, and the same free space without it let the grow through. The minFreeSpacePercent reserve only comes into it on a server whose writable volumes have all passed the limit, where nothing is charged headroom: from 8000 MiB of filesystem upward, a 10Gi PVC measuring 9979, five percent is at least 400 MB, and such a server then refuses a grow only once its disk is down to the reserve. Everywhere else the headroom charge decides, and a refusal can arrive with the disk far from full whatever the reserve. Deleting a collection gives the room back a slot per volume, however little it stored, since the headroom charged to those volumes leaves with them. So, on a variant that runs platform migrations, the upgrade does not change the limit on an existing instance unless its operator already set one, which the previous chart accepted and ignored. Platform migration 57 writes master.volumeSizeLimitMB: 1000 into every existing SeaweedFS instance that has not chosen a limit, and 400 applies only to instances created from here on. It leaves alone only an instance whose master is not an object, which no v1 chart renders, and names that one in its log. The default variant runs no platform migrations, so a cluster there that installed SeaweedFS by hand gets no pin, and the document below gives the same patch to apply before upgrading. A chosen limit is a whole number the new chart accepts; anything else counts as not chosen and is replaced with 1000, which is what the previous chart ran in its place, since it ignored the key and the new chart would refuse to render it. A valid value already set is kept, and since the previous chart ignored it, the upgrade is when it starts to apply; the document below lists the ones that change the limit ahead of it. Raised values are listed as well: existing buckets keep writing, but every volume below the new limit is charged up to it, and on a local SeaweedFS 4.31 setup seven nearly empty volumes left no free slot once the limit went up tenfold, so a new bucket was refused. The value goes into the application's HelmRelease, whose spec.values is the application spec, for an instance created through the API and for one the tenant module renders alike. The tenant module never renders spec.values on that release, and a Helm upgrade leaves a key alone that neither manifest carries: measured with Helm 4.3.0 on client-side, server-side and forced server-side apply, and read in createPatch at the v4.1.1 that helm-controller v1.5.0 runs. Where the migration runs, it is a pre-upgrade hook of the platform release, and the cozystack-packages source every application chart comes from is a manifest of that same release, applied only after its hooks; so no instance meets the new chart unstamped, and a failed migration stops the platform upgrade before that source moves. That holds while the platform source is pinned by digest, as it is out of the box; one that follows a tag, a semver range or a branch lets source-controller move cozystack-packages on its own schedule. Until cozystack-packages moves, each patched release keeps its old chart, which ignores the key, since no v1 release reads it or closes the master object in its schema. An instance kept at 1000 still gains, because at a replication factor of 2 a grow now claims three volumes rather than six, so it has room for roughly twice as many new buckets. Lowering an existing instance is left to its operator, and docs/operations/seaweedfs-lowering-the-volume-size-limit.md is the check for it. It lists each server's volume files and adds one limit for every volume below the limit and a batch for every collection with no volume at or below 90 percent of it, since the master grows such a collection on its own schedule without a write; measured the same way, a collection with every volume in that band and no writes got two new volumes per server within one pass. Where each disk type is served by as many volume servers as the replication factor, as the shipped two at a factor of 2 are, each collection's volumes sit on every server of that type, and that bounds the claim from above without assuming how the collections grew. The master places a new volume on any server of the requested disk type with room rather than where its collection already lives, whichever group or pool that server belongs to, and it reads an empty disk type as hdd; under MultiZone it also grows each zone on its own. So the document keeps zoned instances, and any disk type served by more servers than copies, at 1000. BREAKING CHANGE: on the default variant, which runs no platform migrations, set master.volumeSizeLimitMB to 1000 before upgrading on every SeaweedFS instance that has no valid value and whose master is an object, with the step in docs/operations/seaweedfs-lowering-the-volume-size-limit.md. On the variants that run platform migrations, migration 57 does this. If the step is skipped, every such instance drops to 400 on the upgrade: each of its volumes at or above 400 MB leaves the master's writable set, and a bucket whose volumes have all crossed that line takes no writes until it can grow. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
ae5b626c8f |
docs(gateway): describe the passthrough listeners and what they shed
The chart README gains the native-port field, the three limitations it carries and where each comes from, the one that is a phase-1 narrowing rather than a consequence of the Cilium pin, and the three ways forward for anything that depended on a terminate listener the change takes away. The withdrawal is stated as what it is, a change of serving path rather than the removal of something idle, because on the pinned version both filter chains carry the SNI today and nothing on the objects says which one answers. The layers section is brought back in step with the listeners: which label each listener class selects on, that the attach set is the whole inheriting tenant subtree rather than the tenant plus the admin list, and the route conditions an operator reads, including the one that says the sectionName itself names nothing. The platform values comment on gateway.enabled said every exposed service renders an HTTPRoute, which the chart value the withdrawal reads contradicts. It now separates them: the services whose TLS the Gateway terminates get an HTTPRoute, while api, vm-exportproxy and cdi-uploadproxy get a TLSRoute pinned to their passthrough listener, one each from the packages that own those services. The overlap paragraph no longer describes an HTTPRoute wildcard as keeping its terminate listener. Such a claim is refused under HTTP-01, which issues no wildcard certificate, and the paragraph says so and what the render used to do with it. The route-condition list gains that shape and the port shapes an HTTPRoute now reaches, and the per-app naming bullets carry the hash suffix the rendered names have. Two sentences that were false against the code are corrected: a TLSRoute on a wildcard entry itself leaves the names beneath it their terminate listeners, and an unusable apex is refused by the containment rule at admission before the controller's own check is reached. The edge section says what happens to the entries the HTTP-01 pass wrote once the mode moves. The NoMatchingParent sentence now separates the two kinds: an HTTPRoute pinned to an absent passthrough section is refused whatever answers its hostname, while a TLSRoute is judged where a passthrough listener answers it. A sectionName outside the passthrough prefix is written Accepted like any attached route rather than left without a verdict, and the text says so; the port-80 listener's refusal joins the NotAllowedByListeners shapes and the name-and-port disagreement the NoMatchingParent ones. The attachedNamespaces comment described the list as reaching every listener, which the native-port ones do not: they select on kubernetes.io/metadata.name and name the publishing tenant alone, so a namespace listed there cannot attach to a database port. It also credited the wrong label with keeping the list honest; what a listed entry gets is the namespace.cozystack.io/gateway label, and what keeps a tenant from stamping it on itself is that no tenant role carries patch on Namespace. The class reference is brought along. Its certificate-mode table promised one Certificate per published hostname under http01, which a hostname served by a passthrough listener no longer earns, and its edge table said every attached TLSRoute is refused, where only a route declaring hostnames is and the rest have this controller's entry removed instead. The refusal-shape list is corrected against the code, without a count of its own so the next correction does not have to move one. A route the plain-HTTP listener turns away on its namespace selector was listed under NoMatchingListenerHostname as well as under NotAllowedByListeners; the controller reports the second. The port shape standing beside it swept port 80 back in by naming every port but 443, where a route selecting the plain-HTTP listener is answered by the namespace shapes and never reaches the port refusal at all. And a TLSRoute pinned to an absent section is said to carry NoMatchingListenerHostname where no passthrough listener answers its hostname, which holds only while no HTTPRoute claims that hostname: with one there a terminate listener stands on the name, and the route is judged like any other attached route. The wildcard shape gains the same qualifier. A wildcard claim is refused because no terminate listener can carry a certificate HTTP-01 will not issue, so the refusal reaches a route that wanted a terminate listener and not one that asked for the plain-HTTP listener, which carries no hostname and serves the claim there. The Helm merge paragraph named the two-way path as what happens to a custom resource Helm has no Go type for. That path is one of two in createPatch, the version deciding which runs, and the version comes from helm-controller rather than from this tree, so the claim was not checkable where it was written. It now names both and says the outcome it rests on holds either way. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4b2507b7a1 |
docs(gateway): align gateway prose with the code
The comment on the shared CEL string said what the hostname requirement is but not what it is worth per kind. Two of the three schemas leave spec.hostnames optional at every version the vendored bundle serves, so for HTTPRoute and GRPCRoute this policy is the only rejection a hostname-less route meets; tlsroutes requires the field at v1 and v1alpha3 and not at v1alpha2, so there the policy closes the gap for as long as v1alpha2 is served. State that where the requirement is stated. The comment on the port-80 allowedRoutes filter credited the cert-manager namespace entry for letting ACME complete, which the paragraph above it retracts: the solver route lives in the tenant namespace, and that is what keeps ACME working. The same sentence claimed the filter keeps app routes off port 80, and the Layer 1 paragraph of the gateway README claimed it too. It does not. A tenant that owns its Gateway is its own gateway namespace, so its app routes are admitted by this very filter; what it excludes is cozy-* and inheriting children. Both copies now say that. Nothing said why naming the apex on the redirect costs no coverage, which is the question a reader has once the route stops matching every host. It holds because of the other half of this change: the port-80 listener admits routes only from the tenant namespace and cert-manager's, and the tightened route policy requires every route there to declare hostnames inside the apex. An inheriting child does not widen that set either, and the comment now says so at the point the hostnames are chosen: the gateway hostname policy compares every listener against the host label of the Gateway's own namespace, so a child apex outside it never gets its listener, the Gateway write carrying it being refused at admission whole, in reconcileGateway and before this route is rendered, while a child apex under this one passes that check and the wildcard already covers it. The empty-apex guard in renderHTTPRedirect, and the test pinning it, both read as though the guard is what spares an operator an unreadable schema error. It is, but the condition stated for it named the number of attached routes, which does not enter into it. In HTTP-01 the listener set is built from route hostnames alone, so reconcileGateway never reads the apex ahead of the guard however many routes attach. DNS-01 and existingSecret render an apex listener, and any tlsPassthroughServices entry builds <service>.<apex>, so in those the operator meets the equivalent error against the Gateway first. State that condition once, at the guard, and let the test defer to it rather than carry a second copy that can drift; the test stays green either way because the fake client does not validate against the CRD schema. The e2e suite explained omitting the TLSRoute empty-list probe by the v1 schema rejecting an empty list. Both halves of that are true and the conclusion does not follow: the probe runs at v1alpha2, which marks spec.hostnames neither required nor minItems, so an empty list reaches admission there exactly as an absent field does. What makes the omission safe is that all three policies render one byte-identical CEL string, so the HTTPRoute pair proves that term already. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
900711567a |
fix(cozystack-basics): require route hostnames in tenant namespaces
The route hostname policy short-circuited on an absent field. Its CEL opened with `!has(object.spec.hostnames) ||`, so an HTTPRoute, TLSRoute or GRPCRoute declaring no hostnames satisfied the first operand and was admitted without a single hostname compared against the tenant apex. An explicitly empty list reached the same outcome by a different path, because `all()` over an empty list is vacuously true. A route that names no hostname does not match nothing. Under Gateway API it inherits the hostnames of every listener it attaches to, so it covers whatever that listener covers. Require the field to be present and non-empty before validating it. All three policies rendered by the template share one CEL string, so HTTPRoute, TLSRoute and GRPCRoute pick the change up together. The denial message now names the missing-hostnames case, and the Gateway e2e grep moves to the part of that message common to every denial it issues. The Gateway e2e suite carried the one fixture that relied on the old behaviour: its parentRef probe declared no hostnames and is applied into the suite's tenant namespace. It now reads the apex from the namespace label and names a hostname under it, as the neighbouring cases in that suite already did. Radius is unchanged: the policy still matches only tenant-* namespaces. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
5976ddf02f |
fix(gateway): normalise the GatewayClass name and refuse a blank one
The class lists arrive over the values channel comma-joined, so every reader trims each entry as it splits them. The scalar those entries are compared against was not trimmed anywhere. A class name carrying a stray space therefore failed `has` against its own padded self, and the two readers of that scalar disagreed about the same class: the gateway chart put the tenant on an edge-terminated class, while the cluster-wide ACME issuers still pinned their solver to a listener an edge Gateway does not render. That stops every HTTP-01 certificate in the cluster from issuing, with nothing in any status naming the cause. Normalise once at the single writer so both readers see a clean value, and keep a defensive trim at each reader. The tenant's own choice does not travel that channel — it comes from tenant.spec.gatewayClass — so it is trimmed where the gateway chart reads it. Unset and whitespace-only are different events and no longer share an outcome. Unset means the choice was not made, and falling back to the bundled class stays its documented behaviour. Whitespace-only means a choice was made and is garbage, in the field that decides where TLS terminates: resolving it quietly would take a tenant the operator meant to put behind an edge terminator and publish it directly, with a real certificate on a real address. Both the platform default and the tenant's own choice now refuse it by name. The kind check beside that refusal is part of the same fix, not a tidy up. Through Helm's values path an unquoted numeric scalar arrives as a float64, and the whitespace check compares it against a string, so `className: 123` dies at that comparison — one line before the trim — with an error naming neither the key nor the file. Refusing whitespace without checking the kind first would therefore have made the numeric case worse than it was. `kindIs "string"` is the right test because it never has to name the type that actually arrived. The order of the guard's steps is part of the guard. The value is read before anything normalises it, so a default that fires on any falsy value cannot launder 0 or false into the empty string and resolve them to the bundled class ahead of the kind check. Unset is the one case that still falls back, and each of those four inputs is pinned, so a normalisation drifting back ahead of a check fails rather than quietly covering only the loudest bad input. Also correct a cleanup log line: the wildcard Certificate is deleted for every mode that is not DNS-01, so naming HTTP-01 in that message is wrong for existingSecret and for edge alike. And state the listener name invariant plainly: the cap is 253, not the 63 of a DNS label, and the longest name the builders produce is 84. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
7851969448 |
docs(gateway): document GatewayClass selection and edge TLS termination
An operator setting up a class whose provider terminates TLS upstream had no page answering their questions: which platform values bound the choice, what a tenant writes to make it, what a move between classes does to certificates and to the replicated wildcard Secret, and how the mechanism fails when a configured class name matches nothing installed. Add docs/gateway-classes.md covering that surface. Describe the mode and its consequences in the chart README alongside the other three, including the one constraint stated nowhere else: the publishing tenant cannot select a class whose TLS termination differs from the platform default, because the cluster-wide ACME issuers derive their solver pin from that default alone. State the blast radius where it actually applies. The endpoints an edge Gateway stops publishing exist only where those routes are rendered at all, and both pages say what the transition does to the route conditions the controller owns, so a reader is not left inferring it from the code. The security model names the selector each listener really carries. The port-80 listener selects on the built-in kubernetes.io/metadata.name label, which kube-apiserver writes and a namespace cannot forge; the HTTPS and TLS-passthrough listeners select on namespace.cozystack.io/gateway, written by the tenant chart and the controller, so that guarantee does not extend to them. What closes the exposure is that Namespace is cluster-scoped with no tenant verb on it, and that the route-hostname policies bound what a labelled namespace could publish. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
487cafac3e |
feat(platform): let a tenant pick its Gateway class from an allowlist
Tenant Gateways were pinned to the bundled Cilium class. Give the platform a default in gateway.className, the set a tenant may choose from in gateway.tenantSelectableClasses, and the set whose provider terminates TLS upstream of the Gateway in gateway.edgeTerminatedClasses. A tenant picks its own with tenant.spec.gatewayClass, read only when it owns a Gateway; an inheriting tenant runs on its ancestor's. The selectable set is an allowlist rather than free choice because Tenant is a tenant-writable object while a GatewayClass is cluster-scoped: without it a tenant could attach its Gateway to a class an admin keeps for internal-only traffic. Naming anything outside the set fails that tenant's own gateway release and reaches no other tenant. Where TLS ends is a property of the class, so the gateway chart derives certMode from whichever class the tenant landed on instead of taking it as a separate input. The two therefore cannot be configured out of step. The publishing tenant cannot cross that line by itself. The cluster-wide ACME issuers point their HTTP-01 solver at its Gateway and are rendered from the platform default alone, so a mismatch would pin the solver to a listener that Gateway does not have. Those issuers now drop the sectionName pin when the default class is edge-terminated, letting the challenge attach by hostname instead. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
af19aa933c |
docs(security): record GRPCRoute in the route hostname policy set
The port-443 kinds set still excludes GRPCRoute, TCPRoute and UDPRoute. cilium#45559 governs only that every port-443 listener carries the same set, not what is in it, so the contents are a least-privilege choice: nothing the platform ships needs gRPC, TCP or UDP routing on port 443. TCPRoute and UDPRoute additionally carry no hostname and no admission rule gates them. The gateway README described HTTPS and TLS-passthrough listeners as carrying different kinds. They carry the same one, which is what cilium#45559 requires. The threat model and the self-assessment enumerate which route kinds the hostname policies cover, and both listed two. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
70a675fd6b |
feat(cozystack-basics): gate GRPCRoute hostnames in tenant namespaces
The route hostname policy covered HTTPRoute and TLSRoute. GRPCRoute carries spec.hostnames too, and it can reach a tenant Gateway: the port-443 listeners pin allowedRoutes.kinds to HTTPRoute and TLSRoute, but the port-80 listener leaves kinds unset, and an unset list resolves to the kinds the listener protocol supports, which for HTTP includes GRPCRoute. A GRPCRoute in a tenant namespace could therefore claim any hostname on the plaintext listener with nothing checking it. Tenants hold no gateway.networking.k8s.io RBAC, so this is the same defence-in-depth against a chart bug that the other two rules are, not a tenant-user control. The new rule is the HTTPRoute rule with grpcroutes in place of httproutes and the single served version the bundle provides; the coverage guard pins that list to the bundle like the others. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7ba22369d4 |
fix(cozystack-basics): match every served TLSRoute version in the hostname policy
The tenant route hostname policy names v1alpha2 alone while the vendored Gateway API bundle serves tlsroutes at v1, v1alpha2 and v1alpha3: three served versions against one named. The gap is not new. The v1.5.1 bundle this package shipped before and the v1.6.1 one it ships now serve the same three versions, with the same two marked deprecated, so nothing in this change created or widened it -- a bundle bump is simply when someone last opened the file. Coverage of the two unnamed versions rested entirely on request conversion. matchConstraints.matchPolicy is unset on both rules in this file and so defaults to Equivalent, which rewrites a request submitted under any served version into the one the rule names before the policy sees it. That is why there is no bypass to demonstrate, and equally why the arrangement is brittle: the conversion has nothing left to convert into once no named version is served, and a rule that selects nothing is not caught by failurePolicy: Fail, because nothing failed -- there was simply no match. Name all three served versions, which is the shape the HTTPRoute rule directly above already uses. The defect is a version list drifting from the bundle it must track, so a longer list on its own would not catch the next drift: add a check that reads the served versions out of the vendored bundle and fails when a policy does not cover every one of them, in either direction. Both rules get an exact-value assertion in the chart's own suite, so the coverage check and the pinned value fail for different reasons and neither stands alone. The header comment being replaced claimed TLSRoute stays at v1alpha2 through Gateway API v1.4-v1.5 and that its promotion to v1 was still upstream-only. TLSRoute has been served at v1 since v1.5.0, so that was already false, and false in the direction that hid this gap: a reader who believed it had no reason to look at the version list at all. Widening the rule then falsified every remaining place that restated which versions it matches -- the template's own header, the security-model layer list in packages/extra/gateway/README.md, and the name of the test case that asserts the value. Rather than resync three copies of a list that will move again, they now say which kinds are gated and point at the rule and its guard for the versions. A comment in the same test file now states only the behaviour it guards: an earlier form of the CEL used `... ? true : (...)`, which allowed routes in any tenant-* namespace whose host label was missing or scrubbed. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Alexey Artamonov
|
6fdb5e0840 |
docs(etcd): note that affinity is not applied to existing members
The operator applies spec.affinity only when it creates or replaces a member Pod; it does not reschedule running members. Document that on the `affinity` value so operators do not expect the default (or a newly set rule) to redistribute already co-located members until each is replaced. Addresses the non-blocking review notes on #3962. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
scooby87
|
94614c021a |
fix(etcd): spread members across nodes with default soft pod anti-affinity (#3962)
## What this PR does The `etcd` chart hardcoded only a **soft** `topologySpreadConstraints` (`whenUnsatisfiable: ScheduleAnyway`) and never rendered `spec.affinity`, even though the `EtcdCluster` CRD supports it. On small clusters the scheduler therefore co-locates multiple members on one node; losing that node takes out quorum (or the whole cluster). We hit this in production: a single node reboot left several 3-member etcd clusters with 2–3 members on the downed node, and one tenant's control-plane etcd lost quorum until the node came back. This renders `EtcdCluster.spec.affinity` from a new `affinity` value: - **Default (value left empty):** the chart applies a **soft** (preferred) `podAntiAffinity` on `kubernetes.io/hostname`, so members spread across nodes on a best-effort basis. Soft on purpose — a hard rule would leave a member `Pending` when replicas exceed schedulable nodes, or when node-local storage pins a member to a node, so it is safe to default on. - **Override:** set `affinity` to any rule — e.g. a hard `requiredDuringSchedulingIgnoredDuringExecution` podAntiAffinity — to forbid co-location outright on clusters with enough nodes and non-node-pinned storage. A supplied value replaces the default cleanly. The existing `topologySpreadConstraints` block (and its `_cluster.scheduling.globalAppTopologySpreadConstraints` override) is unchanged. Generated artifacts (`values.schema.json`, `README.md`, and the embedded `openAPISchema` in `etcd-rd`) were regenerated with `make generate`; `make test` passes (18/18). ### Screenshots n/a — no UI changes. ### Downstream repositories Walked the trigger map against the diff: this only adds one values key to an app chart. No package is added/renamed/removed, `kind`/`plural` are unchanged, no installer values or release assets or variants change, and `README.md` is regenerated here (the website bot regenerates reference pages from it on release). No manual downstream follow-up is required. - [x] No downstream repository is affected by this change ### Release note ```release-note fix(etcd): etcd clusters now spread their members across nodes by default via a soft podAntiAffinity, and expose an `affinity` value so operators can set a hard rule; previously a single node could host a quorum of members and take out the cluster when it failed. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable pod affinity rules for etcd deployments. * By default, etcd members use soft anti-affinity to spread across hosts without blocking scheduling. * Custom affinity settings can override the default behavior. * Exposed affinity configuration in the etcd application schema and dashboard. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
9858fb63ac |
test(etcd): cover default and overridden affinity rendering
The affinity value added in this branch introduced a two-branch conditional in templates/etcd-cluster.yaml (default soft podAntiAffinity vs. a supplied value) with no coverage. Add a helm unittest suite pinning both branches: the default preferred per-host rule, and that a supplied affinity replaces the default outright (no preferred rule leaks through). Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Alexey Artamonov
|
c83a22e077 |
fix(etcd): spread members across nodes with default soft pod anti-affinity
The etcd chart only set a soft topologySpreadConstraints and never rendered spec.affinity, so the scheduler could place several members of a 3-node EtcdCluster on the same node; losing that node then took out quorum (observed in production after a single node reboot). Render EtcdCluster.spec.affinity from a new `affinity` value. When it is left empty the chart applies a soft (preferred) podAntiAffinity on kubernetes.io/hostname, so members are spread on a best-effort basis without ever leaving a member Pending on small or node-pinned-storage clusters. Operators can override the value — e.g. a hard requiredDuringScheduling rule — where node count and storage allow. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
IvanHunters
|
bf38655ab5 |
docs(computeplane): nodeGroups allowlist drops unlisted fields, not pass-through
The nodeGroups docs still described the old denylist behaviour: "unlisted NodeGroup fields pass through to the wrapped chart unvalidated". After the switch to an allowlist (pick) both halves are wrong: fields go to the kubernetes-nodes pool chart (not the wrapped kubernetes chart), and anything the pick does not name is dropped, not passed through. A tenant that sets e.g. talos.version on a nodeGroup gets no error and no effect (accepted-but-inert). Correct the source @param and the hand-written README intro, regenerate schema/README/cozyrd, and refresh the stale "omit" wording in the splice test comment to say allowlist. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
3630842172 |
fix(computeplane): reject non-RFC-1123 pool keys at render time
The pool key becomes the pool HelmRelease name and its CAPI object names, which must be RFC-1123 labels, but only the length was guarded. An uppercase/underscore key rendered an invalid metadata.name the apiserver rejects, and a key containing ':' rendered invalid YAML that failed the whole module render -- a non-uniform, late failure next to the guarded length case. Add a render-time RFC-1123 guard on the key alongside the length guard (mirrors migration 56's child-name guard), and pin the length guard's derived remedy budget (15) in the overflow test so a wrong computation is caught, not just the presence of the error. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
c2ee58756e |
fix(computeplane): allowlist pool values instead of denylisting, derive length-guard prefix
The pool-value splice used omit (a denylist), which leaks every key it does not name. Since this module runs untrusted tenant code and the NodeGroup schema has no additionalProperties:false, a tenant could pass the platform-injected _cluster object (cluster-domain/oidc/dns01/proxy, delivered via cozystack-values and merged UNDER spec.values, so a tenant key wins) or nameOverride straight through to the pool chart -- and any future kubernetes-nodes top-level value would silently re-open the same hole. Switch to pick: pass through only the enumerated per-pool shape fields, so everything cluster-level or module-owned is dropped by construction (fail-closed for new fields). Also derive the length-guard's release-name prefix from $clusterRelease so it cannot drift from the name actually rendered. Regression-tested: _cluster and nameOverride no longer reach the child values; maxUnhealthy and the module wiring still do. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
34e801cb9e |
fix(computeplane): stop tenant pool values leaking cluster-level keys and overflowing release names
The per-pool child HelmRelease spliced the tenant nodeGroup dict verbatim (toYaml) after setting cluster/clusterReleaseName, and the NodeGroup schema has no additionalProperties:false. A tenant could therefore (a) set a duplicate cluster/clusterReleaseName key that wins the YAML merge and re-parents the pool onto a tenant-controlled cluster, or (b) pass talos/ version/images through a shape-only knob and pick the worker OS image, installer and kubelet version. omit those five keys at the splice so only the pool's own shape flows through (tolerated extras like maxUnhealthy still do). Separately, each pool renders a 38-char fixed release-name prefix against Helm's 53-char cap, so a nodeGroup key over 15 chars created a HelmRelease the apiserver accepts but Flux never reconciles, with no other signal. Guard the length at render time, next to the ingress-role guard, mirroring the migration overflow branch and the aggregated API's validateNameLength. Also refresh the generated values doc: the module now materialises its own default md0 and pool storageClass falls back to the kubernetes-nodes chart default, not the wrapped kubernetes app. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
d93e136e23 |
fix(computeplane): provision worker pools via kubernetes-nodes after Phase 2 split
Phase 2 moved worker nodeGroups out of apps/kubernetes into apps/kubernetes-nodes, but the ComputePlane module still passed nodeGroups to the wrapped kubernetes chart, which now ignores it: a fresh ComputePlane cluster came up with zero workers (and the pinned ingress-nginx addon stuck Pending), while existing ones were unreachable by migration 56 (it only adopts kubernetes-* releases, not computeplane-cluster). The module now emits one kubernetes-nodes child HelmRelease per nodeGroup, shaped like a natively created pool, and materialises the pre-split default md0 (roles: [ingress-nginx]) when nodeGroups is empty. The pool chart's clusterName gains an internal clusterReleaseName override so pools attach to the module's fixed computeplane-cluster release, which is off the aggregated API's kubernetes-<cluster> convention; the default keeps every API-created pool byte-identical. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
9d1d6808db |
test(seaweedfs,api): bound both Roles against a respelled rule
Each suite forbade a rule by naming it in full, and helm-unittest compares every field it is handed by value equality, so one respelling walks past. For seaweedfs that is not theoretical. Ungating the cnpg block and adding a verb to the rule (get,list,delete rather than get,delete) is the same grant and more, and the guard stayed green while a Client instance held delete on another app's Postgres cluster. Bound the Role by shape: a Client renders exactly two rules, both in the core group, and the cnpg grant is gated between them, so ungating it adds a third whatever its spelling. The rules themselves stay unpinned because the secret list is long and changes on its own schedule. For cozystack-api, add a count bound beside the existing spelling assert rather than in place of it. The two catch different things and neither covers the other: templates/rbac.yaml carries no template directives, so the list is pinned at exactly sixteen rules, while a grant swapped in for an existing one leaves the count where it was and only the named-rule assert sees it. The count is pinned from both sides — a dropped grant is the failure this suite is about, since the apiserver then returns forbidden and List silently drops the dropdown. Measured on all three: append, delete, and swap. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei Kvapil
|
93338cc434 |
refactor(computeplane): drop the in-cluster Flux pin
The fluxcd addon pin was a holdover from the #3150-era model where app charts rendered nested HelmReleases into the target cluster and needed an in-cluster helm-controller. Consumers now keep their releases on the management cluster and remote-apply via spec.kubeConfig, so the management Flux resolves charts and applies manifests — nothing inside the ComputePlane reconciles HelmReleases. This also decouples the module from the FluxCD addon removal (#3379). certManager and ingressNginx stay pinned: Certificate and Ingress objects land inside the cluster and need their controllers there. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
8fc8d23dfd |
fix(computeplane): address review round three
Gate the Proxied ingress Service on _namespace.etcd like the addon HelmRelease it fronts, so an etcd-less tenant with the addon pinned on does not get a selector-less Service (+ test). Correct the secrets framing: the tenant-visible surface is an allowlist (include: [] withholds everything), so drop the never-load-bearing exclude entry, the bats cross-check and the rename-coupling comments, and state the real mechanism. Fix the etcd prerequisite docs (an ancestor's etcd satisfies the wrapped chart; the awaiting-etcd beacon is the diagnostic), reword the tenant toggle to stop advertising the not-yet-shipped placement routing, require maxReplicas >= 1 on the ingress-role node group, and fix comment drift (Service name after the release rename, placement-hook label, package-guard lookup semantics). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
5a3d25b8f5 |
docs(computeplane): document the etcd prerequisite and unlisted-field pass-through
The tenant Kubernetes machinery — including the cluster-autoscaler that brings up the default scale-from-zero md0 group — is backed by the tenant's etcd module, so a ComputePlane without it provisions no workers and gives no diagnostic; state the prerequisite in the README. Also state plainly that unlisted NodeGroup fields pass through to the wrapped chart unvalidated (the schema generator cannot close the object), with a test pinning that behaviour. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
6dcf5373e7 |
fix(computeplane): let Flux derive both release names
The aggregated API cannot represent spec.releaseName: the conversion rebuilds HelmRelease specs without it, so the first tenant edit through the API dropped the module's explicit computeplane-module release name, let it default to the object name computeplane, and collided with the Helm storage record the cluster release owned — with the tenant chart writing the old name back every interval. Drop releaseName from both HelmReleases and derive names from object names, like every other tenant module. Kamaji now writes the admin kubeconfig to computeplane-cluster-admin-kubeconfig; the computeplane-rd secrets.exclude entry follows in the same change, and a bats test pins the exclusion to the name the cluster release actually produces so a future rename cannot silently re-expose the credential. Also add the check-release-name guard the other extra modules carry, and pin the fluxcd addon's distribution artifact to the flux-operator version vendored in system/fluxcd-operator instead of :latest. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
96842d4a94 |
fix(computeplane): validate nodeGroups and require an ingress-nginx role
nodeGroups was a bare object: typos surfaced as a stuck release deep in the wrapped chart instead of an admission error, and groups without the ingress-nginx role silently left the pinned ingress-nginx controller (and with it the ComputePlane ingress) Pending forever, since supplied groups are not merged with the wrapped chart's default md0. Reuse the kubernetes app's nodeGroups typedefs for the schema and refuse rendering when no group carries the role. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
4c3262a4ee |
feat(computeplane): deliver ComputePlane as a tenant module wrapping the kubernetes app
Add the operator-owned computeplane tenant module per the revised design (cozystack/community design-proposals/compute-plane): an extra/computeplane chart that provisions a single-tenant, Cozystack-managed cluster for untrusted-code apps by deploying the ordinary apps/kubernetes chart with operator-fixed values (fluxcd, cert-manager, ingress-nginx Proxied). The package re-declares apps/kubernetes as its own source-only component, so the module is self-contained; the cluster release is named computeplane so Kamaji writes the admin kubeconfig to computeplane-admin-kubeconfig — the contract consumed by placement: ComputePlane applications. The tenant holds no admin kubeconfig and can set cluster shape (nodeGroups) only. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
9c63df7545 |
build(monitoring): stop building and publishing the Grafana rebuild
Remove the custom Grafana image build now that the deployment runs on the upstream grafana/grafana image with catalog plugins. Drops the Dockerfile, the pinned image tag, the image targets in both monitoring Makefiles, and the build step from the root Makefile, so ghcr.io/cozystack/cozystack/grafana is no longer produced or published. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
cozystack-ci[bot]
|
235d38dae1 |
Prepare release v1.6.0 (promoted from v1.6.0-rc.4)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
cozystack-ci[bot]
|
d95685c427 |
Prepare release v1.6.0-rc.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Myasnikov Daniil
|
8355f3e36f |
docs(seaweedfs): correct the stale "D-wedged is adopted" comments
The naming guard now refuses every two-generation state, but two comments in extra/seaweedfs still described a D-wedged duplicate (zero ready replicas) as safe to adopt and rendering through. readyReplicas is a snapshot, not proof the duplicate never served, so it cannot be told apart from D-split and the guard refuses it too. Align the state summary in the seaweedfs template comment and the test header with the presence-only refusal behaviour. No classifier change -- comments only. Address review feedback from coderabbitai on packages/extra/seaweedfs/tests/fullname_override_test.yaml. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
a099b7e086 |
fix(seaweedfs): fail the cleanup Job on a database-owner lookup error
The post-delete cleanup hook read Cluster/seaweedfs-db's Helm owner with `|| true`, so a Forbidden, a timeout or any API error collapsed to an empty owner, took the "nothing to reclaim" branch, and let the Job complete -- permanently orphaning the keep-protected Cluster the hook exists to reclaim, with nothing else that will ever collect it. Only a genuine NotFound now counts as absence; every other error prints the failure and exits non-zero so the Job retries. This matches the fail-closed handling the delete path already had. Add a regression assertion that the ownership lookup no longer swallows failures. Address review feedback from coderabbitai on packages/extra/seaweedfs/templates/hooks/cleanup.yaml. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
b16a51e60e |
test(seaweedfs): pin the naming guard's MultiZone zone-component classification
MultiZone (and Simple-with-pools) tenants render no plain `volume` component — extra/seaweedfs sets volume.enabled=false and the vendored chart emits one component per zone key with suffix `volume-<key>`, truncated at (62 - len(suffix)), which is SHORTER than the 56-char cut the guard's renamedVolumePrefix replays. The final review pass flagged this divergence as a guard bypass. For the supported instance the prefixes cannot diverge: the tenant module hardcodes the name `seaweedfs`, the fullname is 16 chars, and nothing truncates until a zone/pool key of ~40+ characters. Three new cases pin that supported path — legacy-only zones adopt, release-named-only zones refuse as class S, both generations refuse — since the suite previously had no zone shapes at all and the zone objects are the ONLY evidence on a MultiZone tenant. The divergence beyond that boundary (long instance names or absurd keys) is an accepted, documented limit, not a fix: decided in the 1.6 triage, recorded in _naming.tpl (both copies, kept in lockstep) and the runbook's Scope section. Instance names are fixed by the tenant module, so absurd keys are the only reachable trigger; revisit if instance naming is ever opened up. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
601f164c5f |
fix(seaweedfs): stop the naming guard guessing which volume generation holds the data
The guard told a class-S tenant from a D tenant by comparing PVC creationTimestamps,
on the premise that "PVCs are never recreated in place, so the older generation is
where the data was born". The premise is false, and this runbook is the counter-
example: Step 2's re-bind DELETES each release-named claim and recreates it under
the chart name against the same PV.
So a tenant interrupted part-way through Step 2 — after volume-0, before volume-1 —
has data1-seaweedfs-volume-0 created SECONDS ago holding real data, beside
data1-seaweedfs-system-volume-1 with the original timestamp also holding real data.
That is the exact inversion of the rule. Reproduced against the old template:
keeps its data on the release-named volume PVCs (created 2026-06-20T07:01:59Z,
OLDER than the chart-named data1-seaweedfs-volume-* claims created
2026-07-17T09:30:00Z) ... Remove the empty chart-named set ... (Step 2a, then Step 2)
Step 2a then runs `grep -E '^data1-seaweedfs-volume' | xargs kubectl delete pvc` —
deleting the claim Step 2 had just re-bound — and Step 2 had already restored the
PV's original reclaim policy, so on any Delete-policy StorageClass the volume and
the bytes go with it. The guard's own recovery path was a data-loss instruction.
Two more holes in the same classifier:
- The S-damaged branch was gated on $systemPVC, which requires the PVC name to
contain "seaweedfs". seaweedfs.componentName truncates the fullname to 56 chars
before appending -volume, so an instance name >= ~40 chars drops the chart name
off the PVC name — the very case fullname_override_named_test.yaml exists for.
Probed with `archive-of-quarterly-financial-statements-x1`: damaged + renamed set
Ready fell through to the D-split branch ("is running two live SeaweedFS sets ...
Stop the split"), sending the operator to quiesce the set holding ALL the data —
the exact misclassification the parent commit exists to remove. Damaged + not
Ready fired no branch at all and rendered through unguarded.
- readyReplicas == 0 was read as proof the duplicate never served. It is a
snapshot: a duplicate that crashed, was scaled down, or lost readiness AFTER
serving writes reads identically, and adoption strands whatever it wrote.
There is no sound signal here. Claim timestamps are mutable, StatefulSets are
recreated by the adoption hook, and liveness is a snapshot — nothing durable in the
object graph distinguishes "empty set a bad upgrade created" from "the data,
mid-re-bind". Rather than add a fourth heuristic, the guard now decides only what
is decidable:
legacy generation only (or nothing) -> render; adoption is correct
release-named generation only -> refuse, class S (Step 2)
BOTH generations -> refuse; an operator classifies
Both generation flags now OR the PVC and StatefulSet evidence, so the truncated
long-name tenant is seen via its label-matched StatefulSet.
The cost is bounded and lands only where it should. A duplicate exists only on a
tenant that passed through the broken 4.31 rename in 1.5.x — already damaged,
already needing an operator. A 1.4.x tenant upgrading straight to 1.6 never
renames, so it has one generation and is untouched. Confirmed on the stand:
tenant-l (1.4.5-era, never upgraded) is legacy-only and dry-runs clean; the four
both-generation tenants all already need an operator. What is lost is D-wedged
auto-adoption, which was resting on the readyReplicas assumption above.
Classification moves to the runbook, where a human has signals a template does not:
Step 1 now reads the ages of the PVs (durable — they survive the re-bind) via
.spec.volumeName instead of the claims, reports MIXED instead of picking, and walks
the operator through interrupted-Step-2 first, then strict age separation, then
verifying emptiness directly, then escalating rather than guessing. Step 2 stashes
the PV's original reclaim policy ON THE PV and leaves it Retain until Step 5
verifies the tenant, so a mis-step during recovery is survivable. Step 2a grows a
precondition that refuses unless every chart-named claim is bound to a strictly
newer PV.
Tests: 7 of 14 in the enforcing suite fail against the old template, including the
interrupted-Step-2 tenant and both truncated-long-name cases. hack/seaweedfs-guard-
parity.bats pins the two copies (system/ enforcing, extra/ sibling) against silent
drift — they cannot share a library, neither chart depends on cozy-lib, and "keep
these in sync" as a comment is the failure class this whole branch is about; it
also pins that creationTimestamp and readyReplicas never come back.
`make test` in both charts now runs helm template --is-upgrade and greps for the
refusal. The fail-closed canary — the headline safety property — had ZERO automated
coverage: guard_fail_closed_test.yaml passes with AND without the fix, because
helm-unittest 1.0.3 ignores release.isUpgrade. Verified the new target fails when
the canary is removed.
Review round 3 — the refusal was right; the justification was false and the
procedure that replaced the chart's judgement was broken on both axes.
THE COMMENT WAS WRONG. It said "nothing durable in the object graph distinguishes
the empty generation from the data mid-re-bind". A durable signal does exist, and
the reviewers were right to call that out: the Helm release history. Verified on
the stand — sh.helm.release.v1.<name>-system.v1 records the scheme the tenant was
BORN under (tenant-root/l/dsplit/named rev 1 render seaweedfs-master => pre-4.31;
tenant-fresh renders seaweedfs-system-master => 1.5.x), and info.first_deployed is
on every retained revision, so anchoring PV creationTimestamps against it survives
history pruning and the Step 2 re-bind (tenant-root: chart-named +10s vs
release-named +2084s; tenant-fresh: +5225s vs +6s).
BUT IT DOES NOT UNLOCK AUTO-ADOPTION, and that is the substantive disagreement
with the round-3 direction. Birth order answers "which generation is ORIGINAL".
Adoption needs "is the OTHER one EMPTY". Those come apart exactly where it
matters: D-wedged (duplicate never scheduled, safe to adopt) and D-split
(duplicate served writes, adopting strands them) are BOTH born pre-4.31 and are
therefore identical on every durable signal. Verified: tenant-root and
tenant-dsplit have the same rev-1 scheme and the same first_deployed deltas, yet
one duplicate served and the other did not. The only thing that ever separated
them was readyReplicas, which is a snapshot and not evidence. So the render keeps
refusing on presence, and the comment now says why — naming the signal, so nobody
re-derives the broken one.
The signal IS used where it belongs, in the audit.
- naming-guard.yaml / seaweedfs.yaml: match the release-named generation by
RECONSTRUCTING the name 4.31 gives this release's volume component
(templates/_naming.tpl, replaying seaweedfs.fullname + componentName), tested
FIRST because the prefixes nest. Fixes a blocker: an instance legitimately named
`seaweedfs-volume` renders seaweedfs-volume-system-volume and
data1-seaweedfs-volume-system-volume-0, which BOTH satisfy the chart-named
prefixes — a prefix-only guard read its live storage as legacy and would render
onto the empty chart-named claims, stranding the real generation. The parity
bats passed straight through it, because parity pins sameness, not correctness.
Reconstruction also removes the name-match blind spot at source: a truncated long
instance name now matches by PVC, not only by StatefulSet.
- hack/seaweedfs-naming-audit.sh + .bats: the classifier is a tested script now.
Two revisions of it shipped as an untested snippet inside the runbook and both
were wrong in ways that routed a live tenant into the step that strands its data:
a selector `^data1-(.*seaweedfs.*)-volume` that also matched the chart-named
claims, so the release-named range spanned BOTH generations, every tenant read as
"overlapping / mid-rebind", and a genuine S-damaged tenant was routed AWAY from
Step 2a into Step 2 — which deletes its data claim, fails to recreate it, then
deletes its StatefulSets; and a `grep seaweedfs` name match that reported
"L — nothing to do" for a long-named tenant the chart refuses. Both are covered.
The audit reports L/S/MIXED matching the guard, names the original generation
from both durable signals, detects MID-REBIND (both generations at
first_deployed), and states plainly that "original" is not "empty" and that
emptiness must be established from the duplicate's volume files before deleting.
A bats case renders _naming.tpl through helm and compares it to the script's
reconstruction, so the two implementations cannot drift.
- Runbook: Step 1 calls the script. Steps 3/4 were circular — Step 3 ended in an
upgrade the presence-gated guard refuses, and Step 4 told the operator to clear
duplicates AFTER an upgrade that cannot run until they are gone. Step 3 now
removes the duplicate BEFORE the upgrade and re-runs the audit to prove it; Step 4
starts from "every tenant reads L or S". D-wedged no longer claims to need
nothing. Step 3's "enumerate fids that resolve only on the duplicate" was not an
executable procedure and is now an explicit escalation: two masters allocate
volume IDs from independent sequences into one shared seaweedfs-db, so there is
no supported tool that reconciles two volume-ID spaces against one metadata
store. Step 2's re-bind restores the cleanup hook's labels on the recreated
claims, without which a recovered tenant leaks its volumes on a later delete.
Fixture corrected: data1-...-x1-system-volume-0 is not a name the chart renders;
componentName gives ...-x1-system-seaw-volume-0.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
47e24f01d1 |
fix(platform): hand seaweedfs-db over for every instance name, not just the default
Migration 43 performs the v1.5.0 db-split hand-over: it re-owns Cluster/seaweedfs-db from <name>-system to the new <name>-db release and stamps helm.sh/resource-policy: keep, so the <name>-system upgrade — whose post-split chart no longer renders the Cluster — does not prune it as a removed resource. It compared the owning release name against the literal "seaweedfs-system". SeaweedFS is a user-creatable kind, so an instance named `foo` is owned by `foo-system` and was silently skipped: no re-own, no keep. The <name>-system upgrade then deleted the Cluster, CNPG took its PVC with it, and the filer metadata — the index for every object in that tenant's S3 — was destroyed. Data loss, not an outage: the volume PVCs still hold the bytes, nothing can find them. Verified live on the upgrade stand, which is a natural control group. All five tenants sit in the SAME failed-upgrade loop (<name>-system last-deployed is rev 1, a pre-split revision whose manifest still contains the Cluster, so every retry recomputes the same deletion): tenant-root/dsplit/l instance `seaweedfs` owner=seaweedfs-db keep Cluster alive tenant-named instance `foo` no Cluster at all Cluster GONE The only difference is that migration 43 fired for the default-named instances and skipped `foo`. That also answers whether `keep` is sufficient: three tenants survive the identical prune loop with it. Correction to the reported mechanism: this is the 1.4->1.5 db split (PR #2601, v1.5.0), not a 1.5->1.6 change — v1.5.3 and main are identical on every db-split file. And the split does protect the object; it just only protected the instances someone happened to name `seaweedfs`. - lib/seaweedfs-db-adopt.sh: shared hand-over matching the `-system` SUFFIX, so `foo-system` -> `foo-db` exactly as `seaweedfs-system` -> `seaweedfs-db`. Idempotent, and refuses a release named literally `-system` rather than annotating an owner of `-db` that no release would claim. - Migration 43 sources it: covers clusters upgrading from before 43 (1.4.x). - Migration 53 (new) re-runs it: migrations never re-run, so any cluster already at >= 44 ran the hardcoded version and is still exposed. As a pre-upgrade hook it lands before <name>-system re-renders, closing the window on THIS upgrade. This is the one that matters for 1.5.x -> 1.6. - targetVersion 53 -> 54. Tests fail on unfixed code: the non-default-name case produces no ANNOTATE at all against the hardcoded 43, while the default-name case passes — which is precisely the bug's shape. They drive the real migration scripts against a fake kubectl, following hack/migration-50-etcd-adopt.bats. LIMIT, stated plainly: a Cluster that is already deleted cannot be recovered by either migration — there is nothing left to re-annotate and the PVC went with it. Such a tenant needs seaweedfs-db restored from a backup. Runbook Step 0 adds the ownership audit that distinguishes at-risk from already-lost, and records that <name>-db can report Ready while its Cluster is gone (it rendered fine; a later <name>-system prune removed it and Flux has not re-checked) — so the audit trusts `kubectl get cluster`, not the HelmRelease status. Review round 2 — two ways this still lost the database: - OWNERSHIP IS NOT SAFETY. The helper skipped any Cluster already owned by <name>-db, treating ownership as proof the hand-over was done. It is not: where the hand-over was skipped, <name>-system prunes the Cluster and <name>-db RECREATES it under its own ownership with NO keep, while <name>-system's prune baseline still lists it — so the next reconcile deletes it again. That is the observed delete/recreate loop, and the skip walked straight past it. Verified on the stand: tenant-l and tenant-root are seaweedfs-db-owned AND their <name>-system deployed revision (rev 1) still contains the Cluster, so only keep saves them; tenant-fresh is seaweedfs-db-owned without keep but was installed after the split, so its baseline never had one. Telling those apart needs the release's deployed manifest, which this script cannot read cheaply or reliably. The costs are asymmetric — a needless keep leaves an orphan on delete (now reclaimed by the cleanup hook), a missing one loses the database — so keep is stamped on every Cluster owned by either side of the split. - FAIL OPEN, THEN STAMP. `set -euo pipefail` does not abort on a failing command substitution in a `for` word-list: the loop ran zero times, the script continued, and stamp_cozystack_version ran anyway. Migrations never re-run, so one throttle, RBAC hiccup, or not-yet-established CNPG apiservice behind the pre-upgrade hook permanently left every at-risk tenant exposed with no later migration to catch them. Reproduced: against the old helper a failing fleet scan still logged STAMP 54. Now every kubectl failure is fatal EXCEPT the two that genuinely mean "nothing to do" — the CNPG resource type not being served (the fail-open that IS load-bearing: a cluster without CNPG must still upgrade) and a Cluster vanishing between scan and read. An unreadable annotation is no longer indistinguishable from an absent one, and the unowned case now warns instead of passing silently. - ORPHAN ON DELETE. `keep` is permanent (removal deferred to the 1.7 batch migrations) and also survives the <name>-db release's own uninstall, so `kubectl delete seaweedfs <name>` left the CNPG Cluster, its Postgres pods and its PVCs behind — the cleanup hook selects app.kubernetes.io/instance=<name>-system, which the db PVCs (cnpg.io/cluster=seaweedfs-db) do not carry. Migration 53 widened that from pre-1.5 default-named tenants to every instance name, so the hook now deletes the Cluster explicitly; its PVCs carry ownerReferences to it and follow. - Tests assert the stamped VERSION (44 / 54), not a bare "STAMP": a wrong number would loop run-migrations.sh forever and the old assertion could not tell. Review round 3 — the reclaim introduced a data-loss path of its own: - DELETING ONE APP DELETED ANOTHER APP'S DATABASE. The db chart hardcodes the Cluster name `seaweedfs-db` for every instance, and the reclaim was scoped by that name in the namespace rather than by what the release owns. templates/seaweedfs-db.yaml gates the <name>-db HelmRelease on `topology != "Client"`; the cleanup hook had no such gate. Verified by render: `helm template s3-remote . --set topology=Client` emits ZERO s3-remote-db HelmReleases yet still emitted the Role granting delete on clusters/seaweedfs-db AND `kubectl delete cluster.postgresql.cnpg.io -n tenant-root seaweedfs-db`. So in a namespace running a server instance plus a Client, deleting the CLIENT destroyed the SERVER's filer metadata — CNPG takes the PVC via ownerReferences — for an app the operator never touched. Reachable the same way via a second Simple instance, whose <name>-db loses Helm's ownership check but which still installs and still carries the hook. main's hook was instance-scoped; this was new, and mine. Now gated exactly as seaweedfs-db.yaml is, and the Job re-checks meta.helm.sh/release-name == <release>-db at run time before deleting. The gate stops the permission being granted where it can never be legitimate; the runtime check stops the delete where it can. - `|| echo` turned any API/authz/admission failure into exit zero, so backoffLimit never retried and hook-delete-policy removed the only cleanup Job while the keep-protected Cluster stayed orphaned. A failed reclaim now fails the Job. - TEST SHELL. The migrations run under /bin/sh = busybox ash (the image is FROM alpine, run-migrations.sh is #!/bin/sh), but the tests drove them with bash — so the fail-closed guarantees were asserted in a shell that never runs them, which matters more than usual given the bug they cover was a fail-open. The scripts are now invoked via `sh` and the fake kubectl is POSIX sh. Verified directly against the production base image (busybox 1.37.0 / alpine:3.24): `set -o pipefail` is supported, every script passes `ash -n`, and the failing-fleet-scan path exits non-zero without stamping under real ash. The recipe is in the bats header. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
8499b5f4a1 |
fix(seaweedfs): move the naming-migration guard into the render that actually renames
The class-S guard from #3282 lives in extra/seaweedfs, but that render is not in the path of a platform upgrade: the <name>-system HelmRelease pulls system/seaweedfs from a platform-managed ExternalArtifact, so bumping the platform upgrades the release directly from the new artifact and applies fullnameOverride: seaweedfs without extra/seaweedfs ever re-rendering. On a tenant installed fresh on 1.5.x that stood up an empty chart-named set beside the live release-named data, deleted the renamed s3 Service, and left S3 readable only because the empty set's pods cannot start. Worse, the damage flips the guard's own classification: creating the chart-named artifacts makes $legacyData true, so the guard then reports D-split and sends the operator to runbook Step 3 — which quiesces the renamed set, i.e. the one holding ALL the data. - Add the ENFORCING guard to system/seaweedfs (templates/naming-guard.yaml): the only render guaranteed to sit between a platform upgrade and the tenant's workloads. It also refuses to render if the fullnameOverride pin is ever lifted. - Tell S from D by PVC AGE when both generations exist: PVCs are never recreated in place (StatefulSets are, by the adoption hook), so the older volume-PVC generation is where the data was born. Renamed-older ⇒ class S (damaged variant, new Step 2a), never D-split. - Fail closed: the release namespace is a lookup canary; an upgrade that cannot see the cluster refuses instead of classifying blind. Client-side installs (CI lint/unittest) still render — they never touch live data. - Mirror the same classification in extra/seaweedfs so the operator-facing SeaweedFS application reports the correct class. - Tests now model the ExternalArtifact upgrade path in the chart it actually runs through (the old ones mocked lookups only in extra/seaweedfs, which is why this shipped green), including the damaged-S tenant observed live. - Runbook: add S-damaged audit classification (PVC age), Step 2a cleanup of the empty chart-named set, and the D-wedged anti-affinity stall workaround observed on 3-node clusters. helm-unittest 1.0.3 ignores release.isUpgrade, so the canary refusal is verified with 'helm template --is-upgrade' instead of a unit test. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
055c8c755c |
chore(packages): drop duplicate test targets
Three package Makefiles declared an identical `test` target twice, so every invocation printed "overriding commands for target 'test'" and "ignoring old commands". Make names the lines in that warning: it was already running the second recipe and discarding the first, so the first declaration was dead in each case and removing it changes nothing but the noise. Which copy is dead is not the same in all three, so the edit is not mechanical. In monitoring the `.PHONY: test` line sits above the dead recipe rather than the live one; `.PHONY` applies to the target name wherever it is declared, so the declaration stays and only the recipe under it goes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
4a00b502d7 |
fix(seaweedfs): adopt legacy workloads in place across the 4.31 rename (#3282)
## What this PR does Fixes a v1.5.0 upgrade regression where a tenant ends up with **two** SeaweedFS workload sets. **Root cause.** v1.5.0 bumped the vendored SeaweedFS chart `4.0.405 → 4.31.0`. Upstream changed resource naming: before 4.31 the chart named workloads after the **chart** (`seaweedfs-*`, ignoring the release name), and 4.31 names them after the **release**. The data-plane release is `<name>-system`, so every StatefulSet wanted to become `seaweedfs-system-*`. StatefulSet names are immutable, so Helm could not rename in place — it stood up a second, empty set beside the running one while the data stayed on the original `data1-seaweedfs-volume-*` PVCs. Where the node count let both sets run, they shared one `seaweedfs-s3` Service (identical pod labels) and one `seaweedfs-db` filer metadata database, so the empty set answered reads and the two master sets mixed volume IDs from independent sequences into one metadata table. **Fix.** Pin `fullnameOverride` in `system/seaweedfs` values so the rendered names stop following the release name, which is the convention already used by `cozy-proxy`, `flux-operator`, `grafana-operator`, `linstor-scheduler` and `victoria-metrics-operator`. The names go back to what every pre-1.5 cluster already runs, so the upgrade adopts the running workloads **and their volumes** in place. No data migration, no platform migration. **The one case that cannot be adopted.** A tenant installed *fresh* on 1.5.x wrote its data under the renamed workloads, so its volumes are not the chart-named ones. Pinning the chart name there would rename the workloads *away* from that data and bring up an empty cluster beside it. Helm cannot move data between PVCs, so `extra/seaweedfs` refuses to render for such a tenant and points at the recovery runbook, which re-binds the existing PVs onto the adopted PVC names (no copying — the claim is renamed, the volume is not touched). The guard is fail-closed: it fires only when renamed volumes are present and no adopted ones are. Two naming subtleties the guard has to respect, both verified by rendering the shipped 1.5 chart: - Before 4.31 the chart ignored the release name, so an instance running under a non-default name (`SeaweedFS` is a user-creatable kind) *also* stored its data on `data1-seaweedfs-volume-*`. Such a tenant is adopted, not blocked. - On 4.31 the fullname helper appends the chart name when the release name does not contain it, and the component helper truncates to 56 characters. A fresh 1.5.x instance named `archive` therefore wrote to `data1-archive-system-seaweedfs-volume-*`, and a long name loses the chart name from its tail entirely. The guard matches volumes by shape and by the chart labels on the StatefulSet instead of reconstructing those names, so both are caught. **On the duplicate set.** Once the fix reconciles, the `seaweedfs-system-*` StatefulSets and Deployments are no longer part of the release and Helm removes them on its own. What it cannot remove — PVCs it did not template, and cert-manager Secrets that carry no owner reference — is inert, and the runbook clears it explicitly. A pre-upgrade migration was considered and dropped: it would duplicate what Helm already does, while its only unique action (deleting PVCs) cannot distinguish "the duplicate never served" from "it served and is currently down", and any transient delete failure under `set -e` would fail the platform's pre-upgrade hook and block delivery of this very fix. ## Testing `helm unittest` — `system/seaweedfs` 21 tests, `extra/seaweedfs` 17 tests. The guard's `lookup` branches are covered by mocking the Kubernetes provider (the pattern already used in `core/platform` and `system/cozystack-api`): adopted-only, renamed-only, both sets, zoned volumes, a non-default instance name, a truncated long name, and an unrelated `data1-*-volume-*` PVC that must not trip the guard. The s3-consumer suite now re-creates the name/fullname divergence explicitly, so it keeps guarding that regression regardless of the shipped default. The remaining regression test for this class of bug is an upgrade E2E (seed a pre-4.31 named set, upgrade, assert a single set with data intact) — no fresh-install E2E or `helm template` can reproduce an upgrade over pre-existing state. That belongs in the chainsaw upgrade lane and is a follow-up. ### Screenshots N/A — no UI changes. ### Release note ```release-note fix(seaweedfs): adopt existing SeaweedFS workloads and volumes in place when upgrading past the 4.31 chart rename, instead of standing up a second, empty cluster beside them ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an audit & recovery runbook for SeaweedFS 4.31 rename impacts. * **Bug Fixes** * Stabilized SeaweedFS workload/PVC naming across upgrades, including non-default instance names. * Added preflight safeguards to prevent upgrade scenarios that could split or wedge existing data. * Updated post-delete cleanup to remove secrets from both legacy and current naming schemes. * **Tests** * Expanded Helm unit coverage for naming guards, cleanup behavior, and S3 service name consumers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
fbbe4af7df |
refactor(network): drop ExposureClass/ServiceExposure for native loadBalancerClass (#3218)
## What Removes the `network.cozystack.io` API group — `ExposureClass` + `ServiceExposure` (#3081) — and its reconciler, in favor of native `Service` `type: LoadBalancer` + `loadBalancerClass`. ## Why Full rationale and discussion in #3164. For **managed applications** the chart already owns the Service, so: - `type: LoadBalancer` + `loadBalancerClass` + an admin-provisioned pool covers vendor-neutral external exposure without a dedicated API group and controller; - the `externalIPs` (node-IP) default never went through `ServiceExposure` (its backend is a no-op) and is left unchanged; - the only current consumer, the host ingress, keeps rendering `type: LoadBalancer` and just drops the `ServiceExposure`; the admin provisions the address pool / announcer out of band, as the install docs already assume. `ServiceExposure`'s one unique feature — dynamic per-namespace scoping of a shared VIP pool — is a convenience the managed-app path does not need, and it does not justify a second tenant-networking API group next to `sdn.cozystack.io`. ## Changes - Delete `api/network/v1alpha1` and `internal/controller/serviceexposure` (+ CRD definitions, e2e bats, controller registration, RBAC). - Host ingress: drop the `ServiceExposure` template; add an optional `publishing.loadBalancerClass` that sets `spec.loadBalancerClass` on the host-ingress LoadBalancer Service. - The `externalIPs` path and default behavior are unchanged. ## Verification - `go build ./...` — green. - `helm unittest` — `core/platform` 72/72, `extra/ingress` 25/25 (incl. host-vs-tenant `loadBalancerClass` scoping and the immutable-field migration guard). ## Migration None required. `network.cozystack.io` (#3081) has never shipped in a release — it is on `main` only (not backported to `release-1.5`; absent from v1.4.x / v1.5.x), so no upgrading cluster has these CRDs and this removes it before it is ever released. Clusters that track `main` and explicitly set `publishing.exposureClass` (opt-in, only since 2026-06-30) should provision their MetalLB / Cilium address pool themselves and set `publishing.loadBalancerClass`; any orphaned `cozystack-<class>` pool can be deleted by hand. Default clusters (externalIPs path) are unaffected. ## Status **Draft / RFC for the #3164 discussion** — this removes an API group that landed only days ago, so nothing here should merge before the design call. cc @lexfrei (author of #3081) and @lllamnyp (#3164). Deferred: when Kubernetes actually removes `Service.spec.externalIPs`, the node-IP default migrates via a MetalLB `IPAddressPool` with no `L2Advertisement` (allocate-but-don't-announce) — a separate change. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for configuring the ingress host Service with a native `loadBalancerClass` value. * Expanded chart tests to verify class wiring and immutability behavior across install and upgrade scenarios. * **Bug Fixes** * Removed the legacy exposure-class/service-exposure flow and related controller behavior. * Updated templates and CRD packaging so the platform no longer renders or depends on those deprecated resources. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
f6a03e93f8 |
fix(etcd): complete v1alpha2 transition for in-cluster 1.5→1.6 upgrades (#3270)
Consolidates the etcd `v1alpha2` transition fix for **in-cluster 1.5 → 1.6 upgrades** into a single PR, rebased on current `main`. Supersedes #3265 and #3261 — their commits are carried here (authorship preserved), so those PRs can be closed once this lands. Fresh installs use the new shapes directly and are unaffected, so fresh-install CI does not catch these; the 1.5 → 1.6 e2e upgrade path is the authoritative regression guard. ## What this PR does ### 1. Keep the legacy `etcd-headless` Service alive during adoption — `packages/extra/etcd` `etcd-migrate` adopts legacy clusters **in place**: the Pods keep their original `spec.subdomain: etcd-headless` and are dialed at `etcd-<i>.etcd-headless.<ns>.svc` until they roll onto the operator's native `<member>.etcd.<ns>.svc` domain. The v1alpha2 operator only creates the native `etcd` Service and the legacy `etcd-headless` Service is pruned, so those per-pod names stop resolving (`no such host`), `MemberList` fails, and `status.readyMembers` never populates — the `EtcdCluster` never goes `Ready` even though etcd is healthy and in quorum. We ship a chart-managed transitional headless `etcd-headless` Service (selector mirrors the operator's native `etcd` Service via `etcd-operator.cozystack.io/cluster`, `publishNotReadyAddresses: true`) — the DNS counterpart of the legacy `*.etcd-headless.<ns>.svc` SAN already kept for this window. Removable together with that SAN once members roll onto the native subdomain. ### 2. Survive the immutable controller-Deployment selector on upgrade — `packages/system/etcd-operator` (#3242) #2859 replaced the upstream etcd-operator chart with the cozystack-authored one, changing `Deployment.spec.selector.matchLabels`. `spec.selector` is immutable, so `helm upgrade` cannot patch the existing Deployment and the whole HelmRelease upgrade fails (`field is immutable`). A **pre-upgrade hook** (`templates/pre-upgrade-selector-fix.yaml`: ServiceAccount + Role + RoleBinding + Job) deletes the Deployment **only** when its live selector is the pre-1.6 one, so Helm recreates it cleanly. No-op when the selector already matches, never runs on fresh install. ### 3. Raise the operator's memory cold-start floor — `packages/system/etcd-operator` Steady-state working set is ~250Mi; the static `limits.memory: 128Mi` OOMKills a Pod that starts before the VPA admission webhook rewrites it. Raise the floor to `256Mi` (and VPA `minAllowed` to match) as defense in depth so the operator never depends on VPA timing to avoid crashing. ### 4. Make migration 50 (etcd adoption) robust in-cluster — `packages/core/platform/images/migrations/migrations/50` (was #3261) Exact server/peer cert-SAN match, Secret-gated wait on the adoption Secret, and in-cluster (IPv6-safe) kubeconfig handling, so the migration script drives the adoption reliably from inside the cluster. Covered by `hack/migration-50-etcd-adopt.bats`. ### 5. Hardening / review fixes (this PR's original scope) - **Hook `runAsUser: 65532`.** `pre-upgrade-selector-fix.yaml` set `runAsNonRoot: true` but no numeric `runAsUser`; `clastix/kubectl`'s image user is the non-numeric name `nonroot`, which the kubelet cannot verify against `runAsNonRoot` — so the hook Pod fails admission and silently blocks the very upgrade it exists to unblock. Adds `runAsUser: 65532`, matching the postgres-operator webhook-ready hook that runs the same image. - **Digest-pin the kubectl image.** The values comment claimed digest-pinning but shipped a floating `v1.32` tag. Reuses the digest postgres-operator vendors, adds the `renovate` annotation, and templates `repo:tag@digest`. - **Tests.** `etcd-operator/tests/selector-fix-hook_test.yaml` (hook wiring, weight ordering, namespaced least-privilege RBAC, numeric-non-root security context, digest-pinned image), `etcd-operator/tests/deployment_test.yaml` (256Mi cold-start floor), `extra/etcd/tests/etcd-cluster_test.yaml` (transitional `etcd-headless` Service). Each assertion was mutation-tested. ### 6. Derive the default S3 endpoint from the provisioned bucket — `packages/system/backupstrategy-controller` Derive the default S3 endpoint (and per-driver scheme / TLS / `secure_connection`) from the provisioned bucket Secret instead of requiring it to be hand-set, so etcd (and other) backup strategies get a working endpoint by default. Docs in `docs/operations/backup-classes.md`; covered by `tests/endpoint_form_test.yaml`. ### Verification - `helm unittest` green on current `main`: etcd-operator **18/18**, extra/etcd **18/18**, backupstrategy-controller **11/11**. - Behaviours 1 & 2 were reproduced and confirmed live on a 1.5.2 → 1.6.0-rc.1 adoption (3-node cluster) — recreating the `etcd-headless` Service took the adopted cluster to `readyMembers=3 / Available=True`. ```release-note fix(etcd): complete the v1alpha2 transition on in-cluster 1.5→1.6 upgrades — keep the legacy etcd-headless Service alive so adopted members stay resolvable, delete the pre-1.6 operator Deployment via a pre-upgrade hook to get past the immutable selector, raise the operator's memory floor so it does not OOM before the VPA scales it, and make the etcd adoption migration robust in-cluster. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Improved backup storage endpoint handling across supported backup drivers, including provisioned and external S3 storage. * Added compatibility support for legacy etcd pod discovery during migration. * Added an automated upgrade safeguard for etcd operator deployments. * **Bug Fixes** * Improved certificate SAN detection and etcd migration authentication. * Increased the etcd operator’s minimum startup memory to prevent early restarts. * **Documentation** * Clarified backup endpoint, TLS, and driver-specific behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
ca4f99411f |
fix(etcd): harden v1alpha2 transition — hook runAsUser, digest-pin kubectl, tests
Addresses review on #3265: - pre-upgrade-selector-fix hook: add runAsUser: 65532. clastix/kubectl's image user is the non-numeric name `nonroot`, which the kubelet cannot verify against runAsNonRoot: true — so the Pod fails admission and the hook never runs, silently blocking the very 1.5->1.6 upgrade it exists to unblock. Matches postgres-operator's webhook-ready hook (same image). - Digest-pin the kubectl image (the comment already claimed digest-pinning but shipped a floating v1.32 tag). Reuse the digest postgres-operator already vendors, add the renovate annotation, and template repo:tag@digest. - Tests (both charts have CI helm-unittest suites, none previously covered these): - etcd-operator/tests/selector-fix-hook_test.yaml: hook wiring, namespaced least-privilege RBAC, numeric-non-root securityContext (guards the runAsUser regression above), digest-pinned image. - etcd-operator/tests/deployment_test.yaml: assert the 256Mi cold-start memory floor so it can't silently drop back to an OOMKilling value. - extra/etcd/tests/etcd-cluster_test.yaml: assert the transitional etcd-headless Service (headless, publishNotReadyAddresses, member selector, client/peer ports). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
d51d6914d7 |
fix(etcd): keep legacy etcd-headless Service alive during v1alpha2 adoption
etcd-migrate adopts legacy clusters IN PLACE, so the Pods keep their original spec.subdomain: etcd-headless and are dialed at etcd-<i>.etcd-headless.<ns>.svc until they eventually roll onto the operator's native <member>.etcd.<ns>.svc domain. The v1alpha2 operator only creates the native `etcd` Service, and the legacy `etcd-headless` Service is pruned during the transition, so those names stop resolving (no such host), the operator's MemberList fails, and status.readyMembers never populates -- the EtcdCluster never goes Ready even though the etcd processes are healthy and in quorum. Ship a chart-managed transitional headless `etcd-headless` Service (selector mirrors the operator's native `etcd` Service via etcd-operator.cozystack.io/cluster, publishNotReadyAddresses: true). This is the DNS counterpart of the legacy *.etcd-headless.<ns>.svc wildcard already kept in the server/peer cert SANs for the same transition window -- the TLS half of the compat was done, the DNS half was missing. Safe to remove, together with that SAN, once the members have rolled onto the native `etcd` subdomain. Verified live: recreating this Service on an adopted 3-node cluster restored per-pod DNS and the cluster went readyMembers=3 / Available=True. Refs: #3243 Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
IvanHunters
|
c31a9f8ec3 |
fix(monitoring): promote oidc.users[].role to required + regenerate schema
The `OIDCUser` typedef in `packages/extra/monitoring/values.yaml`
declared only `@field {string} email`; `cozyvalues-gen` regenerated
`values.schema.json`, `README.md`, and the mirrored `openAPISchema`
on the `Monitoring` ApplicationDefinition WITHOUT a `role` field.
Consequences:
- The `@enum OIDCRole` declaration was orphan — referenced from
nowhere, so the schema's items object had `required: ["email"]`
and one property.
- A dashboard-driven CR would offer a form without any role picker;
a schema-conformant `{email}` submission would render into the
users-Job with `role: null`, `jq --raw-output` yields the string
`"null"`, the `POST /api/orgs/1/users` body carries
`"role":"null"`, Grafana returns HTTP 400, and the post-install
hook exhausts `backoffLimit: 6` — HelmRelease never becomes Ready.
- Unit + e2e tests all hardcoded `role: Admin` / `role: Viewer`, so
the contract mismatch never surfaced in CI.
Fix:
- Add `@field {OIDCRole} role - ...` to the `OIDCUser` typedef in
`packages/extra/monitoring/values.yaml` so cozyvalues-gen materialises
the field with enum + required.
- Regenerate `values.schema.json`, `README.md`, and mirrored
`openAPISchema` via `make generate`.
- Add `monitoring.oidc.assertUsersRoleShape` template-level guard
(same shape as `assertUsersEmailShape`) so a stale-schema CR or a
direct-helm invocation fails at render time with a legible error
instead of at hook execution with a Grafana 400.
- Consolidate all `spec.oidc.users` guards into a single
`monitoring.oidc.assertUsersPrologue` include and call it once from
`oidc-users-job.yaml`, ordering the incompat check
(`assertCustomSecretRefUsersEmpty`) before the per-entry role /
email regexes so operators see the more actionable message first.
Drop the duplicate include from `grafana.yaml` (Job template renders
in every mode anyway).
- Two new helm-unittest cases lock the regression:
`oidc.users entry missing role fails the render` and
`oidc.users entry with unknown role fails the render`.
- Point the previous `secretRef + users` incompat unit test at the
users-Job template since that's where the prologue now fires.
- `docs/oidc-grafana.md`: correct the failed-hook post-mortem window
claim — with `ttlSecondsAfterFinished: 3600` the Kubernetes TTL
controller reaps the Job + Pod 1h after the terminal condition,
not "until the next helm-upgrade".
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
86e54f1296 |
docs(monitoring): align OIDC docs with the actual mode-agnostic contract
Previous docs claimed "the chart forces skip_org_role_sync = true regardless of mode" — a promise the code did not keep in CustomConfig mode. The chart is now fixed; sync the docs so the guarantee reads truthfully and the incompatibility surface is explicit: - `docs/oidc-grafana.md`: rewrite the CustomConfig section so both chart-forced settings are named explicitly, the `merge` semantics are documented (operator can't override), and the `secretRef` + `users` incompatibility is called out with an explicit failure path. - `system/monitoring/values.yaml`: expand the `config`/`secretRef` comments so operators reading the values doc see the merge behavior and the secretRef limitation before they hit the render-time fail. Add `enabled: "true"` to the "typical keys" list — Grafana defaults `enabled: false` and an inline map without it is inert. - `extra/monitoring`: regenerated values.schema.json + README.md via `make generate` (cozyvalues-gen + hack/update-crd.sh) to pick up the `oidc.users[].email` doc-string touch; mirrored openAPISchema on the ApplicationDefinition follows. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
26e5f06d27 |
fix(monitoring): chart-force skip_org_role_sync + email lookup in every OIDC mode
The chart previously injected the two settings the users-Job's
contract depends on — `skip_org_role_sync = true` and
`oauth_allow_insecure_email_lookup = true` — only inside the
`mode: System` branch of `grafana.yaml`. `values.yaml` and
`docs/oidc-grafana.md` both promised the settings would be forced
regardless of mode, but under `mode: CustomConfig` the operator's
inline `auth.generic_oauth` map was written verbatim, both
settings were absent, and the users-Job's role assignments were
silently stranded (no lookup by email → new Grafana account minted
at login) or transiently overwritten (no skip_org_role_sync → any
`role_attribute_path` or Grafana's `auto_assign_org_role: Viewer`
overwrote the Job's role on each login).
Fix: merge a `$chartForcedOidc` dict on top of both the generated
System block and the operator's inline map (`merge` wins for
overlapping keys, so the operator can NOT override the two
settings). CustomConfig secretRef mode cannot receive the merge
because the ini fragment is authoritative — the chart now hard-fails
render when both `customConfig.secretRef.name` and non-empty
`spec.oidc.users` are set, forcing the operator to either switch to
inline config or manage authorization themselves inside the ini.
Also:
- Align the inner HR `timeout` on `packages/extra/monitoring/templates/helmrelease.yaml`
from 10m to 15m so it matches the outer cozyrds
`release.cozystack.io/helm-install-timeout: "15m"` and does not
fail while the users-Job is still inside its
`activeDeadlineSeconds: 900`.
- Route the users-Job image through `cozy-lib.image` so
mirrored / air-gapped installations resolve it (matches the
pattern in `packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`).
- Add `spec.ttlSecondsAfterFinished: 3600` on the users-Job so a
succeeded Job + Pod are garbage-collected on the next reconcile
interval rather than lingering until the next hook activation
(which never fires on release uninstall).
- Harden the users-Job shell:
* emails read NUL-terminated (`jq --raw-output0`) so an address
containing whitespace cannot corrupt the `for email in ...`
loop's field splitting;
* `/api/users/lookup?loginOrEmail=<email>` URL-encoded via
`jq @uri` so `&`/`=`/`?` in the local part parse correctly on
the Grafana side;
* prune loop switched from pipe-delimited fields to TSV inside
NUL-terminated records, with `cut -f` for extraction.
- Add a template-level assertion (`monitoring.oidc.assertUsersEmailShape`)
rejecting emails that do not match a conservative regex —
cheaper failure mode than a 400 from Grafana.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
0a65dd04a1 |
feat(monitoring): expose spec.oidc.users on the Monitoring CR
Extends the tenant-facing `Monitoring` CR with the new
`spec.oidc.users: [{email, role: Admin|Editor|Viewer}]` field
consumed by the users-Job introduced in the previous commit. Works
for both `System` and `CustomConfig` modes; ignored when
`mode: None`.
Regenerated `packages/extra/monitoring/values.schema.json`,
`README.md`, and the mirrored `openAPISchema` on the
`monitoring-rd` ApplicationDefinition through the standard
`make generate` (cozyvalues-gen + hack/update-crd.sh) pipeline.
Also refreshed the stale annotation comment on the CR that still
claimed the chart provisions three KeycloakRealmGroups.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
Ivan Okhotnikov
|
1679c84bb0 |
chore(monitoring): regenerate extra/monitoring README
Column widths in the OIDC table contract by one character once the tenant-facing `grafanaAdmin` row is gone. Regenerating the table via `make generate` (cozyvalues-gen) so the pre-commit `make generate` diff check in CI stays clean. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
e05ca91ff0 |
fix(monitoring): keep OIDC grafanaAdmin as a platform-only lever
The tenant-facing `Monitoring` CR exposed `spec.oidc.grafanaAdmin` via `packages/extra/monitoring/values.schema.json`, which the ApplicationDefinition mirrored into `openAPISchema`. A tenant with create/patch on `Monitoring` in their own namespace could set the flag on their release, promoting members of their `<clientId>-admin` Keycloak group to server-level `GrafanaAdmin` instead of the intended org-level Admin. Drop the field from the tenant surface. The platform bundle still flips it on for the `monitoring-system` component through `Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin` (the platform Grafana chart continues to read the value verbatim in `packages/system/monitoring/values.yaml`); tenant Grafana releases inherit the chart default `false` and stay at org-level Admin. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
ffeb8e5e4f |
fix(monitoring): close self-review blockers for the OIDC PR
Address the two blocking-class findings from the cozy-review pass on
this PR:
(1) Missing PackageSource dependency on cozystack.keycloak-operator
(MAJOR cold-install race).
Without this edge, the monitoring chart can reconcile before the
v1.edp.epam.com CRDs are registered on a fresh cluster — the
`.Capabilities.APIVersions.Has` guard in
templates/grafana/oidc-keycloak.yaml then silently drops all
Keycloak-side objects for mode: System, and Grafana's
auth.generic_oauth block points at a client Keycloak never
provisions. Add the ordering edge, plus a
sources_monitoring_application_dependson_test.yaml regression
guard so a future edit cannot drop it silently.
(2) allow_assign_grafana_admin driven by `.Release.Name ==
monitoring-system` (MINOR fragile coupling).
Replace the release-name sniff with an explicit
`oidc.grafanaAdmin: false` values field, and flip it on for the
platform Grafana release via
Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin
in packages/core/platform/templates/bundles/system.yaml —
mirroring the pattern monitoring-agents already uses for
per-component overrides. Tenant Grafana releases inherit the
chart default false and stay at org-level Admin.
Two new helm-unittest cases in packages/system/monitoring/tests/
lock the values-driven behaviour on both sides
(grafanaAdmin=true → Admin uplift, default → tenant stays at org
Admin), plus bundles_monitoring_grafana_admin_test.yaml in
packages/core/platform/tests/ regression-guards the Package's
component override so the platform release keeps its uplift on
future edits.
Also document the mode-toggle group-membership loss in
docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the
three chart-owned KeycloakRealmGroups; membership is not restored on
toggle-back. Not a bug, but the operator UX cost of the toggle
deserves a bullet in Failure modes.
Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh):
packages/extra/monitoring/values.schema.json
packages/extra/monitoring/README.md
packages/system/monitoring-rd/cozyrds/monitoring.yaml
Test suites still green: packages/system/monitoring 23/23,
packages/core/platform 81/81.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
|