mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
3b5ede1589e93bbd8177f539b623639e7bca4151
201
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
0b284a65d5 |
test(release): isolate report deduplication from workflow attempts (#5282)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, security (a fix that comes through a security advisory), docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Fixes merged to `main` reach the release lines the backport batch runs for automatically, so do not open backport PRs by hand: https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md --> ## What this PR does Pins the report deduplication test to the first workflow attempt. In a CI rerun, the inherited attempt otherwise adds `/attempts/2` to the synthetic run URL and fails the existing exact-URL assertion. Production run URLs and the deduplication assertions stay unchanged. Validation: a source-derived check of the test setup and original URL assertion fails with an inherited second attempt and passes with the proposed first-attempt setup. Controls with the pin after preview or set to 2 still fail. The full Git-backed test was not run locally; CI must verify it. ### Screenshots Not applicable; no UI changes. <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note test(release): make the report deduplication test independent of the surrounding workflow attempt. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Tests** * Updated automated test configuration to account for GitHub Actions run attempts. No user-facing behavior changes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9978771858 |
feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does Adds `observability-proxy`, a new system component that lets a tenant user read the logs of their own namespace from VictoriaLogs with the same Kubernetes credentials they already use for the console, and with Kubernetes RBAC deciding what they can read. This is the first part of the design proposals [Tenant application logs](https://github.com/cozystack/community/pull/95) and [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94); this PR implements the namespace logs root only. Metrics and the per-application roots follow in separate PRs. Why a proxy: today the log stores are shared per root tenant and have no notion of a caller, so tenants cannot be given direct read access without seeing other tenants' logs. The proxy authenticates the caller, authorizes the request from its path, and rebuilds the upstream query so that its scope cannot be widened by the client. Backward compatibility: this adds a new package and a new HelmRelease; nothing existing is renamed, removed or changed, so no migration is needed. What is included: - `cmd/observability-proxy` and `internal/observabilityproxy`: the proxy. - Serves `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>` for the VictoriaLogs read endpoints the console needs (`query`, `hits`, `stats_query_range`, `tail`, `field_values`, `stream_field_values`, `facets`). Anything else under the root answers `404`. - Authentication by `TokenReview`, authorization by one `SubjectAccessReview` built from the path (`list` on `cozystack.io/logs` in the namespace), both cached briefly. - The upstream request is rebuilt from an allowlist of parameters; the namespace scope is always forced as `extra_stream_filters`. Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL `options(...)` block, and tenant headers such as `AccountID`/`ProjectID` are not forwarded or are refused. - Non-canonical paths (`..`, `//`, encoded separators) are refused with `400` before any authorization is built, because the console nginx matches on the normalized path but forwards the raw one. - Time window, step, bucket count and result limits are clamped; per-user rate, concurrency and open-tail limits answer `429`. Invalid time, step and limit parameters answer `400`. - The store is resolved from the namespace's `namespace.cozystack.io/monitoring` label. - Live `tail` streams are passed through unbuffered, and the upstream tail request disables compression so headers reach the client before the first line. - `packages/system/observability-proxy`: chart and image build, wired into the root `make build`. - Platform wiring: a new `cozystack.observability-proxy` PackageSource, enabled by default on the full variants and opt-in on the slim ones, the same way as `monitoring-agents`. - RBAC: an aggregated `cozy:observability:view` ClusterRole grants `list` on `cozystack.io/logs` to tenant `view` and above. No API server serves this resource; the rule exists only for the proxy's `SubjectAccessReview`. The proxy's own ClusterRole is limited to the calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch` on namespaces). - Console nginx: when the proxy is not installed or cannot be reached, the observability roots answer `404` with a Kubernetes `Status` body, which the console shows as not available on this cluster; the proxy's own `502`/`504` for an unreachable store pass through unchanged. - Console: one regex `location` in the console nginx routes the four observability roots to the proxy ahead of `/apis`. The upstream is resolved at request time, so the console starts even when the proxy is not installed; those paths then answer `404` with a Kubernetes `Status` body. The console Deployment now receives the cluster domain for that resolution. - E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/` with two tenants, covering `401`/`403`/`200`, scope-widening attempts, path normalization and a long-lived `tail`. #### How it was tested - `go test -race` and `go vet` for the proxy packages. - helm-unittest for `observability-proxy`, `dashboard`, `tenant` and `core/platform`; bats tests for the console nginx routing. - The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs as the root store and refuses to run when a real one exists. Its cases were replayed by a script against the real store on a test cluster. - Verified on a test cluster: - `401` without a token and with an invalid token, `403` for another tenant's namespace, `200` for the caller's own. - Client `extra_stream_filters`/`extra_filters` (GET and POST), an `AccountID` header and another tenant's stream named inside the query do not widen the scope; `options(...)` answers `400`; a path with `..` pointing at another tenant answers `400`; an endpoint outside the allowlist answers `404`. - Through the full browser chain (OIDC login, gatekeeper, gateway, console nginx), a single `tail` stayed open for more than two minutes without reconnecting, and requests from the same session to other tenants' roots answered `403`. - Not verified: the `429` limits under real load. `golangci-lint` has not been run locally. #### Known limitations - The proxy is reachable only through the console route; tenant workloads have no egress rule to it. Every request takes its concurrency or tail slot before the body is read; the token and access reviews and the body read run under `--request-timeout`, so a stalled kube-apiserver answers `503` rather than holding the request; and writes to the client carry a deadline (renewed per line for tails). `--request-timeout=0` disables those bounds and should not be used in production. - The first lines a newly started pod writes can be missing from an already open `tail`, although they are in the store. The likely cause is that Fluent Bit picks up a new log file later than the VictoriaLogs live-tail window; later lines of the same pod arrive. A history query shows them. - The per-application roots (`.../<plural>/<name>/log`, `.../metrics`) and the namespace metrics root answer `404` in this PR. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/787 (docs land once the implementation PRs merge) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: `cozystack/website` is reached. A new platform component (`packages/core/platform/sources/observability-proxy.yaml`) triggers `content/en/docs/next/guides/platform-stack/_index.md` and `operations/configuration/licenses.md`; adding it to the system bundle (`packages/core/platform/templates/bundles/system.yaml`) may also need a line in `operations/configuration/variants.md`. A user-facing page for the logs API and the new tenant `view` permission is not required by the trigger map but would be useful. Tracked in cozystack/website#787. Other repositories: no trigger matched. `packages/core/platform/values.yaml` is not changed (only a test fixture), no package under `packages/apps/` gains a values field, and no CRD changes. The root `Makefile` `build` target builds one more image; this does not touch `hack/` or the make targets `ccp` relies on, so `ccp` is not considered affected, but a reviewer may want to confirm. ### Release note ```release-note feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added an observability proxy for tenant-scoped log access through the console, including live log streaming. Requests are authenticated, access-controlled, and subject to configurable rate and concurrency limits. * Added platform packaging and deployment support, with console routing for observability requests. * **Tests** * Added automated coverage for access control, tenant scoping, query handling, streaming, proxy behavior, and deployment configuration. * **Documentation** * Updated end-to-end test guidance with host requirements for running the observability proxy suite. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9fe06a874b |
test(release): isolate report deduplication from workflow attempts
CI reruns expose GITHUB_RUN_ATTEMPT to a test that expects a URL for
the first attempt. Keep its synthetic run identity independent of the
surrounding workflow.
Fixes:
|
||
|
|
2c1cb2f41c |
feat(release): drop an original from one line with run --drop
A fix the rules take for every line can still be wrong for one of them, typically because it builds on code that line does not have. The only ways to keep it out were backport/skip, which keeps it off every line, and --defer, which holds it out of one batch: the next batch tried it again, and a conflict that would never resolve came back every run. run --drop N --reason R records a drop in the line's record, bound to the original's merge commit on main, and every later record carries it, a new batch's included. What the open batch carries of a dropped original comes off it the way --defer takes one off; any other drop changes only the record, so a batch on a line that moved since stays where it is. --undrop withdraws one. Both need an open batch, as every write to the record does. A drop comes after the rules and the carries: backport/skip, rules 2 to 4 or the line itself decide first and the drop stays on record, and backport/include does not lift it. check notes a dropped original, the body lists it, fold refuses it, and a record whose drop names another original's merge stops every command. Each report keeps the record's drops with the merge commit each is bound to and its reason, held on read to the same rules, so an earlier report still says why after an undrop. A reason now travels from a body to terminals, bodies and comments, so a reason or a file name in a record is believed only in the form a run writes it, one clean line, and so is a report's diagnostic. Output that repeats a reason, file names, a report ID or a state GitHub reports escapes it. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
79b335e461 |
fix(release): close the batch --defer leaves empty, recording why while it is open
docs/release/backport-tasks/failed-ci.md says to defer the culprit with run --defer --push. When the culprit was all the batch carried, the push stopped with "close batch pull request #N instead", and closing it by hand left nothing that said why: once closed, the deferred original was simply missing. The run now closes the batch itself, and does every write while the pull request is open: the report, then the body, whose record keeps the deferrals and says the batch is closing, at the head the rebuild started from. The close and the deletion of the branch go only while GitHub and the remote still show that head, and the deletion is leased on it. A close that stops part way is finished by the next run --push, before anything else, and by no other command; a head the close did not freeze is refused, never adopted. The run that finishes it stops there, and fails when it was asked for --defer, since no open batch is left to defer on. Recovery never writes the closed pull request. A same-day batch that reuses the branch name and fails to open its pull request leaves a branch the next run names for a person to delete. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
5a997fb7d4 |
feat(release): post an immutable report of every classification and write record version 2
Each run rewrote the batch body whole, so the only account of what a run found was gone at the next run, and a merged batch kept only its last one. Nothing said what a classification was made of either: not the heads it read, the policy, the tool or the flags. A run now makes a report: every original merged into main since the line branched, in exactly one category, with what the run read. Its ID hashes the classification and names no pull request, so a new batch's report is whole before GitHub numbers the pull request, and a run that finds the same as the last one makes the same report, which is posted once and keeps its first bytes. The body carries the latest report under the record; a comment the tool never edits keeps each one, so report reads any of them back, from merged batches too. The comment goes up before the push and the body edit on an open batch, and after the pull request a new batch opens with the report in its body; a later run posts a report the body kept but no comment did, byte for byte, before its own. Every write is preceded by a read of the pull request, and nothing is written once it is merged or closed. Both the comment and the body are rendered and measured before the first write, so an oversized one stops the run with nothing written. The record becomes version 2: what it was built from, with the run's --no-workflow-files, which --defer keeps rather than resets, and the report it goes with. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
59cbf4b28e |
docs(observability-proxy): say what the request timeout bounds
The flag now also bounds the body read and the token and access reviews, and 0 lifts all of those at once; an operator raising it for slow store queries needs to know that. The store comment named a design document the repository does not have. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
d2ef5e031f |
docs(observability-proxy): describe the proxy as it stands
The package and serve comments described a delivery plan rather than the code: the command named metrics it does not serve, and serve gave a plan as the reason to refuse unknown roots before authenticating. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
9d9f5fcb6c |
docs(observability-proxy): name the source of the logs bucket default
The default of 11000 buckets was justified by a metrics proxy that does not exist on this branch. The number is Prometheus's cap on points per series in a range query, which the metrics root reuses. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
b1108e836a |
fix(observability-proxy): refuse to resolve a store before the cache syncs
An absent namespace resolves to the root store, and before the informer's first list every namespace is absent, so a request served in that window read a labeled namespace's logs from the wrong store. Readiness keeps the Service from routing there, but a client that addresses the pod directly is not stopped by it. The resolver now fails while the cache is unsynced, which answers 503 like any other resolution error. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
cd04c3e6c4 |
fix(observability-proxy): count the aligned buckets in the logs step
vlselect widens the window to step boundaries and returns a zero-filled bucket for each one, so a start off a boundary yields one bucket more than the span divided by the step. Sizing the step from the unaligned span let hits and stats_query_range return MaxBuckets+1 buckets. The minimum step now spans MaxBuckets-1 steps, and the smallest accepted --logs-max-buckets rises to 4 so that this step still fits the third-of-range cap that keeps alignment inside the window. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
2d8de08724 |
fix(observability-proxy): refuse positional arguments
The flag parser stops at the first argument that is not a flag, so a stray word in the container args made every flag after it fall back to its default without an error, the root tenant included. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
2e3edcf30d |
fix(observability-proxy): refuse a non-finite per-user rate
NaN fails the negative check, as every comparison does, and +Inf refills a bucket instantly, so either value switched the per-user rate limit off while the flag looked set. Both now stop the proxy at startup. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
5945d2ddd1 |
feat(release): read a line's record from its newest batch pull request in any state
The deferrals a batch recorded were read only from the open batch pull request. Once it merged, a conflict it had deferred showed as missing and failed check and the cut, until some later run picked something else and opened a new batch; a run that picked nothing opened none. run, check, fold and notify now find the line's record in the newest batch pull request whatever its state: one opened from a batch branch of this repository by an author the batch trusts. A record that is in doubt stops the command instead of letting it fall back to an older one: a block by an author no longer trusted, a bot's batch whose block was removed, a record that does not read whole, a closed record newer than the open batch, or no record within the pages read. The record is still written only to the open batch, and a merged or closed batch is never written again, so a run that picks nothing with no batch open keeps nothing and says so. The reader takes the version 2 record the next change writes; the writer stays at version 1. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3fc444874e |
fix(release): print the batch pull request's URL when --push publishes it
run --push opened or updated the batch pull request and said nothing about which one, so the scheduled job's log and a maintainer's terminal had no link to follow. gh pr create prints the URL; the run now passes it on, on stderr, so stdout stays the body alone, which the scheduled script saves as it is. An update prints the open pull request's URL. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
015fc51c0d |
fix(release): name policy rules instead of internal numbers in output
fold, run --defer and the run's own stderr printed "rule 101" for an original merged before its line started under the policy, and "rule 100" for backport/skip. Those numbers exist only inside the tool; the policy has six rules, so a reader looking for rule 101 finds nothing. Output now names such a rule in words, and the policy's own rules by their number. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
35c47f20c1 |
fix(release): fail check --cut while a workflow original waits alone
check --cut returned 0 whenever nothing was missing and nothing was only on the batch, so a workflow original waiting for a maintainer passed the cut once the pick beside it was labelled backport/skip or reached the line by hand. The waiting original was not even printed: the case that held it swallowed the warning the plain check gives. check now sorts every original into one disposition and exits once, so a wait fails the cut on its own. A backport/skip original the open batch still carries is named with the run --defer that takes it out. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
9e8e10c307 |
fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does Refs #5205. The S3 endpoint the platform backups use is copied twice, and neither copy is refreshed after `publishing.host` changes. The COSI sidecar writes `BucketInfo` once per `BucketAccess`, so the credentials Secret `bucket-cozy-backups-system-credentials` rendered from it keeps the old endpoint. The Velero BSL and the default Strategy CRs keep whatever endpoint the last render of `backupstrategy-controller` saw. The default objects gate only acted on missing objects, so backups kept going to the old host and nothing reported it. The gate now also forces a render when a copy is stale: - the bucket's `<bucket>-system` release, when the credentials Secret's endpoint differs from the COSI Secret it is rendered from; - `backupstrategy-controller`, when the endpoint an object was rendered with differs from the credentials Secret's. The chart stamps the rendered endpoint on the BSL and on every default Strategy as `backups.cozystack.io/endpoint`, because the endpoint itself sits at a different path in every kind. Objects without the annotation count as stale until a render stamps it; `MinForceInterval` bounds the forces. Stale credentials are forced before the BackupClass and the routed objects are read, so an absent BackupClass or an unreadable Strategy still fails the check but does not keep the old endpoint in the credentials Secret. A `BucketInfo` without an endpoint is not compared, since rendering from it would write an empty endpoint or fail the bucket release. Repairing missing objects keeps working while the credentials comparison is stale or fails, and a `systemSecretName` that is not derived from a COSI Secret (no `-credentials` suffix, or the raw COSI Secret itself) is not compared at all. RBAC is unchanged: the controller already holds cluster-wide `get` on Secrets. No new metric: nothing would consume it. What this does not do: the COSI Secret still holds the endpoint the `BucketAccess` was granted with, so after a host change the endpoint follows only once an operator deletes the `BucketAccess` and forces `bucket-cozy-backups` to recreate it (this issues new S3 keys). From there the gate replaces the remaining manual steps of the recovery described in the issue. Automatic re-grant is left for a separate issue, and the operator runbook goes to the website, hence `Refs` rather than `Fixes`. Known limitation: the credentials comparison runs only when `backupStorage.systemSecretName` is `bucket-<bucketName>-system-credentials`, the Secret the forced `bucket-<bucketName>-system` release renders. If an operator sets `bucketName` and `systemSecretName` inconsistently, the comparison is switched off rather than forcing a release that never rewrites the compared Secret, so a stale endpoint in that Secret is not repaired. Repairing a missing credentials Secret still forces the release derived from `bucketName`, as before this PR. Tested with unit tests for the gate (including the comparison normalisation shared with the chart helpers), helm-unittest for the annotation and the wiring, and a bats guard that every rendered Strategy CR and the BSL carry the annotation. On a live cluster, a server-side render of the chart produced exactly the annotation the gate derives from the real credentials Secret, so the steady state does not force anything. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: an operator runbook for changing `publishing.host` with platform backups enabled (re-grant the `BucketAccess`, then let the gate converge; restart ClickHouse pods that read the credentials from environment variables). Not filed yet, pending approval of the text. - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup strategies and storage locations now record their storage endpoint, enabling detection and repair of outdated endpoint settings after bucket access changes. * Bucket credentials and backup objects are refreshed independently when their endpoints are missing or out of date. * **Documentation** * Clarified when endpoint changes trigger updates and how suspended releases are handled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a56ff5050f |
test(observability-proxy): accept the smallest satisfiable step cap
Startup refuses --logs-max-buckets below 3 and --logs-max-range below 3s, but only the refusals were tested, so moving either bound up by one went unnoticed. Pin the smallest accepted pair. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
ca7f283b6a |
fix(observability-proxy): require a range and bucket count the step cap fits
The step is capped at a third of the maximum range so that vlselect's alignment to whole steps cannot widen the window past that range. With fewer than three buckets, or a range under three seconds, the minimum step exceeds that cap and the bound no longer holds, so such settings are refused at startup. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
8eef2fa5d2 |
fix(observability-proxy): reject negative limits at startup
Zero is the documented way to disable a limit; a negative value disabled it too, silently. Refuse it so a typo in the chart cannot turn off a per-user or total cap unnoticed. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
1c345da367 |
fix(observability-proxy): reject negative timeouts at startup
A negative duration silently disables the bound it configures, so a typo in the chart would turn off the request deadline or the tail idle close without any sign. Refuse it when the flags are parsed. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
8b0ac342b5 |
feat(observability-proxy): add the observability-proxy command
The deploy unit builds an image and chart around this binary, so its flags follow the agreed contract: listen and probe addresses, cluster domain, root tenant and auth cache TTL, plus the logs limits. Defaults are chosen so a stock install is safe without tuning; readiness waits for the namespace cache because store resolution reads it on every request. On shutdown open tails are cut after a grace period so their clients reconnect with since to another replica. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
7097cc5cda |
feat(release): make backports automatic with backport-batch (#5106)
## What this PR does Backports become automatic. Fix merged to `main` with `fix` title, or `security` for fix that comes through security advisory, reaches release lines through batch that runs on Monday and Thursday, newest line first, nobody requests, labels or opens backports by hand. Batch runs for two newest lines with stable release (`released-lines` in policy block) and every line branched after them that has no stable release yet: with 1.6 and 1.7 released, `release-1.8` created at `v1.8.0-rc.1` joins them, and `v1.8.0` drops 1.6. Line counts as released once `vX.Y.Z` tag without pre-release suffix exists on origin. Rules are in `docs/release/backports.md`, `cmd/backport-batch` applies them, `.github/workflows/backport-batch.yaml` runs it. Design is in cozystack/community#114, merge shape of each original in cozystack/community#119. - `backport-batch run --line release-X.Y` takes PRs merged to `main` since line branched (title and body read from merge commit) and skips what rules exclude: other types and `deps`, breaking changes, platform migrations, `targetVersion`, changed CRD documents, API changes that API review gate calls sizeable (batch runs `internal/apigate` itself whenever PR changes a file the gate parses: `pkg/apiserver/apiserver.go`, an ApplicationDefinition, or YAML under `packages/` or `internal/` naming `CustomResourceDefinition` or `APIService`), and on older line what next newer line doesn't carry yet. The rest goes onto `backport-batch/<line>-<date>` in merge order. PR merged to `main` with merge commit goes as its own commits picked with `cherry-pick -x --signoff`, so each keeps its author, message and sign-offs and gets sign-off of whoever runs the batch (with `-m 1` author was whoever merged on `main`), merges of `main` into PR branch are left out, and merge commit joins them to the batch with original's merge message and `Backported-from:` trailer. Squash-merged PR goes as one pick. Conflict on any commit takes branch back to where it was before that original, it's listed with the commit that failed, run goes on. `--push` opens or updates draft PR, `--continue` carries on open batch PR as fast-forward only, PR body it didn't write is never rewritten. Same-day run over batch branch whose PR never got opened refuses and names `run --line <line> --onto <head> --push`, which continues it. Run or `--defer` that resets local batch branch over commits only it has prints `git branch` command that restores them. - `run --defer <N> --reason <why>` takes original out of open batch (usually the one failing CI is pinned on). It rebuilds branch replaying batch unit by unit without it, so merges and pushed conflict resolutions keep their shape, and only this rebuild pushes with `--force-with-lease`. Deferred original comes back in next batch. - Line carries something only if its own history shows it: picks whose `-x` line, the one git appends, names PR's merge commit or every commit of its own, or pick that changes nothing. Picked commits keep their messages and batch merge carries PR body, so text in them (`-x` lines, `Backported-from:`, `Backport of #N`) counts only on commits from before `batch-only` day in policy block, where it records old hand backports and batches. Otherwise text in a merged fix could mark another fix as already backported, and it would never reach the line. PR against line never counts. Open batch PR counts only from this repo and from author with write access or the batch's own bot (`BACKPORT_BATCH_BOT`, exact login, `cozystack-ci[bot]` when unset, so maintainer's run by hand continues scheduled batch instead of opening second one), and is read the same way. Actions bot isn't trusted anymore, any workflow token writes as it. - Any failed GitHub API or git call stops the run with message naming the call, no retries. Failed permission lookup used to count batch PR out as look-alike, so `--continue` opened second batch PR and `notify` posted second comment; now `run`, `check`, `verify` and `notify` exit 2 and write nothing. - Policy applies fully from `release-1.7`. `release-1.6` takes only fixes merged from 2026-10-26, older ones need `backport/include`. Start days are in `backport-policy` block on the policy page, test keeps page and code in sync. - `backport-batch check --cut` replaces `backport-audit` as gate before patch cut and before every rc after the first, it also fails while a fix is only on open batch PR. Plain `check`, which scheduled run uses, lets those through. `backport-batch fold` checks conflict resolution in the same shape: merge on batch head whose second parent is straight line of picks of PR's own commits in order, each with its `-x`, only PR's files (one pick for squash). `backport-batch verify` holds every original on batch to that shape, clean picks included, and `pr-labeler.yaml` runs it on every push to batch PR: `pull_request_target` so it reaches `release-*`, tool built from `main`, batch head fetched as data, read-only token. Batch PR body says per original how many commits it brought and which picks differ from their commits. `backport-batch notify` reports batch that is still open from previous run and deferred conflicts in one comment on batch PR, edited in place, with `BACKPORT_ALERT_TEAM` mention. - `hack/backport-batch-scheduled.sh` is what the workflow runs and maintainer can run by hand (`--dry-run` changes nothing on GitHub): for each line newest first `run --continue --push`, `check`, `notify`. It pushes with `BACKPORT_BATCH_TOKEN` only and fails without it, never `GITHUB_TOKEN`, because its pushes don't start CI. Workflow runs it as cozystack-ci App: it mints token for this repo only with Contents and Pull requests write, passes it as `BACKPORT_BATCH_TOKEN`, commits as `cozystack-ci[bot]`, and sets `BACKPORT_BATCH_BOT` so the tool trusts that bot's batch PR and comment. Job timeout is 30 minutes, under the token's hour. - App has no workflows permission, so workflow runs the script with `--no-workflow-files`: original that changes `.github/workflows/` isn't picked and waits on batch PR for maintainer, listed with its workflow files in body and in notify comment. It's never a conflict. `check` warns about it while it waits and `check --cut` fails on it. Maintainer brings it over with own token by `run --continue --push` without the flag, or takes it out with `--defer`. - Deferred conflicts and failed batch CI wait for maintainer. `notify` keeps one comment on batch PR listing stalled batch, conflicts and workflow changes, mentioning `BACKPORT_ALERT_TEAM` (team or login) when it's set. Failed batch CI isn't in the comment, it shows on batch PR checks, and `check --cut` fails while batch is unmerged. - Two steps that need judgement (resolve conflict, pin failed CI on original and defer it) are written as runbooks in `docs/release/backport-tasks/`: inputs, the only allowed output, check it must pass before push, what never to do. - `backport/include` and `backport/skip` let maintainer override rules for single PR, they are new `backport/` category in `labels.yml`. Skip wins over include, if open batch already picked it `--defer` takes it out. - Text written by other people (titles, release notes, reasons, file names) goes to batch PR body only inside code spans or release-note block, control, bidi and zero-width characters are dropped, file names in deferred block included, git args go after `--end-of-options`, no shell. - Removed: `cmd/backport-audit`, `backport.yaml`, `kind/backport` and `kind/backport-previous` labels with aliases and stale exemption, trailer-check exemption for bot branches. `Commit trailers` job exempts only `backport-batch/*` branches of this repo, changelog instructions read originals from batch commits. Exemption lives in main's `pre-commit.yml`, and batch PR runs its line's copy: `release-1.7` has its own copy that exempts only old `backport-<pr>-to-<branch>` branches, `release-1.6` has no such job. So on `release-1.7` picked commit with bad trailer fails batch CI until line gets main's copy. - `hack/close-manual-backport.sh` runs from `pr-labeler.yaml` on `opened` and closes PR against `release-*` from outside contributor (association contributor, first-time contributor or none, permission read or none). Owners, members, collaborators, anyone with triage or more and bots are left alone. When GitHub's answer on author's permission can't be read, it closes nothing and fails the job naming the failed read. Maintainer keeps release-only fix by reopening it. Release-only PRs are occasional and opened by maintainers, for fix with no counterpart on `main`, or one that on `main` came inside another PR or was written for `main`; if its counterpart on `main` is a fix the batch would take, the batch skips it once picking it changes nothing on the line, otherwise maintainer labels it `backport/skip`. - `docs/release.md` is split into pages under `docs/release/`, PR template doesn't ask for `kind/backport` anymore. `security` type is added to commit format in `docs/agents/contributing.md`, PR template, Gemini review guide and labeler's list (labeler maps no `kind/*` to it, `labels.yml` has no fitting one). - `SECURITY.md`, `docs/security/incident-response.md` and `GENERAL_TECHNICAL_REVIEW.md` now say two latest minor releases are supported, same window the batch runs for, and link the backport policy for which fixes go there. Dry runs without push, on 2026-11-06, with `--no-workflow-files` as the workflow runs it: `lines` gives `release-1.7` and `release-1.6`. On `release-1.7` batch picks 12 fixes, defers 3 conflicts (#4756, #4867, #5082), leaves #4888 and #4971 to a maintainer because they change workflows, and skips 49. On `release-1.6` it picks #5088, defers 3 conflicts (#5071, #5086, #5087), leaves #5142 to a maintainer, and skips 590. `verify` passes both batches. Setup after merge: no new secret. Batch runs as cozystack-ci App from existing `COZYSTACK_CI_APP_ID` and `COZYSTACK_CI_PRIVATE_KEY`, job fails when either is missing. App's installation already has Contents and Pull requests write on all org repos, and no Workflows is needed: PRs that change `.github/workflows/` wait for maintainer, who brings them over with own token as "Changes to workflows" in `docs/release/backports.md` says. Today that's #4888 and #4971 on `release-1.7` and #5142 on `release-1.6`. Set `BACKPORT_ALERT_TEAM` to a team or a login (for example `cozystack/maintainers`): unset, batch comment mentions nobody and nobody is notified. Once this PR is merged, `Backport batch shape` can be required check in the release lines ruleset (`release-*` excluding `release-*.*.*`), with "Do not require status checks on creation" on, because `cut-prerelease.yaml` creates `release-X.Y` by push. `PR Auto-Label` with both new jobs runs on `pull_request_target`, so it needs #5135 sorted before 2026-12-02. `kind/backport*` labels have to be deleted by hand, label sync doesn't delete. ### Screenshots Not a UI change. ### Downstream repositories cozystack/community `contributors/README.md` lists commit types, so it needs `security` too, it has no backport label text. Website has no backport label instructions, and its links to `docs/release.md` carry no anchor, the page stays as index. Trigger map lists commit convention change for cozystack/ccp: this adds `security` type, but not message format or trailer, and `package-bump` writes only `chore(...)` commits and restates the trailer, so nothing changes there. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: add `security` to commit types in `contributors/README.md` ### Release note ```release-note feat(release): backports are automatic. Fixes merged to main with fix or security title reach release lines through a batch built by cmd/backport-batch every Monday and Thursday, by the rules in docs/release/backports.md. The batch runs for the two newest lines with a stable release and for a line in its rc window: the release-X.Y branch the first rc creates joins at once, and its vX.Y.0 drops the oldest line. Features, chores, docs, dependency bumps, breaking changes, migrations, CRDs and API changes that the API review gate calls sizeable wait for the next minor. The policy applies fully from release-1.7, release-1.6 takes only fixes merged from 2026-10-26. Each original reaches the line as its own commits under a merge commit, keeping their authors and sign-offs. Maintainers override the rules for single PR with backport/include and backport/skip labels and take an original out of a batch with backport-batch run --defer. backport-batch check replaces backport-audit as the gate before a patch release and before a later rc. The label-driven backport workflow and kind/backport labels are removed. Backport PR opened by hand against a release branch by author with read access or less is closed automatically. The scheduled batch runs as the cozystack-ci App and needs no secret of its own; a fix that changes .github/workflows waits on the batch pull request for a maintainer, since the App may not push workflow changes. Release documentation now lives in one page per topic under docs/release/. ``` |
||
|
|
dfd7cbed36 |
fix(operator): reject a Git cert secret before installing CRDs and Flux
CRDs and Flux are installed before the platform source, so refusing a cert secret on a Git source only when the source is written exits the operator after a partial install. The check now runs right after flag parsing, and the source install reuses it. The guard runs on every start, so a Git source without a cert secret is now covered over https and ssh: a guard that misfired there would crash-loop the operator for every Git platform source. The update test also renames the cert Secret, as a CA rotation does. Assisted-by: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
eaa0169ff3 |
chore(release): retire the backport bot, its labels and backport-audit
The batch replaces the label-driven backport workflow and the audit that gated a cut against it. The kind/backport labels, the stale exemption for them, their contract tests and the audit's make target go with them. labels.yml gains a backport/ category instead, with backport/include and backport/skip, the overrides the batch reads off a merged pull request. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
9f8c88b4f1 |
feat(release): build the backport batch with backport-batch
The rules need a tool that applies them the same way on every run and shows what it did. backport-batch reads main's first-parent history, judges each pull request by the policy block it reads from the page on main, and picks what qualifies onto one batch branch per line. Each original lands as its own commits under a merge that names it, so authors and sign-offs survive and the line's own history shows what it carries. check is the gate before a cut. fold holds a hand resolution of a conflict to the batch's shape, verify holds a whole batch to it, and notify reports what waits on the batch for a maintainer. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c15a0d01b8 |
feat(operator): let the platform source trust a private registry CA
The operator generates the cozystack-platform OCIRepository from its own arguments and rewrites it on every start, and cozystack-packages copies that spec. A registry behind a private CA therefore had no durable way to be trusted: a certSecretRef patched onto the live object is dropped on the next operator restart, after which source-controller fails TLS and platform updates stop arriving. --platform-source-cert-secret sets spec.certSecretRef from the operator itself, so the CA (or a client certificate) survives the rewrite. Git sources are refused instead of ignored, because GitRepository has no certSecretRef and Flux reads a Git CA from the source secret instead. Assisted-by: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
cd91138f33 |
fix(backups): re-render the default backup objects when the S3 endpoint changes
The S3 endpoint the platform backups use is copied twice and neither copy is refreshed. The COSI sidecar writes BucketInfo once per BucketAccess, so the credentials Secret rendered from it keeps its endpoint, and the Velero BSL and the default Strategy CRs keep whatever endpoint the last render of backupstrategy-controller saw. The default objects gate only acted on objects that were missing, so after a publishing.host change every backup kept going to the old host with nothing reporting it. The gate now also forces a render when a copy is stale: the bucket's <bucket>-system release when the credentials Secret's endpoint differs from the COSI Secret it is rendered from, and backupstrategy-controller when the endpoint an object was rendered with differs from the credentials Secret's. That endpoint sits at a different path in every kind, in some only as a bare host, so the chart stamps the rendered value on each object as backups.cozystack.io/endpoint and the gate compares that. Objects rendered before the annotation existed count as stale until one forced render has gone through, which stamps it. The COSI Secret itself still holds the endpoint the BucketAccess was granted with; it follows a host change only once the BucketAccess is re-granted. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
4087218a02 |
docs: move the repo-only guides onto the website (#5034)
## What this PR does Seven guides under `docs/` were operator facing and written alongside the code, and left in the source tree where no user looks. They are on the website now, so this deletes them here and repoints every reference. ``` docs/operations/backup-classes.md -> operations/services/backup-classes docs/operations/resource-presets.md -> guides/resource-management docs/storage-immutability.md -> operations/storage-class-immutability docs/operations/system-memory-limits.md -> operations/configuration/system-memory-limits docs/gateway-classes.md -> networking/gateway-classes docs/observability/distributed-tracing.md -> operations/services/monitoring/distributed-tracing docs/operations/seaweedfs-capacity-for-backups.md -> operations/services/object-storage/capacity-for-backups ``` `docs/release.md`, `docs/deprecating-applications.md` and the two SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`, a migration and a bats suite read them by path, which is the test from #3777. References point at `https://cozystack.io/docs/v1/...`. The site rewrites that short prefix to the latest version in `layouts/404.html`, so the links follow releases and pin none. Backup Classes, resource management and `storageClass` immutability exist in the latest version already. The other four pages are only in `next/` until 1.7.0 is promoted, and their links start to resolve then. The generated mongodb files come from `values.yaml` through `make generate` with cozyvalues-gen v1.7.0, the version pinned in `.github/workflows/pre-commit.yml`. Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114) and kubernetes-nodes (223), go tests for backupcontroller, presets and the operator, rd-presets-check, the migration-50 bats suite. ### Downstream repositories - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#763 Walked the rest of the trigger map against the diff. No provider follow-up: the mongodb `values.schema.json` diff is one `description` string, with no field added, removed or renamed, no enum, no default, and no `kind`, `plural` or `release.prefix` change. `packages/core/platform/values.yaml` changed in comments only and no key, so the hand written platform-package table on the site is unaffected. Please merge after cozystack/website#763, the links in the code only resolve once those pages deploy. ### Release note ```release-note docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there. ``` |
||
|
|
a444ac4498 |
Merge branch 'main' into fix/tap-pin-digest
Reconcile the tap digest-pin change with #5071 (no-install-on-connect), now merged into main. Keep this branch's resetTapPinState, the verify-before-pull Trust text, and the index-README trust model, while adopting #5071's wording that installing (not tapping) runs charts and dropping the stale "re-registration" phrasing from the shared tap.go and rest.go comments. Conflicts resolved in cmd/cozypkg/cmd/tap.go, pkg/registry/core/tap/rest.go, and hack/packages-index/README.md. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
09b0688abd |
fix(marketplace): fail closed on an unparseable verified digest, and cover the generation gate
A verified cosign signature whose manifest digest could not be parsed fell back to an unpinned tag pull, which reopens the tag-mutation window the verification was meant to close. Return an error instead. Add a test that isolates the status.observedGeneration == metadata.generation conjunct of artifactReady; the existing tests exercised only the tag-prefix conjunct, so a mutation of the generation check went uncaught. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
0195e73d38 |
fix(marketplace): verify the signature before pulling, and anchor the identity match
With --certificate-identity and --certificate-oidc-issuer, `cozypkg tap` pulled by tag without verifying and applied the artifact's PackageSources with ForceOwnership; only the Flux source carried the verify policy. A tag moved by someone without the signing identity could land unsigned content in the cluster. Verify the signature at tap time and pin the verified digest, so the content pulled and applied is what a trusted identity signed. Flux evaluates a cosign matchOIDCIdentity issuer and subject as Go regexps, and they were written unanchored. An exact identity such as alice@example.com would also accept malice@example.com, and the dots would match any character. Anchor and quote both values at the API and the CLI. Reject --tag together with a digest reference instead of silently ignoring the tag. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
96758b9e3d |
fix(marketplace): drop the index-entry digest from the tap pin change
The index-entry `digest` field, and the gate and README claims around it, assume a published index that does not exist yet: decision 0002 in the marketplace proposal defers it, and #4981 tracks that work. The gate also never compared a recorded digest against the artifact it verified, so the field added surface without the check that would make it trustworthy. Drop it from this PR. Digest pinning still works from an explicit oci://host/org/repo@sha256:... reference and from the digest a tag resolves to; only the index-entry path is removed. The tag-based pinning and the verify-flag wording stay in the index README. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
2f39e304f0 |
fix(marketplace): reconcile tap texts with no-install-on-connect
Removing on-connect auto-registration made several user-facing and doc texts wrong: they promised that tapping registers a repository's apps in the catalog and runs its charts. Tapping now installs nothing, so the texts told operators to wait for apps that never appear. Reword the `cozypkg tap`/`untap` help, the tap success output (now printing a `cozypkg add <name>` line per materialized PackageSource), the `cozypkg init` scaffold README, the packages-index trust model, the dashboard Repositories banner and per-repository empty state, and the stale re-registration code comments. No behaviour changes beyond the added `cozypkg add` hint in the tap output. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
0fd970304c |
fix(marketplace): preserve verify and pull secret on CLI re-tap
A re-tap applies the OCIRepository under field owner cozypkg via server-side apply; a field this owner set earlier and now omits is removed. Re-tapping to move the pin without repeating --certificate-identity/--certificate-oidc-issuer therefore silently dropped the signature-verification policy, and re-tapping a private repo without --secret dropped the pull credential. The CLI now carries an existing source's spec.verify and secretRef onto the applied object when the invocation does not set them, mirroring the API connect path, so moving the pin never silently downgrades trust or breaks a private pull. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
5747f94781 |
docs(marketplace): document digest pinning and verify on tap
Update the tap command help and the community-index README for the new behaviour: digest references, the automatic tag-to-digest pin, the --certificate-identity / --certificate-oidc-issuer pull-time verify flags, and the optional index entry digest. Extend the generated openapi definitions and model-name helper for the new TapVerify type (code generation is otherwise run in CI). Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
c59ced04cc |
feat(marketplace): record and pin a digest in community index entries
An index entry can now carry a digest (sha256:<64 hex>) alongside its version. Resolving a short name pins that digest, so 'cozypkg tap <name>' consumes the exact content the CI gate verified rather than following a mutable tag; the version is kept informational. A malformed digest in an entry is a hard load error, consistent with the rest of the strict index decoding. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
c56b9876f4 |
feat(marketplace): accept a signature-verification policy at tap connect
A tap can now carry a keyless cosign verification policy (spec.verify), so a
reference is only pulled when signed by the recorded identity. The CLI takes
--certificate-identity and --certificate-oidc-issuer (both required together,
mirroring validate); the API takes spec.verify.{issuer,subject} on the Tap
connect object. Both map to a Flux OCIRepository verify block with provider
cosign and a matchOIDCIdentity entry. A half-configured policy (only one of
the identity pair) is refused rather than set, since a policy that matched any
signer is worse than none. A key-based policy an admin sets on the source by
hand is preserved across repeat connects.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
|
cbd116b40d |
fix(marketplace): reset the operator digest pin on re-tap (CLI)
A re-tap applies the OCIRepository under field owner cozypkg via server-side apply, which cannot clear spec.ref.digest once the operator owns it. A tag-based re-tap would therefore leave the source pinned to the old digest and never follow the newly requested tag. After applying, the CLI now clears the operator's digest pin and the pinned-digest annotation for a tag-based re-tap (an explicit-digest re-tap keeps the digest it just set), alongside the materialized-revision annotation, so the operator re-pins the new reference. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
840a600543 |
feat(marketplace): pin an explicit digest at tap connect (CLI)
buildTapOCIRepository now writes spec.ref.digest when the reference carries one, and the pull resolves the artifact by digest so it fetches the exact content regardless of where the tag points. A tag-only reference keeps being pinned to a digest later by the tap materializer; an explicit @sha256 reference is pinned here at connect time, with any tag left informational. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
b8a129e356 |
feat(marketplace): accept @sha256 digest references in tap connect
parseOCIRef (CLI) and parseConnectURL (API) now accept a pinned digest reference (oci://host/org/repo@sha256:...), with an optional informational tag before it. The digest must be a full sha256:<64 hex>, the same contract the platform source's digest= reference enforces. A reference carrying only a digest no longer defaults its tag to latest, since the digest is the authoritative reference Flux resolves before any tag. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
b83cc087cc |
docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the source tree, where no operator looks. The website carries them now, so a second copy here only lets the two drift again: the Backup Classes page had already fallen behind its original. docs/release.md, docs/deprecating-applications.md and the two SeaweedFS runbooks in docs/operations stay. A migration, a bats suite, a workflow or AGENTS.md reads each of them by path, which is the test for keeping a document in this repository. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
8bcb709ce8 |
fix(fluxcd): refuse a flux-plunger drain deadline that leaves no time
A drain deadline of zero or less is already spent when the drain starts, so flux-plunger would exit holding every uninstall fence without listing one. The chart could produce one: its 40-second floor was a literal, so raising the manager's stop bound or the margin before SIGKILL, which the drain subtraction is already tied to, would leave the floor rendering a non-positive deadline. The binary now refuses such a flag at startup, before it reaches for a cluster, and the chart's floor and the timings the README and the values comment state are held to the same constants. A whole number of a million seconds or more printed in exponent form and was refused as a fraction; the range is now checked first, so it gets the range message. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
42a27567ce |
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
flux-plunger recovers a "has no deployed releases" HelmRelease by briefly suspending it, deleting the stale storage Secret and unsuspending. helm-controller v1.5.x skips the uninstall and drops its finalizer when it handles a delete on a suspended HelmRelease, and it wakes on the same delete event flux-plunger does, so a delete inside that window orphaned the release's DataVolume, PVCs, RBAC and storage Secret. Lifting the suspension on the delete event cannot win that race reliably. Hold the release with a flux-plunger finalizer while it is suspended. The finalizer is applied together with the suspension in one server-side apply. On a delete flux-plunger lifts its suspension and keeps the finalizer, helm-controller reconciles the unsuspended generation and runs the uninstall, and flux-plunger removes the finalizer once helm-controller's own is gone and that generation is observed. This holds whichever controller handles the delete first. A completed recovery lifts the suspension and the finalizer together, and a terminating release never begins recovery. Decide ownership of a suspension by which field manager owns spec.suspend in metadata.managedFields rather than by an annotation. The apiserver transfers the field away the moment an operator, cozyhr resume or a backup controller writes a different value to it, so a later suspension is never misread as flux-plunger's, and one set through the shared flux-client-side-apply manager (platform migrations, the tenant delete hook, a manual kubectl apply) is never claimed. A release deleted under another manager's suspension is left to that manager's decision. Acquire applies without forcing first and, on a conflict, re-fetches and forces only if spec.suspend is still false: a backup controller owns the field at false through a dynamic-client Update, which a non-forced apply can never take, while true is an active suspension to leave alone. Every apply is preconditioned on the resourceVersion and uid of the read that decided on it. Without the resourceVersion, the unforced apply of true over a foreign true is accepted as shared ownership, a window the optimistic-locked merge patch this replaces had closed, and lifting the suspension and the finalizer from a release read as live would delete it without an uninstall if helm-controller had meanwhile handled a delete. Without the uid, an apply to a release that has gone would create an empty one. The apply carries only spec.suspend and the finalizer, so it cannot leak the required spec.interval. Read the release, every re-check and the storage Secret list from the apiserver rather than the informer cache. A lagging cache would let the processed-version guard delete the revision below the one just reclaimed, and the ClusterRole grants secrets get and list but not the watch an informer needs. Emit a Warning when flux-plunger holds a suspension it failed to clear, and Normal events when it leaves an external suspension alone or defers a recovery. An external suspension on a release flux-plunger has recovered before is a Warning instead: it is most likely one an earlier flux-plunger left under the shared manager, which nothing lifts unless an operator does. A release deleted under another manager's suspension is a Normal event when that was a migration's choice, and a Warning when the release carries flux-plunger's processed-version annotation, since then the skipped uninstall is an accident. Keep a single writer with leader election. A second process able to act under the same field manager could clear a suspension the first is still recovering, and no rollout strategy prevents one: the next pod starts as soon as the old one is terminating, and an eviction or a node drain replaces a pod the same way. The old pod keeps its Lease, unreleased, through its stop and drain, and the Lease outlasts both, so the next pod acts only after the old one is done; the price is that a new pod waits for the Lease to run out. The strategy sets maxSurge 0 with maxUnavailable 1 and stays RollingUpdate because existing installs carry the apiserver-defaulted rollingUpdate block and helm-controller applies server-side, where type Recreate over it is rejected on upgrade. Give the finalizer a way out when flux-plunger itself goes. Once the manager has stopped, within a bounded time, and with it every reconcile, flux-plunger drains its finalizers under a deadline that, with the stop, fits inside the pod's termination grace period. A stop that overran its bound is drained too and still reported, and only a pod that led drains. The drain writes its events straight to the apiserver, since the manager's recorder stops with the manager. A recovery in progress is aborted, the suspension and the finalizer lifted together for the next flux-plunger to start again, and a release being deleted has the suspension lifted and keeps the finalizer until helm-controller has run the uninstall. A release helm-controller does not handle in time keeps it. The waiting lives in the controller, where the envtest suites hold it against a real apiserver's generations and finalizers, rather than in a shell script against a model of them. A pre-delete hook on the chart is the backstop for what the drain could not finish, so uninstalling the component never leaves a release that tenant teardown waits on forever. It stops flux-plunger, waits out its grace period, then lifts any remaining suspension and finalizer without waiting again, naming each release it lets go of before helm-controller uninstalled it. It lifts a suspension only where flux-plunger alone holds it, with a JSON patch preconditioned on the resourceVersion and uid it read: kubectl apply --server-side on a release last written by a client-side apply moves that apply's fields to the applying manager and strips every one the body leaves out. The Job's deadline sits under the 300s helm-controller gives a pre-delete hook by default. A Deployment that cannot be stopped or a listing that never succeeds ends the hook before any finalizer is touched, and a finalizer it cannot remove ends it after the sweep; either way it exits non-zero: a failed pre-delete hook leaves the release untouched and helm-controller retries the uninstall, hook included, from a state it can still converge from, where exiting 0 would remove the component and strand the finalizer. A flux-plunger pod still present after its grace period, typically one stuck Terminating on a NotReady node, is only reported, since failing on it would fail every retry until the node returns; because it may still be setting finalizers, they are listed again after the sweep and any found fails the hook. The Job runs once, since helm-controller's retry is the one that starts afresh, and is kept after success so that its report stays readable; helm.sh/hook-output-log-policy would make helm-controller call a nil output writer (fluxcd/helm-controller#1562). A delete still held after ten minutes raises a Warning, and the held state is re-checked on a timer instead of only on helm-controller's next status write. The chart README documents the finalizer and how to clear it. Run the envtest suites on every pull request through a new make target that fetches a kube-apiserver and etcd with a pinned setup-envtest and takes the HelmRelease CRD from the in-tree Flux manifests: the unit tests drive a fake client, and everything the finalizer rests on (apply preconditions, the finalizer set-merge, real managedFields) only holds against a real apiserver. CI caches those binaries, and the target uses them without asking the release index, so a pull request does not depend on their sources being reachable; the control plane gets a longer start timeout because it starts beside the other test goals. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
8c1a78dc9b |
feat(operator): default system namespace memory via LimitRange (#3494)
## What this PR does
System components now get a default container memory limit, so their pod
cgroups carry a `memory.max` and drop out of the Talos userspace OOM
handler's victim set. `cozystack-operator` maintains a LimitRange named
`cozystack-system-defaults` in every system namespace it reconciles.
The Talos handler (v1.12+) scores cgroups with `memory_max.hasValue() ?
0.0 : {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current`
and drops everything that scores zero. Only a memory limit sets
`memory.max`, so only a limit takes a pod out of the candidate set. A
request just moves a pod from BestEffort to Burstable, which under the
default `strictCgroupClassOrdering: true` still means killed second.
`PriorityClass` does nothing here, because the handler reads cgroupfs
and never talks to the apiserver. Victim selection is also unrelated to
the trigger, so a tenant workload thrashing against its own limit can
drive node memory pressure while a system component with no limit is
what gets killed.
An audit of a running cluster found around 70 workloads across ~30
packages with at least one container missing a memory limit. Per-chart
resources would not cover that set, and would not keep covering it as
packages are added. So the LimitRange is written by
`reconcileNamespaces`, which already creates these namespaces and
already tells system from tenant. It also covers init containers, which
#3495 can't reach: one of them without a limit leaves the pod cgroup
without `memory.max`, whatever the main containers declare.
Tenant namespaces are skipped. Where the tenant chart's own
`tenant-range-limits` exists (only when the operator configured
`resourceQuotas`), a second LimitRange next to it would leave the result
to the order LimitRanger iterates them. Where it doesn't exist, a tenant
namespace is not the place for the platform to impose a default.
### Why the default is 32Gi
The handler grants immunity for `memory.max` existing at all, whatever
its value. So the ceiling doesn't need to be fitted, and 32Gi is high
enough that nothing plausibly reaches it. A ceiling low enough to bind
would turn a rare pressure-driven kill into a deterministic one for any
component whose real usage nobody measured. This repo already has one
such case, recorded in the `vertical-pod-autoscaler` values, where a
110Mi limit OOM-killed the updater and failed HelmReleases with it.
Scheduling is unaffected, since pods are scheduled against requests and
the paired default request is 32Mi. An operator who wants the ceiling to
bind can lower it, per component in that component's chart or fleet-wide
with the knob, once real usage is measured.
### When the default is withheld
A default below a container's own request makes that pod impossible to
admit, so the operator checks before it applies. It scans Deployments,
StatefulSets, DaemonSets, CronJobs, Jobs and Pods in the namespace, init
containers included, for a container that requests more than the
configured limit and declares no limit of its own. If it finds one, it
withdraws the managed LimitRange and records a hold in the
`operator.cozystack.io/system-memory-state` annotation on the Namespace.
The hold does not clear by itself when the blocker disappears, because
an empty scan looks the same as a pod between deletion and recreation.
Lowering a previously applied limit also creates a hold, even with no
visible blocker, because LimitRanger writes the defaulted limit onto
admitted pods and the scan can no longer tell which containers declare
one. A cluster administrator releases the hold by copying the
`<hold>:<target>` token from the state into the
`operator.cozystack.io/system-memory-ack` annotation. The token is used
once and only for that hold and target. If a blocker is still visible,
the token is rejected and the hold stays. Package reconciles are
scheduled at least every five minutes, so an acknowledgement takes
effect on a quiet cluster too. The reason and the token also appear in
the operator log under `withholding the default container memory limit`.
### Namespace another LimitRange already governs
If a system namespace already has a LimitRange that says anything about
memory, at the Container or the Pod scope, the operator writes nothing
there and withdraws what it wrote before. A second default leaves the
effective value to iteration order. A foreign `max` below this default,
a `min` above the default request or a `maxLimitRequestRatio` rejects
pods outright, which takes out a whole namespace. At the Pod scope the
collision can't be decided in advance, because it depends on how many
containers a future pod has. A LimitRange that bounds only cpu or
storage coexists normally. A same-named LimitRange without the
operator's managed-by label is treated as foreign and left alone.
This matters most in `kube-system`, which is in scope because
`cozystack-scheduler` installs there, and where a distribution's or an
administrator's own LimitRange is likeliest to sit.
### Disabling
Setting `cozystackOperator.systemNamespaceMemoryLimit` to `0` disables
the feature. On the next reconcile the operator removes every LimitRange
it owns cluster-wide, found by the managed-by label. While the feature
is on, it also removes its LimitRange from namespaces no active Package
targets any more. The safety state on the Namespace stays in both cases.
### Docs
`docs/operations/system-memory-limits.md` landed in #3496 ahead of this
code, so main describes a `4Gi` default and installer values that do
nothing yet. The page is corrected to the 32Gi default and now describes
withholding, the acknowledgement, and the foreign LimitRange rule.
### Not in this PR
linstor policy. #3824 extends the controller's existing exemption from
limits to the satellite, and a linstor limit here would contradict it,
so linstor stays with that PR.
The scan reads six workload kinds through `APIReader`, uncached, on
every reconcile of every system namespace. That cost is tracked in
#3977, since the system namespace set isn't known when the cache is
built and the right fix depends on a reconcile-rate measurement nobody
has taken.
### Verification
`go vet` and `go test` pass for `cmd/cozystack-operator` and
`internal/operator`, and `helm unittest` passes on the installer chart
for the knobs unset, set, and the limit at `0`. The reconciler tests
cover withholding on an oversized request, the hold surviving the
blocker's removal, the hold on a lowered limit, acknowledgement and
token rejection, every memory field of a foreign LimitRange at both
scopes, ownership by label, the disabled sweep, and a deleting Package
not creating policy.
The withhold design has not been run on a live cluster yet. An earlier
design of this PR was, and that run does not carry over.
### Screenshots
Not a UI change.
### Downstream repositories
I walked the trigger map against the diff.
`packages/core/installer/values.yaml` gains two keys, but the
`ansible-cozystack` trigger is for a required key or a rename of one the
role passes, and these are new, optional and default to `""`, so the
role is unaffected. Nothing else in the map is reached by the Go
changes, the operator template or the page.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(operator): system namespaces now get a default container memory limit through a LimitRange maintained by cozystack-operator, which keeps system components out of the Talos userspace OOM handler's victim set. Tune it with cozystackOperator.systemNamespaceMemoryLimit (default 32Gi) and cozystackOperator.systemNamespaceMemoryRequest (default 32Mi); set the limit to 0 to disable it. The default is far above any real working set on purpose, because the handler grants immunity for a memory.max existing at all, not for its value. A namespace with a container that requests more than the limit and declares no limit of its own gets no default, and stays without one until a cluster administrator acknowledges the hold recorded on the Namespace (see docs/operations/system-memory-limits.md); lowering a previously applied limit needs the same acknowledgement. A namespace where another LimitRange already sets anything about memory is left alone. A container that declares its own memory limit is never changed.
```
|
||
|
|
798094e725 |
feat(release): add backport-audit, a pre-release backport triage tool (#3527)
## What this PR does
This adds `cmd/backport-audit`, a maintainer tool that checks, before a
release or an rc is cut, whether everything labelled for backport
actually landed on the release branch. It prints the URLs of the PRs
that did not and exits non-zero, so the cut can gate on it.
`docs/release.md` and `docs/agents/releasing.md` now send the patch-cut
check through it.
The docs it replaces inventoried backports with `gh pr list --search
"is:merged label:kind/backport"`. That lists every PR ever labelled, not
what reached the branch. Answering the real question by hand means
reading the branch history per PR, and that is how backports go missing
quietly.
```console
$ go build ./cmd/backport-audit
$ ./backport-audit release-1.4
=== release-1.4 === 30 candidate PRs: MISSING=5 dropped=4 backported=21
MISSING -- labelled for this line, no trace of it here (5):
https://github.com/cozystack/cozystack/pull/2704
#2704 fix(info): use root-host for Keycloak OIDC issuer URL in tenant kubeconfig
label=kind/backport author=myasnikovdaniil merged=2026-05-26 -- no backport PR, nothing on branch
```
A `kind/backport` label does not name a branch. `backport.yaml` resolves
the target when the PR merges, so the same label means `release-1.5` on
a PR merged in June and `release-1.6` on one merged in August. The audit
reproduces that rule instead of trusting the label text. The rule
changed once, on 2026-08-03, when
|
||
|
|
5b585d6eda |
fix(operator): persist holds for unsafe memory defaults
An admitted Pod loses the distinction between explicit and defaulted limits. Its disappearance cannot prove a lower default is safe for its replacement. Keep that decision on the Namespace until an administrator acknowledges the particular target, without waiving current blockers or foreign policy ownership. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e83b4eb492 |
fix(operator): align memory policy ownership messages
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
b3168a3824 |
test(operator): pin system memory defaults
Keep the parser inputs tied to the constants registered as flag defaults, but compare the results with literal 32Gi and 32Mi expectations. A shipped default can no longer move while the defaults case stays green. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |