## What this PR does
Stacked on #5288, merge that one first.
PR touching only `cmd/backport-batch` was treated as code change, so it
rebuilt every image, installed platform in e2e sandbox and ran full
chainsaw suite plus whole unit lane. #5283 (one-line test fix) spent 3.5
hours in that e2e and hit job timeout. And backport-batch go tests were
running on every code PR, whatever it touched.
- plan job `code` output is now false when every changed path is under
`docs/`, `cmd/backport-batch/` or `cmd/api-gate/` (CI-only tools, no
image builds them). Builds, finalize and e2e are skipped and `E2E Tests`
posted green, same as for docs-only PRs. `e2e-fork.yaml` skips on same
prefixes, `hack/gate-concurrency-contract.bats` fails if two lists
diverge.
- `hack/build-matrix.sh` and `hack/select-e2e.sh` don't escalate to full
rebuild or full suite because of these two dirs, so mixed PR is scoped
by its other paths. Other `cmd/` dirs still escalate.
- `test-backport-batch` removed from `make unit-tests`. Checks job runs
it only when diff touches path from `cmd/backport-batch/test-inputs.txt`
(tool itself, in-repo packages it imports, docs pages its tests read,
`go.mod`, `go.sum`). `TestInputsListed` compares this list with `go list
-deps -test` and fails when import is missing there.
- new `docs` output keeps bats lane for docs changes and drops it for
tool-only ones.
- includes test fix from #5283 (pin run attempt in
`TestReportDedupAndTrust`), this PR replaces it.
### Screenshots
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
ci: changes only in docs/, cmd/backport-batch/ or cmd/api-gate/ don't build images and don't run e2e anymore, backport-batch tests run only when their inputs change.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* CI now skips builds, E2E tests, and full-pipeline checks for changes
limited to documentation and designated CI-only tools.
* Backport-batch tests run when relevant files change, without being
included in the general unit-test target.
* Changes to other command-line tools continue to trigger the
appropriate build and E2E checks.
* **Documentation**
* Clarified how documentation, dashboard, and backport-batch changes
affect test selection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Depends on #5219. This PR is stacked on that branch; the description
covers only what it adds on top.
Adds the per-application roots to the observability proxy, as described
in the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94):
```text
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/log/select/logsql/<endpoint>
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/metrics/api/v1/<endpoint>
```
This lets the console show the logs and resource consumption of one
application rather than of the whole namespace.
What is added:
- Plural-to-Kind mapping from an informer over `ApplicationDefinitions`.
The Kind is looked up only after the `SubjectAccessReview` (`get
<plural>/log` or `get <plural>/metrics`), so an unauthenticated caller
cannot probe which plurals exist. An unknown plural answers `404`; an
unsynced cache or a plural declared by two kinds answers `503`.
- Logs: a second `extra_stream_filters` value with the application Kind
and name, on top of the unchanged namespace filter. VictoriaLogs ANDs
the two, on every endpoint including `tail`.
- Metrics: the forced `extra_filters[]` becomes an OR of three branches,
each still bound to the namespace:
1. series carrying the application identity labels;
2. series of the application's pods, found through `kube_pod_labels`;
3. series of the application's PVCs, found through
`kube_persistentvolumeclaim_labels`.
- The pod and PVC lookups cap the number of names
(`--metrics-app-max-pods`, `--metrics-app-max-pvcs`); over the cap
answers `422`. A failed pod or PVC lookup fails the request (`502`, or
`504` on the deadline) rather than answer without the application's
series, and so does a lookup that vmselect marks `isPartial`, since a
storage node that did not answer leaves names out (a partial list
already over the cap still answers `422`); vmselect's series-limit
refusal of a lookup is passed through as `422`. A query that reads
further back than `--metrics-app-lookback` (default 720h, chart value
`observabilityProxy.metrics.appLookback`), or by an unknown amount, and
a query whose samples lie at a time the lookup cannot cover answer `422`
on the application root; this includes every query with an `@` modifier,
`WITH` with a negative offset, and offsets that are not plain durations.
The namespace root still serves them. `rollup_candlestick`, a backslash
or a `#` comment outside a string, an offset directly followed by a
minus (compound durations such as `-1h-30m` or `1h-2h`), and a subquery
step that is not a plain duration are refused as well. The lookup
reaches past the evaluation time by any negative offset and by two steps
per subquery level.
- Application routes require read access to the namespace root as well
as the application subresource; a grant narrowed with `resourceNames`
alone answers `403`.
- The proxy's ClusterRole gains `list`/`watch` on
`cozystack.io/applicationdefinitions`. No tenant RBAC change is needed:
the `view` role already allows `get` on every `apps.cozystack.io`
resource and subresource.
- `/readyz` now also waits for the `ApplicationDefinitions` cache, so a
missing CRD or RBAC rule keeps the whole proxy unready, namespace roots
included.
#### Data comes from the identity labels
The application roots find data only through identity labels added by
three PRs that are now in main:
- #5215 `feat(monitoring-agents)`: export PVC labels from
kube-state-metrics (PVC branch).
- #5216 `feat(monitoring)`: stamp application identity on scraped series
(identity branch).
- #5217 `feat(monitoring-agents)`: keep application identity in
container log streams (logs root).
Merge after #5219.
The pod branch does not depend on them, because `kube_pod_labels`
already exports all pod labels. Logs and series written before those
changes rolled out carry no identity and are reachable only through the
namespace root.
#### Security condition: a convenience filter, not an isolation boundary
The application roots narrow the namespace roots; they do not isolate
applications from each other. Each application route therefore requires
two grants: the application subresource (`get <plural>/log` or `get
<plural>/metrics`) and read access to the namespace root (`list
cozystack.io/logs` or `cozystack.io/metrics` in the namespace). A grant
narrowed with `resourceNames` alone is refused. Every tenant role that
can read applications already has both. The application scope cannot
isolate yet because:
1. The lineage webhook keeps `apps.cozystack.io/application.*` labels
that a user sets on a pod or PVC that has no application owner, so such
an object can claim another application's identity.
2. The webhook's `objectSelector` skips any object created with
`internal.cozystack.io/managed-by-cozystack`, pods included, so forged
labels on such an object are never examined.
3. The pod and PVC branches match by name over the lookup window, so an
object that reuses the name of another application's recently deleted
pod or PVC pulls that object's older series into scope.
The proposed fix (a separate change to the lineage webhook) is to strip
the identity labels when no owner resolves, and to stop skipping CREATE
requests for objects carrying the managed-by label.
#### Other known gaps
- PVCs created from a StatefulSet's `volumeClaimTemplates` usually get
no lineage labels, so their volume series are not in the application
scope.
- ingress-nginx series intentionally carry no application identity, so
they are reachable only through the namespace root.
- A request for an application that does not exist answers `200` with an
empty result rather than `404`: the proxy checks authorization, not
existence. Nothing leaks; whether a `404` is wanted is open for review.
#### How it was tested
- `go test -race` and `go vet` for the proxy packages, including a
narrowed-grant test (access to application `a` only): every application
path for `b` answers `403` and no request reaches the store. Name
escaping in the pod and PVC branches is covered for regex metacharacters
and quotes.
- helm-unittest for `observability-proxy` (new RBAC rule).
- Verified on a test cluster, with the three identity PRs above applied:
the application logs root of a Redis instance returned only that
instance's lines and none from a neighbouring pod in the same namespace;
its application metrics root returned only the Redis pods' series.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: no trigger matched. Candidate: none affected, because the
diff on top of #5219 is confined to the proxy's Go code and its own
chart RBAC; it reads `ApplicationDefinitions` but does not change their
semantics or the CRD. The security condition above may deserve a note in
the design proposals in `cozystack/community`, which is not a
trigger-map item.
### Release note
```release-note
feat(observability-proxy): add per-application logs and metrics roots under `/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/{log,metrics}/`, scoped by application identity labels. Data appears for logs and series collected after the application identity changes to Fluent Bit, VMAgent and kube-state-metrics are rolled out.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Added a tenant-aware observability proxy for logs and metrics, with
namespace and application scoping, access checks, and per-user request
limits.
* Enabled the proxy in the system package bundle and routed console log
and metrics requests through it.
* Added live log streaming and health, readiness, and metrics endpoints.
* **Bug Fixes**
* Invalid or excessive query parameters are rejected; requests with
uncertain application-metric scope are refused rather than returning
incomplete results.
* **Tests**
* Added end-to-end and integration coverage for tenant isolation,
console routing, access controls, and log streaming.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
This clears the golangci-lint findings on main and fixes the errors
several of them pointed at. With the default linter set (errcheck,
govet, ineffassign, staticcheck, unused), main had 197 findings when
this branch started. Rebased on main at 0f169540, the branch leaves 38,
all deprecations that need a design decision rather than a lint fix;
they are listed below.
Most commits are mechanical. They remove unused helpers and constants,
reword error strings, take the staticcheck simplifications, discard
best-effort cleanup errors explicitly, and move to the FieldsV1 and ptr
accessors. A loop in pkg/lineage that always ended in its first pass
becomes a helper, with a regression test for each fallback. These change
behaviour:
- Lost output is now a failure for the commands this PR touches: cozypkg
list, the validation report of validate, push and tap, the push status
line, the tap summary, and check-readiness. The command exits non-zero
and says whether the action it reports on already happened. A closed
pipe ends the process with SIGPIPE, as for other Unix tools.
- backport-batch writes its output before it pushes or publishes. Once a
write to stdout or stderr fails, the run changes nothing more: no push,
no GitHub write, no local branch move, and no new pick. A pick already
in flight still finishes its rollback. The run exits 2, and it catches
SIGPIPE, so a closed pipe is handled the same way. A confirmation
printed after an action is lost only when stderr itself fails, so a lost
confirmation can only be reported by the exit code.
- The SSE server drops a client as soon as a write to it fails. A stream
still open at shutdown no longer panics on a double close, the handler
waits for its pump before returning, and a stream that opens during
shutdown gets 503.
- The kube-ovn plunger now fails its manager when the status server
cannot bind :18080, as a metrics server bind failure does. Before, it
ran without the endpoint and logged nothing.
- A timed-out check-readiness --wait still ends with its verdict, and
the help lists the exit statuses.
The 37 findings left are SA1019 deprecations, kept out of this PR on
purpose:
- GetEventRecorderFor (12, in cmd/*/main.go). The events.k8s.io recorder
has a different interface and needs RBAC for that API group. It changes
what events look like, so it needs its own design.
- client.Apply (12, one more site arrived from main). Moving to
Client.Apply with apply configurations changes how every server-side
apply call is built and how the fake client in tests handles it. It
overlaps the dependency rollup in #4869 and should come after it.
- Result.Requeue (10). Requeue retries with rate-limited backoff,
RequeueAfter with a fixed delay. The interval has to come from whoever
owns each controller.
- scheme.Builder (3, in api/*). Replacing it changes the type of the
exported SchemeBuilder that other repositories import.
- cel.Functions (1, in pkg/reservation). It stays for its late binding,
as the comment next to it explains.
Each behaviour change has tests that failed before the fix. I ran go
test on every changed package (for cmd/backport-batch, the tests around
output and publishing, since the full package takes about ten minutes),
go build ./..., go vet on the changed packages, and a final
golangci-lint run with uncapped output on the rebased tree:
`golangci-lint run --verbose --max-same-issues=0
--max-issues-per-linter=0`, which reports the 38 findings above and
nothing else.
The tests in pkg/lineage, pkg/ovnstatus, pkg/reservation, cmd/cozypkg
and cmd/check-readiness now run in CI: a separate commit adds those
packages to go-unit-tests. Some packages with tests are still in no CI
target: cmd/cozystack-controller, cmd/cozystack-operator,
cmd/observability-proxy, pkg/apiserver, pkg/apis/apps/defaulting,
pkg/apis/apps/install, pkg/apis/apps/presets, pkg/apis/apps/validation,
pkg/apis/core/install, pkg/apis/sdn/install and pkg/version. That
predates this PR.
One known nit: when backport-batch refuses an action after its output
was lost, stderr can show two "output was lost" lines.
### Screenshots
No UI change.
### Downstream repositories
I walked the trigger map against the diff. cozypkg counts as developer
tooling, but only its error paths changed, not a flag, its usage or its
output format, so the development page needs no update. Telemetry metric
names and labels are unchanged.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
chore: cozypkg, check-readiness and backport-batch now fail when their output cannot be written, and backport-batch pushes and publishes nothing once that happens. The kube-ovn plunger fails when its status server cannot start, drops status clients whose writes fail, and no longer panics when it shuts down with a client connected.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* Commands now report output failures instead of silently succeeding.
Backport operations stop before further external changes when required
output cannot be written; readiness checks return a failure status when
output writing fails.
* Package listings, validation reports, push status messages, and tap
summaries now report output errors. Tap commands indicate when changes
were applied before a summary could be written.
* Live event streams handle failed writes and shutdown more reliably,
and status-server startup errors are reported.
* **Documentation**
* Readiness-check help now explains exit statuses and closed-pipe
behavior.
* **Improvements**
* Error messages for several backup and inventory operations now provide
clearer context.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Use tagged switches where staticcheck suggests them and drop two test
assignments whose values are never read. Behaviour is unchanged.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
backport-batch writes its body, summary, findings and warnings before
it pushes, opens or edits a pull request, closes a batch or posts a
comment. A run that kept going after that output was lost acted on
what nobody read, and a closed pipe was worse: with Go's default
SIGPIPE handling the process died silently mid-run. Lines that confirm
an action after it happened, such as the URL of an opened pull request,
go to stderr, so a lost stderr matters as much as a lost stdout.
A failed write to either stream is kept, and from then on every push and
every GitHub write is refused at the point it would be made: the GitHub
client is wrapped so that each write checks first, and each push checks
before it runs. Once output is lost, the run changes nothing more: no
push, no GitHub write, no local branch move. No new unit starts after
output is lost; a unit already in flight finishes its rollback. A reset
whose own hint, the only way back to the commits it drops, cannot be
written does not run either. The run exits 2 saying the output was lost;
a confirmation lost after an action fails the run too. Such a
confirmation is lost only when stderr itself fails, so a lost
confirmation can only be reported by the exit code. A test holds every
method of the GitHub interface to being a known read or a guarded write.
The tool catches SIGPIPE with signal.Notify, so a closed pipe is a
write error like any other. It does not use signal.Ignore, whose ignored
disposition the git and gh children would inherit across exec; a test
checks that a child still dies of SIGPIPE.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A temporary file or directory that cannot be removed, or closed after
its write already failed, leaves nothing to act on: the run's result is
the same, and the primary error is the one returned. The test that
unsets an environment variable now fails if it cannot.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every report, check and verify result goes to stdout, and every write
error there was dropped, so a run whose output never reached its
reader still exited 0. The terminal writer now keeps the first stdout
error, and the run fails with it unless it already failed. Only stdout
is tracked; a write error on stderr is still discarded.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
For the commands this change touches (cozypkg list, the validation
report of validate, push and tap, tap's summary, and check-readiness),
a write error the process observes is a failure: the command exits
non-zero and says whether the action it reports on already happened.
Progress lines printed before the decisive step stay best-effort. A
closed pipe ends the process with SIGPIPE, as for other Unix tools.
`cozypkg list` and the validation report dropped write errors, and the
list deferred its tabwriter flush, so a closed or full stdout still
reported success with nothing printed. The list now writes to the
command's writer and returns the flush error, and the report returns
its write error. When validation refuses a push or a tap, or fails a
validate run, the refusal stays the reported reason and the report's
write error is joined to it. A push whose report cannot be written
stops before flux runs and says "nothing was pushed"; a tap whose
summary cannot be written fails with "tap applied, but writing its
summary failed".
check-readiness fails --help, --version and its readiness result the
same way, in one-shot, --wait and --watch mode, with exit 1 as for a
result that is not ready, which its help now lists among the exit
statuses along with SIGPIPE for a closed pipe; the watch loop stops at
the first frame it cannot write, and a timed-out wait still ends with
its verdict.
push writes its "Pushing" line before it runs flux and stops with
"nothing was pushed" when that line cannot be written, with validation
skipped or after the report.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Remove helpers and constants nothing references, escape format
characters in test literals, and take the simplifications staticcheck
suggests where they read the same. Error strings no longer start with a
capital: they keep their meaning, with the kind or component name moved
out of first position. Callers match errors by type, not text; the one
test that matches text, for Kafka, keeps the phrase it looks for.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Stacked on #5264, that one merges first.
Clean cherry-pick doesn't mean the original landed. On release-1.6 the
batch carried a fix whose two deletions hit files the line never had, so
only its test arrived. Other originals conflicted on every run because
they change code that never reaches the line. Batch now tells these
cases apart instead of picking half a fix or retrying a conflict nobody
can resolve.
- Every file an original modifies or deletes is checked at its
counterpart on the line, renames followed on both sides. If part of the
original has no effect there (change to a file the line lacks, deletion
of a file it never had, deleted file the unit keeps), the original is
parked as partial and its release note stays out. A file that already
matches what the original makes counts as there (same blob, binary too,
or clean `git merge-file`). This runs in the picker, `fold`, `verify`
and over units already on the open batch, so such a unit fails `check
--cut` whatever its label until `run --defer` takes it out.
- Pick that differs from its original keeps its release note under
"Check these notes" with a `git range-diff` command to compare, record
and report mark it `changed`.
- Run names what an original builds on that the line lacks: the commit
on `main` that added a file the fix modifies or deletes, and blame of
removed lines whose text is still missing on the line. Generated files
and generated keys of ApplicationDefinitions are skipped. Lines a fix
only adds are not checked, a dependency that lives only there still
needs `--drop`.
- If a prerequisite never reaches the line (`feat`, direct push,
excluded or itself parked) the original is parked and not tried. If it
is still to come, the original waits for it and is retried later.
Prerequisites the line or open batch already carry, or this run picked,
count as there. `run`, `check` and record-only commands decide in the
same order. On scheduled run a workflow original whose only missing
dependency is in tests or docs waits for a maintainer instead of
parking. `backport/include` lifts neither park, wait nor partial.
- `notify` announces each park once on the open batch and marks it in
the record, nothing is announced while no batch is open.
Record moves to version 3 and report to format 2, format 1 reports still
read. I ran the new tool next to the previous one without `--push` on
today's history. On release-1.7 nothing changes category and `check`
prints the same. On release-1.6 #5071 is parked on the two features it
builds on, #5088 already on the open batch now fails the cut as a
partial unit, and #5142 on scheduled run still waits for a maintainer
with a note that only its tests depend on what 1.6 lacks. Rule 6 holds
twelve originals on 1.6 right now, run over them directly flags the
seven that change code 1.6 never got and none of the other five.
Detection adds under 2 s per line.
After this merges, CI on the open 1.6 batch goes red at its next push
since `verify` refuses #5088 there. `run --line release-1.6 --defer 5088
--reason ... --push` takes it out.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(release): backport batch holds back a pick that applies only part of its original and fails the cut on such unit already on a batch, sets apart the note of a changed pick, parks an original that builds on code its line never gets, waits for one whose prerequisite is still to come, and announces each park once.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Backport batches now identify partial picks and missing prerequisites,
and can park affected originals or leave them waiting.
* Batch reports and notifications provide details about parked, partial,
changed, and waiting picks.
* Release notes for picks that differ from their originals are set apart
for review.
* **Bug Fixes**
* Verification, folding, and batch checks reject picks that omit effects
from the original; renamed paths are supported.
* **Documentation**
* Updated backport guidance to explain parking, prerequisites, partial
picks, and release-note checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
A pull request that changed only cmd/backport-batch built every image,
installed the platform for e2e and ran the whole unit lane, so a
one-line test fix waited hours on suites that cannot exercise it. The
backport-batch tests themselves ran on every code pull request,
although only a change to the tool, to what it imports or to the pages
its tests read can break them.
Their inputs are listed beside the tool and checked against its import
graph, because a hand-kept list that fell behind would switch the
tests off without anyone noticing.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->
## What this PR does
Pins the report deduplication test to the first workflow attempt. In a
CI rerun, the inherited attempt otherwise adds `/attempts/2` to the
synthetic run URL and fails the existing exact-URL assertion. Production
run URLs and the deduplication assertions stay unchanged.
Validation: a source-derived check of the test setup and original URL
assertion fails with an inherited second attempt and passes with the
proposed first-attempt setup. Controls with the pin after preview or set
to 2 still fail. The full Git-backed test was not run locally; CI must
verify it.
### Screenshots
Not applicable; no UI changes.
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
test(release): make the report deduplication test independent of the surrounding workflow attempt.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Tests**
* Updated automated test configuration to account for GitHub Actions run
attempts. No user-facing behavior changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.
Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.
Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.
What is included:
- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.
#### How it was tested
- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.
#### Known limitations
- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.
### Release note
```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
CI reruns expose GITHUB_RUN_ATTEMPT to a test that expects a URL for
the first attempt. Keep its synthetic run identity independent of the
surrounding workflow.
Fixes: 5a997fb7d4 ("feat(release): post an immutable report of every classification and write record version 2")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A park is a decision a maintainer has to act on: port what applies as
a pull request of its own, or drop the original. Listing every park in
every notify comment would make that list noise within a week, and a
park nobody hears about is a fix that silently never ships.
notify now names each park the open batch's record holds that is not
yet marked announced, and once its comment is written marks those parks
in the record with gh pr edit --body-file. It writes only while the
pull request is open and its body still carries the record notify read,
so a run that rewrote the body in between loses nothing; a mark that
fails exits 2 and the next notify names the park once more. run carries
the mark into the next record, a merged or closed carrier's too, only
while the park is the same event: merge, kind, prerequisites, commits
and files. A park that changed is news and is named again. With no
batch open nothing is named (M2); the next batch's first notify does
it.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
On 1.6, the fix in #5071 conflicts on every run: it modifies files
that the feat in #3940 added on main, and that feat never reaches the
line. The fix in #5082 builds on #5071 and on code that never comes
either. Trying them on each run says nothing a maintainer can act on,
and a conflict reads as work to do.
Edges come from the dependencies the previous commit reads. One whose
prerequisite the line or the open batch carries, whatever its rule or
label, that this run picked or found empty, or whose evidence is
present at the branch head once earlier picks moved it, is satisfied
and dropped first. A direct push, an original rules 2-4, the start, a
skip label or a drop keep out, or one itself parked never comes and
parks the original. A prerequisite still to come (a conflict, a
deferral, a workflow wait, rule 6) holds it as deferred{needs}, the
strongest kind deciding: missing, workflow, deferred, rule 6; a chain
inherits.
run, check and the metadata operations share one order: partial unit
on the batch, a direct reason, an unpicked partial, the workflow
exception, a dependency park, a wait, then the original's own state.
In the scheduled mode a workflow original whose remaining
never-dependencies are only tests or docs waits for a maintainer
instead of parking, which is the case of the release fix in #5142.
check takes the mode the open record saved, and the scheduled one
with no batch open, where a closed record's workflow waits wait on
nothing any more and the original is judged as the scheduled run
judges it. check and the record-only operations keep a partial park
the open record holds while the batch head still shows it, since only
the run's picker sees it after the picks before it. Records refuse
waits and parks that form cycles, point at later merges or contradict
each other. notify lists a wait after what decides it, from the
record alone. backport/include lifts neither park nor wait.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
On 1.6, seven of the twelve originals rule 6 holds back build on code
the line never got: #5071 changes files #3940 added on main, #4974
reads a variable only #4386 defines. Each conflicts on every run, and
nothing tells a maintainer which missing pull request the conflict
comes from, which every human backport so far had to work out by hand.
For each original the policy takes and nothing carries, run now reads
what it builds on from git alone. A modified or deleted file the line
lacks is named by the first-parent commit on main that added it (V1).
A removed or changed line is blamed at M^1, mapped to the first-parent
commit that brought the blamed commit, and kept only while its text,
edges trimmed, is absent from the line's file (V2c), so a hand backport
under another SHA clears it. Generated files and the keys
update-crd.sh writes into an ApplicationDefinition are not blamed.
Paths go to git literally, blame gets a validated full SHA and no
--end-of-options, which git refuses there, and every call has a
two-minute deadline. Caps, deadlines, binary files, symlinks,
submodules, unmapped commits and skipped rename detection become
diagnostics shown with the original on every outcome and in the
report; malformed git output stops the run. This commit only
annotates; nothing is parked yet.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A release note describes what the original does on main. A pick git had
to fit to a line that already had part of the change, a resolution, or
a unit whose first commit came out empty no longer does the same thing,
yet its note went into the batch's release-note block with the rest,
and the changelog took it from there.
A unit now counts as changed when a surviving pick's patch-id differs
from its source or the unit's net patch-id differs from the original
merge's; the latter catches an empty first commit, which leaves no pick
to compare. A changed original's note moves under "Check these notes",
fenced and neutralised as before, with a range-diff per differing pick
and the two diffs to compare. The range-diff uses a creation factor of
999 because the default leaves a pick that lost most of its change,
#5088's test-only pick for one, unpaired. The record and the report
list such originals as changed.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A clean cherry-pick says nothing about whether the original landed
whole: git deletes a file the line never had without a conflict and
leaves a file the line lacks untouched. #5088 reached the 1.6 batch
that way, carrying only its test while the two deletions that are the
fix had no effect.
Every M and D path of an original is now judged at its counterpart on
the line, followed through renames on main, on the line, or both. A
modified path missing before and after, or left unchanged when the line
holds neither the original's blob nor a clean git merge-file result, is
omitted; a deleted path absent before is omitted. Content merge-file
refuses, mode changes and skipped rename detection become diagnostics
and never count as omissions; I/O failures stop the command.
The same guard runs in four places: the picker, which resets a partial
pick and parks the original; a check-time pass over every unit on the
open batch at its own first parent, whatever its label, which fails
check --cut and verify; the same pass over every unpicked original's
clean net merge, which parks a partial one with no record needed; and
fold. contained accepts the renamed path, so a fix of a renamed file
passes verify and fold. The unit mapping also closes the found() gap: a
merged original whose first pick came out empty is on the batch, not
missing.
The record moves to version 3 with parked entries and partial units,
and reports to format 2 with typed parked entries; format 1 reports
still read.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A fix the rules take for every line can still be wrong for one of
them, typically because it builds on code that line does not have.
The only ways to keep it out were backport/skip, which keeps it off
every line, and --defer, which holds it out of one batch: the next
batch tried it again, and a conflict that would never resolve came
back every run.
run --drop N --reason R records a drop in the line's record, bound to
the original's merge commit on main, and every later record carries
it, a new batch's included. What the open batch carries of a dropped
original comes off it the way --defer takes one off; any other drop
changes only the record, so a batch on a line that moved since stays
where it is. --undrop withdraws one. Both need an open batch, as every
write to the record does.
A drop comes after the rules and the carries: backport/skip, rules 2
to 4 or the line itself decide first and the drop stays on record,
and backport/include does not lift it. check notes a dropped original,
the body lists it, fold refuses it, and a record whose drop names
another original's merge stops every command. Each report keeps the
record's drops with the merge commit each is bound to and its reason,
held on read to the same rules, so an earlier report still says why
after an undrop.
A reason now travels from a body to terminals, bodies and comments, so
a reason or a file name in a record is believed only in the form a run
writes it, one clean line, and so is a report's diagnostic. Output
that repeats a reason, file names, a report ID or a state GitHub
reports escapes it.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
docs/release/backport-tasks/failed-ci.md says to defer the culprit with
run --defer --push. When the culprit was all the batch carried, the
push stopped with "close batch pull request #N instead", and closing
it by hand left nothing that said why: once closed, the deferred
original was simply missing.
The run now closes the batch itself, and does every write while the
pull request is open: the report, then the body, whose record keeps
the deferrals and says the batch is closing, at the head the rebuild
started from. The close and the deletion of the branch go only while
GitHub and the remote still show that head, and the deletion is
leased on it.
A close that stops part way is finished by the next run --push, before
anything else, and by no other command; a head the close did not
freeze is refused, never adopted. The run that finishes it stops
there, and fails when it was asked for --defer, since no open batch
is left to defer on. Recovery never writes the closed
pull request. A same-day batch that reuses the branch name and fails
to open its pull request leaves a branch the next run names for a
person to delete.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Each run rewrote the batch body whole, so the only account of what a
run found was gone at the next run, and a merged batch kept only its
last one. Nothing said what a classification was made of either: not
the heads it read, the policy, the tool or the flags.
A run now makes a report: every original merged into main since the
line branched, in exactly one category, with what the run read. Its
ID hashes the classification and names no pull request, so a new
batch's report is whole before GitHub numbers the pull request, and a
run that finds the same as the last one makes the same report, which
is posted once and keeps its first bytes. The body carries the latest
report under the record; a comment the tool never edits keeps each
one, so report reads any of them back, from merged batches too.
The comment goes up before the push and the body edit on an open
batch, and after the pull request a new batch opens with the report in
its body; a later run posts a report the body kept but no comment did,
byte for byte, before its own. Every write is preceded by a read of
the pull request, and nothing is written once it is merged or closed.
Both the comment and the body are rendered and measured before the
first write, so an oversized one stops the run with nothing written.
The record becomes version 2: what it was built from, with the run's
--no-workflow-files, which --defer keeps rather than resets, and the
report it goes with.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The comment said the application lookback default matches
--metrics-max-range, but nothing ties the two: changing the range
flag leaves the lookback where it is. It now gives only the reason
for the length.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The note that an application metrics request makes three store calls
sat in a source comment, where an operator reading --help never sees
it. It is now part of the --rate-limit-per-user help text.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
With the lookback refusal, the 24h default turned common application
queries into 422: a day's increase, and month-long panels at the step
a dashboard picks for them. The lookback estimate is a loose upper
bound, made for the store choice, and counts every "(" as an implicit
subquery, so it far exceeds what such queries read.
The lookback is only the refusal threshold, since the lookup itself
spans the window the query reads, so its default now matches
--metrics-max-range. The chart passes it from values, so an operator
can tune it.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The per-user rate counts client requests, but an application metrics
request makes three store calls, so an operator sizing the rate
against store capacity needs to know the multiplier.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Every query reads some time before its window, at least the implicit
rollup window, so with the lookback refusal a zero
--metrics-app-lookback would turn every application metrics query
into a 422. It is now refused at startup instead.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The pod and PVC lookup never started earlier than --metrics-app-lookback
before the window. A query that read further back, such as rate(x[7d])
with the 24h default, or whose lookback is unknown (an @ modifier, WITH,
a step-multiple window), answered 200 without the series of pods that
existed only before the lookup start.
Such queries now answer 422 on the application root, matching how a
query whose latest sample cannot be covered is handled; the namespace
root has no lookup and still serves them.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Graphs of one application need its series without the caller knowing
which pods and volumes it ran on. Exporter series carry the identity
the scrape class stamps, but cAdvisor and kubelet volume series do not,
so the scope ORs three branches: the identity, and the application's
pods and PVCs as kube-state-metrics labelled them over the window. Each
branch also carries the namespace, so the scope is never wider than the
namespace root.
The lookup reaches back by the query's own lookback, bounded by a flag,
so pods replaced inside an expression's range still count. A failed
lookup only drops its branch. A list over its cap answers 422 instead
of being truncated, which would hide series without saying so.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
A viewer of one application gets that application's lines without
writing a namespace query and filtering by hand, and a grant narrowed to
one application by resourceNames is honoured on every logs endpoint,
tail and the pickers included. The namespace filter stays and the
application's identity is ANDed next to it as a second stream filter,
so the result can only be narrower than the namespace root.
The plural is mapped to its kind only after the caller is authorized,
so an anonymous client cannot probe which applications exist, and the
proxy is not ready until the ApplicationDefinitions cache is.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The flag defaults test named a design document that is not in the
repository as the origin of the cost limits; the limits stand on their
own as flag defaults.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
An equal vmselect timeout still lost the race: the proxy's deadline
starts before the request reaches vmselect, so the proxy cut first
and answered 504 for a store within its time. The vmselect timeout
must now be strictly below a set request timeout, the flag help says
so, and tests pin the accepted side too, including no request timeout.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The timeout forwarded to vmselect could be set longer than the proxy's
own request timeout. The proxy then cut a query vmselect was still
allowed to run and answered 504 for a store that was within its time.
Such a combination now stops the proxy at startup.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The per-user concurrency cap on the metrics root had no test, so
dropping it from the handler went unnoticed. The new test holds one
request in the upstream, expects the same user's second one to be
refused and another user's to go through. Overrides of the metrics
flags were only checked for their defaults and refusals.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The store names and the short-term retention mirror metricsStorages in
the monitoring package, which an operator can rename or resize; they
are flags so the chart can follow such a change without a rebuild. The
cost limit defaults are the ones the metrics proposal settles on: 30d
range, 11000 points per series, 30s timeout.
Store names become part of a Service host name, so they are checked
as DNS labels at startup.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The flag now also bounds the body read and the token and access
reviews, and 0 lifts all of those at once; an operator raising it for
slow store queries needs to know that. The store comment named a
design document the repository does not have.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The package and serve comments described a delivery plan rather than
the code: the command named metrics it does not serve, and serve gave a
plan as the reason to refuse unknown roots before authenticating.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The default of 11000 buckets was justified by a metrics proxy that
does not exist on this branch. The number is Prometheus's cap on
points per series in a range query, which the metrics root reuses.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
An absent namespace resolves to the root store, and before the
informer's first list every namespace is absent, so a request served
in that window read a labeled namespace's logs from the wrong store.
Readiness keeps the Service from routing there, but a client that
addresses the pod directly is not stopped by it.
The resolver now fails while the cache is unsynced, which answers
503 like any other resolution error.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
vlselect widens the window to step boundaries and returns a
zero-filled bucket for each one, so a start off a boundary yields one
bucket more than the span divided by the step. Sizing the step from
the unaligned span let hits and stats_query_range return MaxBuckets+1
buckets.
The minimum step now spans MaxBuckets-1 steps, and the smallest
accepted --logs-max-buckets rises to 4 so that this step still fits
the third-of-range cap that keeps alignment inside the window.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The flag parser stops at the first argument that is not a flag, so
a stray word in the container args made every flag after it fall back
to its default without an error, the root tenant included.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
NaN fails the negative check, as every comparison does, and +Inf
refills a bucket instantly, so either value switched the per-user rate
limit off while the flag looked set. Both now stop the proxy at
startup.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
The deferrals a batch recorded were read only from the open batch pull
request. Once it merged, a conflict it had deferred showed as missing
and failed check and the cut, until some later run picked something
else and opened a new batch; a run that picked nothing opened none.
run, check, fold and notify now find the line's record in the newest
batch pull request whatever its state: one opened from a batch branch
of this repository by an author the batch trusts. A record that is in
doubt stops the command instead of letting it fall back to an older
one: a block by an author no longer trusted, a bot's batch whose block
was removed, a record that does not read whole, a closed record newer
than the open batch, or no record within the pages read.
The record is still written only to the open batch, and a merged or
closed batch is never written again, so a run that picks nothing with
no batch open keeps nothing and says so. The reader takes the version
2 record the next change writes; the writer stays at version 1.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
run --push opened or updated the batch pull request and said nothing
about which one, so the scheduled job's log and a maintainer's
terminal had no link to follow. gh pr create prints the URL; the run
now passes it on, on stderr, so stdout stays the body alone, which the
scheduled script saves as it is. An update prints the open pull
request's URL.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
fold, run --defer and the run's own stderr printed "rule 101" for an
original merged before its line started under the policy, and "rule
100" for backport/skip. Those numbers exist only inside the tool; the
policy has six rules, so a reader looking for rule 101 finds nothing.
Output now names such a rule in words, and the policy's own rules by
their number.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
check --cut returned 0 whenever nothing was missing and nothing was
only on the batch, so a workflow original waiting for a maintainer
passed the cut once the pick beside it was labelled backport/skip or
reached the line by hand. The waiting original was not even printed:
the case that held it swallowed the warning the plain check gives.
check now sorts every original into one disposition and exits once,
so a wait fails the cut on its own. A backport/skip original the open
batch still carries is named with the run --defer that takes it out.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>