Merge branch 'main' into feat/kubernetes-nodes-talos-sysctls

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
This commit is contained in:
Aleksei Sviridkin committed 2026-11-14 03:55:52 +03:00
commit eafe1adde5
18 files changed
+459 -95

No files matched your search

+1
View File
@@ -13,6 +13,7 @@
## What this PR does
<!-- Link the issues this PR addresses. For a complete fix targeting the default branch, use Closes #N (Closes owner/repository#N for another repository). Otherwise use a non-closing reference; release-branch PRs need manual issue linking and closure when appropriate. -->
### Screenshots
+16 -1
View File
@@ -36,7 +36,7 @@ This file provides structured guidance for AI coding assistants and agents worki
- **E2E tests and E2E CI** (e.g., "write/fix an e2e test", "stabilize a flaky test", "add a bats test", "change the e2e workflow", "why did e2e fail")
- Read: [`e2e-testing.md`](./docs/agents/e2e-testing.md)
- Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), no assertion beginning with `!` (errexit exempts an inverted status, so it cannot fail — write `if cmd; then echo "FAIL: …"; false; fi`), fail-fast on HelmRelease readiness
- Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), no assertion beginning with `!` (errexit exempts an inverted status, so it can be silently ignored — write `if cmd; then echo "FAIL: …"; false; fi`), fail-fast on HelmRelease readiness
- **General questions about contributing**
- Read: [`contributing.md`](./docs/agents/contributing.md)
@@ -51,10 +51,22 @@ This file provides structured guidance for AI coding assistants and agents worki
- PR titles: a Conventional Commits header (`type(scope): description`, types from [`contributing.md`](./docs/agents/contributing.md)) auto-applies `kind/*` and `area/*` via `.github/workflows/pr-labeler.yaml`. Append `!` (or add a `BREAKING CHANGE:` footer) to apply `kind/breaking-change`
**Important rules:**
- **Match scope, then read first.** Load a process file only when the task genuinely falls in its scope, and load it before starting work — not midway through. Tasks outside a file's scope should not pull it into context.
- **Follow it exactly.** Do not assume you already know the process, and do not skip steps (`changelog.md` especially). Where the documentation and your instinct disagree, the documentation wins.
- **[`overview.md`](./docs/agents/overview.md) is the standing reference** for structure and conventions, useful even when no route above points at it.
## Working Principles
These hold for every task, whichever document the routes above point at.
- **A causal claim needs a mechanism.** A failure that appeared after a commit, a bump or a rerun was not thereby caused by it, and matching symptoms prove nothing either: unrelated faults often look identical from outside. Back a root cause with a log line, a reproducer, a bisect, or a ruled-out alternative; without one, name at least two hypotheses and what would tell them apart. Before tying a problem to a change, check whether it predates the change (an older run, a condition's `lastTransitionTime`, `git blame`). Close a hypothesis when evidence rules it out. Missing evidence means inconclusive, not coincidence: record that limit, stop treating the hypothesis as a cause, and revisit it only with new evidence or a concrete discriminating check.
- **Verify, do not recall.** Versions, CLI flags, API fields and defaults drift faster than a model's training data. Check the upstream source, the documentation at the pinned version, or `--help` before writing code or advice that depends on them. When a problem is not solved after two attempts, stop and research it instead of trying a third variation.
- **Long flag names.** New and changed commands in docs, scripts, CI and PR text spell options out (`--namespace`, `--output`, `--filename`, `--message`), so that a reader does not need to know what `-f` means to this particular tool. An option with no long form is the only exception.
- **Name the cluster.** Management and tenant clusters are different targets. Every `kubectl` call, including reads, carries `--context`; cluster-facing Helm commands carry `--kube-context`. Use the human-approved target and the task's kubeconfig where one was supplied, not the ambient current context. Before running wrappers or cluster-facing tests, verify that they propagate the chosen target. Tests that need no cluster must not inherit live cluster credentials.
- **Respect the reader's time.** Lead with the answer or decision; prefer direct wording over formalities, filler and repeated conclusions. Put necessary detail after a short summary or in a linked document.
- **Public text.** Everything that reaches GitHub (commits, PR and issue bodies, review comments, code comments) is in English. It names no private infrastructure (cluster, customer or internal namespace names) and none of an agent's private tooling (custom skills, slash commands). Nothing is published on someone's behalf without their approval, unless they have delegated it.
## Project Overview
**Cozystack** is a Kubernetes-based platform for building cloud infrastructure with managed services (databases, VMs, K8s clusters), multi-tenancy, and GitOps delivery.
@@ -62,6 +74,7 @@ This file provides structured guidance for AI coding assistants and agents worki
## Quick Reference
### Code Structure
- `packages/core/` - Core platform charts (installer, platform)
- `packages/system/` - System components (CSI, CNI, operators)
- `packages/apps/` - User-facing applications in catalog
@@ -70,6 +83,7 @@ This file provides structured guidance for AI coding assistants and agents worki
- `api/` - Kubernetes CRDs
### Conventions
- **Helm Charts**: Umbrella pattern, vendored upstream charts in `charts/`
- **Go Code**: Controller-runtime patterns, kubebuilder style
- **Git Commits**: Conventional Commits (`type(scope): description`) with `--signoff`; an agent-assisted commit adds one `Assisted-by: LLM` trailer
@@ -77,6 +91,7 @@ This file provides structured guidance for AI coding assistants and agents worki
- **Prose**: One continuous line per paragraph — see [Prose Formatting](#prose-formatting)
### What NOT to Do
- ❌ Edit `/vendor/`, `zz_generated.*.go`, upstream charts directly
- ❌ Modify `go.mod`/`go.sum` manually (use `go get`)
- ❌ Force push to main/master
+6
View File
@@ -101,6 +101,7 @@ func parseFlags(args []string) (options, error) {
fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 30*24*time.Hour, "how far before the window an application metrics query may read; one reading further back, or by an unknown amount, answers 422")
fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422")
fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422")
fs.StringVar(&o.metrics.PlatformNamespacePrefix, "metrics-platform-namespace-prefix", "cozy-", "namespaces with this prefix run platform exporters, whose exported_namespace label adds series to that namespace's scope; empty disables")
fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables")
fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained client requests per second per user; an application metrics request makes three store calls (pod lookup, PVC lookup, query); 0 disables")
fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user")
@@ -121,6 +122,11 @@ func parseFlags(args []string) (options, error) {
return o, fmt.Errorf("invalid metrics store name %q", store)
}
}
// The prefix is matched literally, so one no namespace name can start
// with would turn the exporter branch off without a word.
if p := o.metrics.PlatformNamespacePrefix; p != "" && len(validation.IsDNS1123Label(p+"x")) > 0 {
return o, fmt.Errorf("invalid metrics platform namespace prefix %q", p)
}
switch {
case o.rootTenant == "":
return o, errors.New("--root-tenant must not be empty")
+16 -1
View File
@@ -46,6 +46,10 @@ func TestParseFlagsDefaults(t *testing.T) {
if m.RootTenant != o.rootTenant {
t.Errorf("metrics root tenant = %q, want --root-tenant", m.RootTenant)
}
// The prefix every platform namespace carries.
if m.PlatformNamespacePrefix != "cozy-" {
t.Errorf("platform namespace prefix = %q, want cozy-", m.PlatformNamespacePrefix)
}
}
func TestParseFlagsOverrides(t *testing.T) {
@@ -67,6 +71,7 @@ func TestParseFlagsMetricsOverrides(t *testing.T) {
"--root-tenant=tenant-x", "--metrics-shortterm-store=fast", "--metrics-longterm-store=slow",
"--metrics-shortterm-retention=24h", "--metrics-max-range=168h", "--metrics-max-points=500", "--metrics-max-timeout=10s",
"--metrics-app-lookback=2h", "--metrics-app-max-pods=7", "--metrics-app-max-pvcs=3",
"--metrics-platform-namespace-prefix=sys-",
})
if err != nil {
t.Fatal(err)
@@ -74,9 +79,15 @@ func TestParseFlagsMetricsOverrides(t *testing.T) {
m := o.metrics
if m.RootTenant != "tenant-x" || m.ShortTermStore != "fast" || m.LongTermStore != "slow" || m.ShortTermRetention != 24*time.Hour ||
m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second ||
m.App.Lookback != 2*time.Hour || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 {
m.App.Lookback != 2*time.Hour || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 ||
m.PlatformNamespacePrefix != "sys-" {
t.Errorf("overrides not applied: %+v", m)
}
// An empty prefix switches the exporter branch of the scope off.
o, err = parseFlags([]string{"--metrics-platform-namespace-prefix="})
if err != nil || o.metrics.PlatformNamespacePrefix != "" {
t.Errorf("empty platform namespace prefix: %q, %v", o.metrics.PlatformNamespacePrefix, err)
}
}
func TestParseFlagsInvalid(t *testing.T) {
@@ -112,6 +123,10 @@ func TestParseFlagsInvalid(t *testing.T) {
{"--metrics-app-lookback=0s"},
{"--metrics-app-max-pods=0"},
{"--metrics-app-max-pvcs=0"},
// No namespace name starts with these, so they would match nothing.
{"--metrics-platform-namespace-prefix=Cozy-"},
{"--metrics-platform-namespace-prefix=cozy.*"},
{"--metrics-platform-namespace-prefix=-"},
{"--unknown"},
// The proxy would cut the request while vmselect is still within
// the timeout it was given.
+46 -66
View File
@@ -42,14 +42,14 @@ After every change, verify the relevant artifacts before opening the PR:
- **Helm templates changed:** `make show` in the package directory renders cleanly.
- **Go code changed:** `make generate && make manifests` at the repo root, then commit the regenerated files.
- **`values.yaml` annotations changed:** `make generate` in the package; review the `values.schema.json` diff.
- **Migration scripts:** confirm what data actually exists first with `kubectl get <resource> -A -o jsonpath='{...}'` on a dev cluster before writing the logic.
- **Migration scripts:** when the task supplies a kubeconfig, export its path as `KUBECONFIG`. Set `KUBE_CONTEXT` to the human-approved dev-cluster context and confirm the existing data with `kubectl --context "${KUBE_CONTEXT:?}" get <resource> --all-namespaces --output=jsonpath='{...}'` before writing the logic.
## Commit Format
Follow [Conventional Commits](https://www.conventionalcommits.org/) with `--signoff`:
```bash
git commit --signoff -m "type(scope): brief description"
git commit --signoff --message "type(scope): brief description"
```
**Types:** `feat`, `fix`, `security`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`
@@ -67,12 +67,23 @@ Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREA
The subject is imperative and specific; `fix`, `wip` and `update` are not subjects. The body, when there is one, explains why the change is made and only that: the diff already shows what changed and how. Write for a reader who has only `git log`, so no review-iteration or planning vocabulary (`address review comments`, `pass 2`, `batch 1`, `as discussed`) and no bare issue or ticket number in place of the explanation.
**Examples:**
```bash
git commit --signoff -m "feat(dashboard): add config hash annotations to restart pods on config changes"
git commit --signoff -m "fix(postgres): update operator to version 1.2.3"
git commit --signoff -m "docs(contributing): add installation guide"
git commit --signoff --message "feat(dashboard): add config hash annotations to restart pods on config changes"
git commit --signoff --message "fix(postgres): update operator to version 1.2.3"
git commit --signoff --message "docs(contributing): add installation guide"
```
### One Commit, One Logical Change
A commit is the unit a backport cherry-picks and a revert undoes, so it carries exactly one logical change: one fix, one feature, one refactor, together with the regenerated files and tests that belong to it. When the content of a commit changes during review, rewrite that commit (`git commit --amend`, or a `fixup` in `git rebase --interactive`) rather than stacking a correction on top; the branch should read as if each change had been right the first time. If another commit in the series names the rewritten commit in a `Fixes:` trailer, update that reference to its new hash and subject.
### References to Commits and Issues
A commit that fixes a defect introduced by an earlier commit carries a kernel-style `Fixes:` trailer with at least 12 characters of the hash and the full subject, on one unbroken line, which `git log --max-count=1 --abbrev=12 --format='Fixes: %h ("%s")' <sha>` prints. `Fixes: #N` is not used, so that the `Fixes:` trailer always points at a commit.
Use `Closes: #N` only when the change fully resolves the issue. For partial work, use a non-closing reference such as `Refs: #N`; a closing commit trailer can close the issue when it reaches the default branch even if the PR body uses a non-closing link. For an issue in another repository, qualify the reference as `owner/repository#N`, including in a closing trailer (`Closes: owner/repository#N`). Reference trailers go after the body and before `Assisted-by:` and `Signed-off-by:`.
### AI Agent Attribution
When an AI agent authors or materially assists with a commit, add exactly one trailer line next to the `Signed-off-by:` trailer that `--signoff` produces:
@@ -104,13 +115,13 @@ Editing the title does not re-run it. The `edited` trigger was dropped in #3211
**Type → `kind/*`:**
| type | label |
| --------- | ------------------ |
| feat | kind/feature |
| fix | kind/bug |
| docs | kind/documentation |
| chore | kind/cleanup |
| refactor | kind/cleanup |
| type | label |
| --- | --- |
| feat | kind/feature |
| fix | kind/bug |
| docs | kind/documentation |
| chore | kind/cleanup |
| refactor | kind/cleanup |
| security, style, perf, test, build, ci, revert | (no kind label) |
**Scope → `area/*`** (full mapping in `.github/workflows/pr-labeler.yaml`):
@@ -150,13 +161,17 @@ If the branch has extra commits, clean it up:
git fetch upstream
git checkout -b my-feature upstream/main
git cherry-pick <your-commit-hash>
git push -f origin my-feature
git push --force origin my-feature
```
## Pull Request Body
Fill in the template at [`.github/PULL_REQUEST_TEMPLATE.md`](../../.github/PULL_REQUEST_TEMPLATE.md). It includes the required `release-note` block and the [Downstream Repositories](#downstream-repositories) checklist.
Link the issues the PR addresses in its body. Use `Closes #N` when it fully resolves an issue, or `Closes owner/repository#N` for another repository; link partial work without a closing keyword. A closing keyword in a commit can close the issue on merge into the default branch, but does not create the PR's linked-issue entry. Put the reference in the PR body even when the commit already carries it.
GitHub interprets closing keywords in a PR description only when it targets the default branch. For a release-branch PR, use a non-closing reference and have a maintainer with write access link the issue manually and close it separately when appropriate. See [GitHub's issue-linking rules](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue).
Create the PR with `gh pr create --title "type(scope): brief description" --body-file <file>`.
`--body` and `--body-file` replace the body wholesale, so the template is not applied and its checklists are silently dropped. Start the body file from the template instead:
@@ -167,6 +182,14 @@ cp .github/PULL_REQUEST_TEMPLATE.md /tmp/pr-body.md
gh pr create --draft --title "type(scope): brief description" --body-file /tmp/pr-body.md
```
### What Goes in the Commit, What Goes in the PR Body
The commit message is the permanent record: a reader of `git log` or `git blame` years later has it and nothing else. It carries what broke or was missing and for whom, the exact error string verbatim when there is one, so that `git log --grep` finds it, why this design and not the obvious alternative, why it is safe where the diff does not show that, and one line on how it was tested, including the failure path when the change adds one.
The PR body is read once, next to those commits, so it does not repeat them. Its first screen says what is broken or missing, where, and what that costs. After that come one sentence per commit and only what no single commit can carry: results that need the whole series, the environment they came from, decisions the reviewer has to make across commits. Questions to reviewers and open doubts belong in the PR body and never in a commit, where they go stale the day it merges. A fact that fits both goes in the commit.
Size a commit body against accepted commits touching the same files, not against a fixed line cap, which cuts the answers before the narrative. After shortening any of these texts, check that every must-keep fact survived, that every number still sits next to the version or setup it was measured on, that nothing new was added, and that a claim about a set ("every caller", "never happens") is backed by a grep or rewritten as a mechanism.
## Downstream Repositories
Cozystack is upstream for repositories that are **not** kept in sync with it automatically. A change here can break them silently, because nothing in CI compares the two sides.
@@ -292,60 +315,17 @@ The release process keeps a **separate** list of repositories, in [`changelog.md
- A bug-fix PR should include a behavioural regression test, not just a single field assertion — prove the bug can no longer recur.
- Keep the PR description aligned with the template: `## What this PR does` plus the `release-note` block.
- Before posting a claim about a set ("all callers", "the only user", "never happens"), grep for the counterexample, or state the mechanism instead.
- Disagreement carries a mechanism: a code path, a number, a spec clause, a failing test. Time spent and urgency are not arguments.
- Answer the point directly and explain why it is accepted or declined. Read brief feedback literally rather than inferring tone, and avoid unnecessary notifications. Follow the [prose guidance](../../AGENTS.md#working-principles).
## Fetching Unresolved Review Comments
## Collecting Review Feedback
Cozystack uses GitHub review threads with resolution status. Only unresolved threads are actionable — resolved threads are already handled.
Read the feedback from these sources, including existing replies:
The REST endpoint `/pulls/{pr}/reviews` returns review summaries, not individual review comments. Use the GraphQL API to access `reviewThreads` with `isResolved` status:
- Unresolved inline review threads.
- Review submission bodies, even when the review also has inline comments.
- General PR conversation comments.
- Comments posted directly on the PR's commits.
```bash
gh api graphql -F owner=cozystack -F repo=cozystack -F pr=<PR_NUMBER> -f query='
query($owner: String!, $repo: String!, $pr: Int!) {
repository(owner: $owner, name: $repo) {
pullRequest(number: $pr) {
reviewThreads(first: 100) {
nodes {
isResolved
comments(first: 100) {
nodes {
id
path
line
author { login }
bodyText
url
createdAt
}
}
}
}
}
}
}' --jq '.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false) | .comments.nodes[]'
```
Compact one-line variant:
```bash
gh api graphql -F owner=cozystack -F repo=cozystack -F pr=<PR_NUMBER> -f query='
query($owner: String!, $repo: String!, $pr: Int!) {
repository(owner: $owner, name: $repo) {
pullRequest(number: $pr) {
reviewThreads(first: 100) {
nodes {
isResolved
comments(first: 100) {
nodes {
path
line
author { login }
bodyText
}
}
}
}
}
}
}' --jq '.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false) | .comments.nodes[] | "\(.path):\(.line // "N/A") - \(.author.login): \(.bodyText[:150])"'
```
Verify each human or automated finding against the current code before deciding whether it needs a change. Do not post a duplicate answer to a point already addressed.
+5 -5
View File
@@ -81,8 +81,7 @@ Trap-based cleanup was the single biggest source of false failures in the BATS s
The install test fails if **any** HelmRelease is not Ready. A toothless gate (backgrounded `kubectl wait` discarding child exit codes + a bare `echo` instead of `exit 1`) once shipped a permanently-failing platform HR through green CI for weeks.
- Use a single `kubectl wait hr --all -A`, then an outcome-based re-list (covers HRs created after the snapshot), and `exit 1` on any non-Ready HR.
- Dump the full Ready-condition message per non-Ready HR so the real error is in the test output. See `hack/e2e-install-cozystack.bats`.
Use `cozy_wait_all_helmreleases_ready` from `hack/e2e-wait-helmreleases.sh`, as `hack/e2e-install-cozystack.bats` does, and preserve its failure status and diagnostics.
### 6. Assert no HelmRelease failed an action it retries in place — via `status.history`
@@ -160,6 +159,7 @@ A fork's `pull_request` run carries no registry credentials, so it cannot push t
Consequences to keep in mind when touching either workflow:
- **A `workflow_run` workflow always runs the default branch's copy of itself.** An edit to `e2e-fork.yaml` therefore has no effect on any fork PR, the one carrying the edit included, until it is merged to `main`. To see it work, merge first and then push to a fork PR: re-running an existing `e2e-fork` run replays the commit and workflow file it was first triggered with.
- **Fork builds export, they do not push (`OCI_EXPORT_DIR`).** The fork's unprivileged build sets `OCI_EXPORT_DIR` (see `hack/common-envs.mk`) so every image is written to a per-image OCI archive artifact instead of being pushed; the privileged `e2e-fork.yaml` later pushes those archives to the registry **by digest**. Any build recipe that shells out to push directly — e.g. a `skopeo copy … docker://…` that bypasses the `image-tags`/`oci-output` macros — must honour `OCI_EXPORT_DIR` too, or a fork build dies with an anonymous-push `denied` (#3257). This is covered by `hack/common-envs_test.bats`.
- **"E2E Tests" is a commit status — not a job, and not a check-run.** The required merge-gate context named `E2E Tests` is a GitHub **commit status**, posted by the `plan`/`e2e-report` pair for same-repo PRs and by `e2e-fork.yaml`'s `resolve`/`report` pair for forks — **not** the `e2e` job itself. Each lane opens the context with `pending` and concludes it when the suite is done, and the opener is not optional: a commit status never expires, so a head SHA that already went green would otherwise keep reading green from the moment a re-run, a reopen or a `full-e2e` label starts a new run until that run reports, leaving the pull request mergeable across a suite that may be about to fail. While the gate was a job, a fresh non-success check run appeared automatically and closed that window. The two lanes close it at different speeds, and the fork lane does not close it entirely. Same-repo, the opener is `plan`'s first step, so the window lasts seconds. On a fork the opener is `resolve`, which cannot run until the unprivileged build has finished, so re-running an already-green fork head leaves the old green readable for the length of that build. The gap is structural rather than an oversight: a fork's `pull_request` token is read-only and cannot write a status at all, which is the same property the rest of this design relies on. It is bounded by the build and it is the one case where "the opener closes the window" is not literally true. Two reasons it is a status:
- *Not a job:* a required *job* that is skipped (as the `e2e` job is for forks) counts as passed by branch protection — exactly how fork PRs used to merge with e2e never run (#3257). A status is `success` only when the report step posts it; a never-posted required status stays "Expected" and blocks the merge. If you rename the `e2e` job, do **not** name it `E2E Tests` and do not wire branch protection to a job name — the status must stay the gate.
@@ -181,9 +181,9 @@ Consequences to keep in mind when touching either workflow:
### 11. Never negate a BATS assertion with `!`
An assertion spelled `! cmd` inside an `@test` body cannot fail, whichever runner executes it. POSIX exempts a command whose return value is inverted with `!` from `set -e`, so the failing status is dropped where it is produced. `hack/cozytest.sh` then appends `return 0` to every test function, so it never resurfaces as the test's own status. The `bats` binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and `bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap, and exits zero. The one shape `bats` does catch is a negated assertion standing **last** in the body, where the inverted status is what the function returns — put any statement below it and `bats` passes the test too. This is the same family as the ban on test-level `EXIT`/`RETURN` traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer.
An assertion spelled `! cmd` inside an `@test` body can silently pass. POSIX exempts a command whose return value is inverted with `!` from `set -e`, so a failing inverted status does not stop the test body. `hack/cozytest.sh` then appends `return 0` to every test function, so it never resurfaces as the test's own status. The `bats` binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and `bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap, and exits zero. The one shape `bats` does catch is a negated assertion standing **last** in the body, where the inverted status is what the function returns — put any statement below it and `bats` passes the test too. This is the same family as the ban on test-level `EXIT`/`RETURN` traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer.
The exemption is also wider than the "negated pipeline" phrasing found in older comments: a bare `! true`, a negated `[ … ]`, one inside a `for` body, and one following `;`, `&&` or `||` all pass. Write the absence so that it fails — `if cmd; then echo "FAIL: <what must not have happened>"; false; fi` — and let the `false` carry the verdict. `hack/bats-no-negated-assert.bats` enforces this across every `hack/**/*.bats`, subdirectories included. Inverted **conditions** (`if !`, `elif !`, `while !`, `until !`) are untouched, because `set -e` is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal, failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate `printf` argument.
The exemption applies to any inverted command, not just pipelines: `! true`, negated `[ … ]`, negation inside a `for` body, and negation following `;`, `&&` or `||` do not trigger errexit. A later successful statement can hide their failure. Write the absence so that it fails — `if cmd; then echo "FAIL: <what must not have happened>"; false; fi` — and let the `false` carry the verdict. `hack/bats-no-negated-assert.bats` enforces this across every `hack/**/*.bats`, subdirectories included. Inverted **conditions** (`if !`, `elif !`, `while !`, `until !`) are untouched, because `set -e` is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal, failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate `printf` argument.
## Chainsaw v0.2.15 gotchas
@@ -209,7 +209,7 @@ The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026);
7. Failure path attaches scoped diagnostics via a `catch:` block, never a silent pass. The node-join deadline is no exception, and no longer has one: with the soft-red marker gone it fails its suite like any other deadline and its diagnostics run from the same `catch:`.
8. If it touches HelmRelease install/upgrade behavior, add the `status.history` guard: fail the run on a `failed` or `uninstalled` Snapshot of a release whose install strategy is `RetryOnFailure`, fail it on a parent whose strategy is anything else, and report the footprint of every other release without failing. What is reusable today is the predicate and the jsonpath in `hack/e2e-chainsaw/_lib/remediation-guard.sh`; the policy that weighs the footprint against `.spec.install.strategy.name`, judges an empty history against the release's own readiness, and keeps each read's exit status apart from its output lives in `cozy_guard_helmrelease` in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` — read it before rebuilding any of that for another suite. If the suite has a failure path that exits before the guard, give that path `cozy_report_helmrelease_remediation` as well, or the runs the guard exists for are the runs it does not see.
9. A test that creates a tenant `Kubernetes` cluster waits for its release to reach `Ready` **before the assertions it ends on**, with a `--timeout` the guard can read as positive and no longer than a day. Reads earlier in the run are expected and are not what this bounds. The mechanism, how to size the budget and what the guard does with it are under "Do not mask cleanup or teardown failures". One thing to carry from there rather than look up: the naming is part of the rule, not a detail of the check, so an end-assertion added under any name outside the `cozy_assert_*` / `cozy_switch_and_assert_*` convention is invisible to it and passes before the wait without reddening anything — as long as one recognised call site is left behind it. Rename every recognised call and the guard reds instead, on finding none at all, which is the one shape of this mistake it does catch.
10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and the assertion cannot fail; use `if cmd; then echo "FAIL: …"; false; fi`.
10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and can be silently ignored; use `if cmd; then echo "FAIL: …"; false; fi`.
## The container lane: what the srv nodes are, and what that costs
+39 -2
View File
@@ -16,7 +16,7 @@ The platform exposes infrastructure services via the Kubernetes API with ready-m
## Code Layout
```
```text
.
├── packages/ # Main directory for cozystack packages
│ ├── core/ # Core platform logic charts (installer, platform)
@@ -50,7 +50,7 @@ The platform exposes infrastructure services via the Kubernetes API with ready-m
Every package is a Helm chart following the umbrella chart pattern:
```
```text
packages/<category>/<package-name>/
├── Chart.yaml # Chart definition and parameter docs
├── Makefile # Development workflow targets
@@ -110,13 +110,35 @@ A chart also declares the limits on its own resource name there, with `## @name
- **E2E tests:** Kyverno Chainsaw suites in `hack/e2e-chainsaw/` (one directory per app), run with `chainsaw test`. Cluster bootstrap (`hack/e2e-install-cozystack.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in [`e2e-testing.md`](./e2e-testing.md).
- **Go tests:** standard `go test`, with Ginkgo/Gomega for controllers.
### Test first, failure path first
These rules apply to changes with testable behavior. Documentation-only edits and unmodified upstream updates need the relevant checks, not an artificial failing test. First-party adaptations made during a dependency bump remain covered by these rules.
1. Write the test before changing the implementation and watch it fail for the intended reason, not on a typo or a missing fixture. Then make it green, then refactor under it.
2. Cover relevant failure paths before the happy path: empty or null values, invalid input, a number where a string is expected, a missing resource, a timeout or an API error. Assert the resulting message, exit status and post-failure state required by the operation's contract.
3. Fix regressions in the implementation. Do not skip, weaken or delete a test, or silence a linter, merely to get a green run. When the intended contract changes, update or replace the obsolete assertions and retain coverage of applicable failure cases.
4. A stub needs a failing test for the intended behavior during development. Finish that behavior and make the test pass before merge; do not make the test accept an unintended not-implemented result just to get green CI.
5. A bug found along the way that is out of the PR's scope gets an issue and a test pinning the current broken behaviour, marked `FIXME(#N)`. The PR that fixes the bug flips the test to pin the fixed behaviour.
Do not negate a BATS assertion with `!`: it can silently pass. Bats catches a failing negation only when it is the last command in the test body; `hack/cozytest.sh` appends a successful return and misses it there too. See [§11 of `e2e-testing.md`](./e2e-testing.md#11-never-negate-a-bats-assertion-with-).
### Linters
Fix every linter finding before pushing, the cosmetic ones included. When a rule genuinely conflicts with the project, change the linter configuration and state the reason there, instead of suppressing the finding inline.
Markdown follows these markdownlint rules: every code fence names a language (`text` when nothing else fits), table separators have spaces around the pipes (`| --- |`), and there is a blank line before and after every list and after every heading. Line length (MD013) is the exception: prose stays one line per paragraph, as [Prose Formatting](../../AGENTS.md#prose-formatting) explains.
## Conventions
### Helm Charts
- Follow **umbrella chart** pattern for system components
- Include upstream charts in `charts/` directory (vendored, not referenced)
- Override configuration in root `values.yaml`
- Use `values.schema.json` for input validation and dashboard UI rendering
- An unquoted numeric value (`name: 123`) reaches a template as a number: `len` on it aborts the render with `len of type float64` (`int64` for a value given with `--set`) before any `fail` message is reached, and `if not $v` treats `0` as unset. For fields validated as text that permit numeric inputs, reject missing or null values with `kindIs "invalid"`, then coerce with `printf "%v"` and check emptiness with `eq $v ""`. Coercing null first produces the non-empty string `<nil>`.
- For integer fields, keep string inputs on their lexical validation path. Validate numeric inputs for type, integrality and bounds before conversion, then format the validated integer in decimal. Generic `printf "%v"` can emit exponent notation for a float (`4200000000` becomes `4.2e+09`), so it is not integer normalization.
- For nested values, validate or normalize each parent map before reading its children; checking only the leaf cannot prevent a nil-pointer error from a missing or null parent.
**No `lookup` that copies another object into the render.** A `lookup` reads cluster state at render time, and nothing re-renders the HelmRelease when that state changes, so whatever it copied into the output (a bucket name, an endpoint, a CA, a list of StorageClasses) stays stale until something unrelated upgrades the release. Lookups of this kind already in the charts have caused failures, and several have since been moved to values, so they are not precedent: do not add one of this kind unless a maintainer has authorized that specific one.
@@ -124,7 +146,18 @@ A `lookup` is fine when a repeated render gives the same output and carries noth
To use a fact about the cluster instead, read it from values: `_cluster` and `_namespace` in the `cozystack-values` Secret, or `valuesFrom` on a specific Secret, whose content change is a values change and so upgrades the release. Ask whether an API is served with `.Capabilities.APIVersions.Has`, since a `lookup` on an unserved group aborts the whole render. Otherwise check at run time in a hook or Job, or let a controller own the object.
### Container images
For a new container image built from source maintained in this repository, name the build definition `Containerfile` and pass it to `docker buildx build` with `--file` in the package `Makefile`. Keep existing `Dockerfile`s and vendored build definitions unchanged.
### Examples and fixtures
Illustrative addresses and names in docs, examples and test fixtures come from the reserved documentation ranges, never from a routable value that merely looks realistic: IPv4 from RFC 5737 (`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`), IPv6 from RFC 3849 (`2001:db8::/32`), domains from RFC 2606 and RFC 6761 (`example.com`, `.example`, `.test`, `.invalid`), ASNs from RFC 5398 (`64496`–`64511`, `65536`–`65551`), MAC addresses from RFC 7042 (`00:00:5E:00:53:00`–`00:00:5E:00:53:FF`). Where no reserved range exists, use an obviously synthetic placeholder such as `<NODE_IP>`.
When a test depends on a protocol-defined address's behavior, use that value, such as `127.0.0.1` or `localhost` for a local test server. Keep network tests isolated; this exception does not permit arbitrary live endpoints.
### Go Code
- Follow standard **Go conventions** and idioms
- Use **controller-runtime** patterns for Kubernetes controllers
- Prefer **kubebuilder** for API definitions and controllers
@@ -172,6 +205,7 @@ Before working on these areas, read the relevant doc first:
## Things Agents Should Not Do
### Never Edit These
- Do not modify files in `/vendor/` (Go dependencies)
- Do not edit generated files: `zz_generated.*.go`
- Do not change `go.mod`/`go.sum` manually (use `go get`)
@@ -179,13 +213,16 @@ Before working on these areas, read the relevant doc first:
- Do not modify image digests in `values.yaml` (generated by build)
### Version Control
- Do not commit built artifacts from `_out`
- Do not commit test artifacts or temporary files
### Git Operations
- Do not force push to main/master
- Do not update git config
- Do not perform destructive operations without explicit request
### Core Components
- Do not modify `packages/core/platform/` without understanding migration impact
+37 -8
View File
@@ -22,6 +22,7 @@ import (
"net/http"
"net/url"
"regexp"
"slices"
"strconv"
"strings"
"time"
@@ -47,6 +48,9 @@ type MetricsConfig struct {
ShortTermRetention time.Duration
Limits MetricsLimits
App AppScopeLimits
// PlatformNamespacePrefix names the namespaces whose exporters may
// attribute a series to another namespace; empty trusts none.
PlatformNamespacePrefix string
// UpstreamURL maps a store name to its Prometheus API base URL, without
// a trailing slash; nil means the in-cluster vmselect Service.
UpstreamURL func(store string) string
@@ -103,15 +107,37 @@ func namespaceSelector(ns string) string {
return "{namespace=" + strconv.Quote(ns) + "}"
}
// namespaceScope selects the series of namespace ns. Besides its own, it
// takes the series an exporter in a platform namespace attributes to ns:
// vmagent sets namespace to the scraped target's and, without honorLabels,
// moves the exporter's own namespace label to exported_namespace. Any pod
// can write that label, so it is trusted only where the platform's vmagent
// set namespace to one with the platform prefix, in which tenants cannot run
// pods or scrape objects. Series remote-written from a tenant's nested
// cluster carry whatever namespace that cluster chose, so they are outside
// this guarantee for both values alike.
func (c MetricsConfig) namespaceScope(ns string) []string {
scope := []string{namespaceSelector(ns)}
if c.PlatformNamespacePrefix != "" {
// MetricsQL anchors =~ at both ends.
platform := strconv.Quote(regexp.QuoteMeta(c.PlatformNamespacePrefix) + ".*")
scope = append(scope, "{namespace=~"+platform+",exported_namespace="+strconv.Quote(ns)+"}")
}
return scope
}
// BuildMetricsParams rebuilds the upstream parameters from an allowlist and
// adds the namespace scope as the only extra_filters[] value. vmselect ORs
// repeated extra_filters, so a client value would widen the scope; it ANDs
// the single one into every selector of the query and every match[], which
// is what lets the query pass through unparsed.
func BuildMetricsParams(endpoint string, in url.Values, namespace string, now time.Time, lim MetricsLimits) (metricsRequest, *HTTPError) {
// sends scope as the only extra_filters[] values. vmselect ORs repeated
// extra_filters, so a client value would widen the scope; it ANDs each into
// every selector of the query and every match[], which is what lets the
// query pass through unparsed.
func BuildMetricsParams(endpoint string, in url.Values, scope []string, now time.Time, lim MetricsLimits) (metricsRequest, *HTTPError) {
req := metricsRequest{path: endpoint, params: url.Values{}}
if len(scope) == 0 {
return req, errorf(http.StatusInternalServerError, "no metrics scope")
}
out := req.params
out.Set("extra_filters[]", namespaceSelector(namespace))
out["extra_filters[]"] = slices.Clone(scope)
timeout := lim.MaxTimeout
if v := in.Get("timeout"); v != "" {
d, herr := parseMetricsDuration("timeout", v)
@@ -226,8 +252,11 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti
}
// vmselect started with -search.ignoreExtraFiltersAtLabelsAPI drops
// extra_filters[] on the label APIs unless a match[] is present
// (getCommonParamsInternal), so the scope is repeated here.
out.Set("match[]", namespaceSelector(namespace))
// (getCommonParamsInternal), so the scope is repeated here. Where both
// are sent, vmselect ANDs every match[] with every extra_filters[];
// a cross pair is narrower than one of its halves, so the union is
// still the scope.
out["match[]"] = slices.Clone(scope)
req.scopedMatch = true
}
return req, nil
@@ -19,6 +19,7 @@ package observabilityproxy
import (
"net/http"
"net/url"
"slices"
"strconv"
"strings"
"testing"
@@ -29,13 +30,20 @@ func testMetricsLimits() MetricsLimits {
return MetricsLimits{MaxRange: 30 * 24 * time.Hour, MaxPoints: 11000, MaxTimeout: 30 * time.Second}
}
const wantMetricsScope = `{namespace="tenant-a"}`
// wantMetricsScope is the namespace root's scope under the default
// platform prefix: the namespace's own series, then the series a platform
// exporter attributes to it.
var wantMetricsScope = []string{`{namespace="tenant-a"}`, `{namespace=~"cozy-.*",exported_namespace="tenant-a"}`}
func testScope() []string {
return MetricsConfig{PlatformNamespacePrefix: "cozy-"}.namespaceScope("tenant-a")
}
func unix(t time.Time) string { return strconv.FormatInt(t.Unix(), 10) }
func buildMetrics(t *testing.T, endpoint string, in url.Values) metricsRequest {
t.Helper()
req, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits())
req, err := BuildMetricsParams(endpoint, in, testScope(), testNow, testMetricsLimits())
if err != nil {
t.Fatalf("BuildMetricsParams(%s, %v): %v", endpoint, in, err)
}
@@ -43,7 +51,7 @@ func buildMetrics(t *testing.T, endpoint string, in url.Values) metricsRequest {
}
func metricsStatus(endpoint string, in url.Values) int {
_, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits())
_, err := BuildMetricsParams(endpoint, in, testScope(), testNow, testMetricsLimits())
if err == nil {
return http.StatusOK
}
@@ -84,8 +92,8 @@ func TestBuildMetricsParamsAllowlist(t *testing.T) {
t.Errorf("parameter %q forwarded: %v", k, req.params)
}
}
if got := req.params["extra_filters[]"]; len(got) != 1 || got[0] != wantMetricsScope {
t.Errorf("extra_filters[] = %q, want exactly [%s]", got, wantMetricsScope)
if got := req.params["extra_filters[]"]; !slices.Equal(got, wantMetricsScope) {
t.Errorf("extra_filters[] = %q, want exactly %q", got, wantMetricsScope)
}
for _, m := range req.params["match[]"] {
if strings.Contains(m, "tenant-b") {
@@ -116,8 +124,8 @@ func TestNamespaceSelectorEscapes(t *testing.T) {
func TestBuildMetricsParamsLabelEndpointsAlwaysMatch(t *testing.T) {
for _, endpoint := range []string{"api/v1/labels", "api/v1/label/__name__/values"} {
req := buildMetrics(t, endpoint, url.Values{})
if got := req.params["match[]"]; len(got) != 1 || got[0] != wantMetricsScope {
t.Errorf("%s without match[] sends %q, want the namespace selector", endpoint, got)
if got := req.params["match[]"]; !slices.Equal(got, wantMetricsScope) {
t.Errorf("%s without match[] sends %q, want the namespace scope %q", endpoint, got, wantMetricsScope)
}
req = buildMetrics(t, endpoint, url.Values{"match[]": {"up", `{job="x"}`}})
if got := req.params["match[]"]; len(got) != 2 || got[0] != "up" || got[1] != `{job="x"}` {
@@ -198,7 +206,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) {
})
want := url.Values{
"query": {"up"}, "start": {"2026-11-04T11:00:00.250Z"}, "end": {"2026-11-04T12:00:00.000Z"}, "step": {"90"},
"timeout": {"30"}, "extra_filters[]": {wantMetricsScope},
"timeout": {"30"}, "extra_filters[]": wantMetricsScope,
}
if req.params.Encode() != want.Encode() {
t.Errorf("params = %v\nwant %v", req.params, want)
@@ -0,0 +1,237 @@
/*
Copyright 2026 The Cozystack Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package observabilityproxy
import (
"net/http"
"net/url"
"regexp"
"slices"
"strconv"
"strings"
"testing"
"time"
)
func TestNamespaceScope(t *testing.T) {
tests := []struct {
prefix string
want []string
}{
{"cozy-", []string{`{namespace="tenant-a"}`, `{namespace=~"cozy-.*",exported_namespace="tenant-a"}`}},
{"", []string{`{namespace="tenant-a"}`}},
// A prefix is literal: a dot in it must not match any character.
{"sys.", []string{`{namespace="tenant-a"}`, `{namespace=~"sys\\..*",exported_namespace="tenant-a"}`}},
}
for _, tt := range tests {
if got := (MetricsConfig{PlatformNamespacePrefix: tt.prefix}).namespaceScope("tenant-a"); !slices.Equal(got, tt.want) {
t.Errorf("prefix %q: scope = %q, want %q", tt.prefix, got, tt.want)
}
}
}
func TestNamespaceScopeEscapesExportedNamespace(t *testing.T) {
for _, ns := range []string{`a"} or {namespace=~".+`, `back\slash`, "new\nline"} {
scope := MetricsConfig{PlatformNamespacePrefix: "cozy-"}.namespaceScope(ns)
filters, ok := parseSelector(scope[1])
if !ok || len(filters) != 2 || filters[1].label != "exported_namespace" || filters[1].value != ns {
t.Errorf("scope %q does not decode to exported_namespace=%q: %+v", scope[1], ns, filters)
}
}
}
// TestBuildMetricsParamsScopeOnEveryEndpoint checks that every endpoint of
// the namespace root sends the whole scope and nothing a client adds.
func TestBuildMetricsParamsScopeOnEveryEndpoint(t *testing.T) {
endpoints := map[string]url.Values{
"api/v1/query": {"query": {"up"}},
"api/v1/query_range": {"query": {"up"}, "start": {unix(testNow.Add(-5 * time.Minute))}, "end": {unix(testNow)}, "step": {"60"}},
"api/v1/series": {"match[]": {"up"}},
"api/v1/labels": {},
"api/v1/label/pod/values": {},
}
for _, prefix := range []string{"cozy-", ""} {
scope := MetricsConfig{PlatformNamespacePrefix: prefix}.namespaceScope("tenant-a")
for endpoint, in := range endpoints {
in = cloneValues(in)
in["extra_filters[]"] = []string{`{namespace=~".+"}`}
req, herr := BuildMetricsParams(endpoint, in, scope, testNow, testMetricsLimits())
if herr != nil {
t.Fatalf("%s: %v", endpoint, herr)
}
if got := req.params["extra_filters[]"]; !slices.Equal(got, scope) {
t.Errorf("prefix %q, %s: extra_filters[] = %q, want %q", prefix, endpoint, got, scope)
}
}
}
}
// An empty scope would send no extra_filters[] and read every namespace.
func TestBuildMetricsParamsRefusesEmptyScope(t *testing.T) {
if _, herr := BuildMetricsParams("api/v1/query", url.Values{"query": {"up"}}, nil, testNow, testMetricsLimits()); herr == nil || herr.Code != http.StatusInternalServerError {
t.Errorf("empty scope: %v, want 500", herr)
}
}
// TestBuildMetricsParamsCopiesScope guards against the forwarded parameters
// sharing the caller's slice.
func TestBuildMetricsParamsCopiesScope(t *testing.T) {
scope := testScope()
req, herr := BuildMetricsParams("api/v1/labels", url.Values{}, scope, testNow, testMetricsLimits())
if herr != nil {
t.Fatal(herr)
}
scope[1] = `{namespace=~".+"}`
if !slices.Equal(req.params["extra_filters[]"], wantMetricsScope) || !slices.Equal(req.params["match[]"], wantMetricsScope) {
t.Errorf("params follow the caller's slice: %v", req.params)
}
}
// TestVmselectSeesScopedSeries runs the parameters the proxy builds
// through a model of vmselect v1.136's series selection: repeated
// extra_filters[] are ORed (searchutil.GetExtraTagFilters), and every
// match[] or query selector is ANDed with each of them
// (searchutil.JoinTagFilterss, called from getCommonParamsInternal and
// promql eval). Regex filters match the whole value.
func TestVmselectSeesScopedSeries(t *testing.T) {
series := []struct {
labels map[string]string
visible bool
}{
{map[string]string{"namespace": "tenant-a"}, true},
{map[string]string{"namespace": "cozy-clickhouse-operator", "exported_namespace": "tenant-a", "chi": "clickhouse-db"}, true},
// A tenant pod claiming another namespace is not a platform exporter.
{map[string]string{"namespace": "tenant-y", "exported_namespace": "tenant-a"}, false},
{map[string]string{"namespace": "xcozy-a", "exported_namespace": "tenant-a"}, false},
{map[string]string{"namespace": "cozy-clickhouse-operator", "exported_namespace": "tenant-b"}, false},
{map[string]string{"namespace": "cozy-clickhouse-operator"}, false},
{map[string]string{"namespace": "tenant-b"}, false},
}
for _, endpoint := range []string{"api/v1/series", "api/v1/labels"} {
in := url.Values{}
if endpoint == "api/v1/series" {
in.Set("match[]", `{__name__=~".*"}`)
}
req := buildMetrics(t, endpoint, in)
for _, s := range series {
if got := vmselectSelects(t, req.params, s.labels); got != s.visible {
t.Errorf("%s: series %v visible = %v, want %v", endpoint, s.labels, got, s.visible)
}
}
}
// Without a prefix the exporter's series stay out of the namespace.
req, herr := BuildMetricsParams("api/v1/labels", url.Values{}, MetricsConfig{}.namespaceScope("tenant-a"), testNow, testMetricsLimits())
if herr != nil {
t.Fatal(herr)
}
if vmselectSelects(t, req.params, map[string]string{"namespace": "cozy-clickhouse-operator", "exported_namespace": "tenant-a"}) {
t.Error("exporter series visible with the platform prefix disabled")
}
}
func cloneValues(in url.Values) url.Values {
out := url.Values{}
for k, v := range in {
out[k] = slices.Clone(v)
}
return out
}
type labelFilter struct {
label, value string
regex bool
}
func (f labelFilter) matches(labels map[string]string) bool {
v := labels[f.label]
if f.regex {
return regexp.MustCompile("^(?:" + f.value + ")$").MatchString(v)
}
return v == f.value
}
// parseSelector reads the selectors the proxy and these tests write:
// {name="value",name=~"regex",...} with values quoted as MetricsQL reads
// them (strconv.Unquote in metricsql's extractStringValue).
func parseSelector(s string) ([]labelFilter, bool) {
body, ok := strings.CutPrefix(s, "{")
if !ok {
return nil, false
}
var out []labelFilter
for body != "}" {
i := strings.IndexAny(body, "=!")
if i <= 0 {
return nil, false
}
f := labelFilter{label: body[:i]}
body = body[i:]
switch {
case strings.HasPrefix(body, "=~"):
f.regex, body = true, body[2:]
case strings.HasPrefix(body, "="):
body = body[1:]
default:
return nil, false
}
lit, err := strconv.QuotedPrefix(body)
if err != nil {
return nil, false
}
if f.value, err = strconv.Unquote(lit); err != nil {
return nil, false
}
out = append(out, f)
body = strings.TrimPrefix(body[len(lit):], ",")
}
return out, true
}
func vmselectSelects(t *testing.T, params url.Values, labels map[string]string) bool {
t.Helper()
parse := func(sels []string) [][]labelFilter {
var out [][]labelFilter
for _, s := range sels {
f, ok := parseSelector(s)
if !ok {
t.Fatalf("cannot parse selector %q", s)
}
out = append(out, f)
}
return out
}
matches, extra := parse(params["match[]"]), parse(params["extra_filters[]"])
if len(matches) == 0 {
matches = [][]labelFilter{nil}
}
for _, m := range matches {
for _, e := range extra {
all := true
for _, f := range append(slices.Clone(m), e...) {
if f.label == "__name__" {
continue
}
all = all && f.matches(labels)
}
if all {
return true
}
}
}
return false
}
+1 -1
View File
@@ -297,7 +297,7 @@ func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Rout
return herr
}
now := s.now()
req, herr := BuildMetricsParams(route.Endpoint, r.Form, route.Namespace, now, s.cfg.Metrics.Limits)
req, herr := BuildMetricsParams(route.Endpoint, r.Form, s.cfg.Metrics.namespaceScope(route.Namespace), now, s.cfg.Metrics.Limits)
if herr != nil {
return herr
}
@@ -177,7 +177,9 @@ func TestServerApplicationMetricsScope(t *testing.T) {
if got := r.form["match[]"]; len(got) != 1 || got[0] != lookups[r.path] {
t.Errorf("%s match[] = %q", r.path, got)
}
if got := r.form["extra_filters[]"]; !slices.Equal(got, []string{wantMetricsScope}) {
// lookupNames filters by the namespace selector alone, not by the
// namespace root's scope.
if got := r.form["extra_filters[]"]; !slices.Equal(got, []string{`{namespace="tenant-a"}`}) {
t.Errorf("%s extra_filters[] = %q", r.path, got)
}
// The query's own lookback is known and shorter than the 24h cap.
@@ -20,6 +20,7 @@ import (
"io"
"net/http"
"net/url"
"slices"
"sync"
"testing"
"time"
@@ -75,7 +76,7 @@ func TestServerMetricsRebuildsUpstreamRequest(t *testing.T) {
if r.path != "/api/v1/query_range" {
t.Errorf("upstream path = %s", r.path)
}
if got := r.form["extra_filters[]"]; len(got) != 1 || got[0] != wantMetricsScope {
if got := r.form["extra_filters[]"]; !slices.Equal(got, wantMetricsScope) {
t.Errorf("extra_filters[] = %q", got)
}
if r.form.Has("extra_filters") || r.form.Has("extra_label") || r.form.Has("nocache") {
@@ -105,7 +106,7 @@ func TestServerMetricsStoreByRange(t *testing.T) {
if want := []string{"shortterm", "longterm", "shortterm"}; len(h.stores) != 3 || h.stores[0] != want[0] || h.stores[1] != want[1] || h.stores[2] != want[2] {
t.Fatalf("stores = %v, want %v", h.stores, want)
}
if r := h.got[2]; r.path != "/api/v1/label/pod/values" || r.form.Get("match[]") != wantMetricsScope {
if r := h.got[2]; r.path != "/api/v1/label/pod/values" || !slices.Equal(r.form["match[]"], wantMetricsScope) {
t.Fatalf("label values upstream = %s %v", r.path, r.form)
}
}
@@ -93,6 +93,7 @@ func newPolicyHarness(t *testing.T, cfg Config, handler http.HandlerFunc, allow
cfg.Metrics.UpstreamURL = cfg.UpstreamURL
if cfg.Metrics.ShortTermStore == "" {
cfg.Metrics.ShortTermStore, cfg.Metrics.LongTermStore, cfg.Metrics.ShortTermRetention = "shortterm", "longterm", 72*time.Hour
cfg.Metrics.PlatformNamespacePrefix = "cozy-"
}
if cfg.Metrics.Limits == (MetricsLimits{}) {
cfg.Metrics.Limits = testMetricsLimits()
@@ -46,6 +46,7 @@ spec:
- --metrics-longterm-store={{ .Values.observabilityProxy.metrics.longtermStore }}
- --metrics-shortterm-retention={{ .Values.observabilityProxy.metrics.shorttermRetention }}
- --metrics-app-lookback={{ .Values.observabilityProxy.metrics.appLookback }}
- --metrics-platform-namespace-prefix={{ .Values.observabilityProxy.metrics.platformNamespacePrefix }}
ports:
- name: http
containerPort: 8080
@@ -63,6 +63,9 @@ tests:
- contains:
path: spec.template.spec.containers[0].args
content: --metrics-app-lookback=720h
- contains:
path: spec.template.spec.containers[0].args
content: --metrics-platform-namespace-prefix=cozy-
- it: follows renamed metrics stores
set:
@@ -85,6 +88,19 @@ tests:
path: spec.template.spec.containers[0].args
content: --metrics-app-lookback=48h
# An empty value must reach the binary as an explicit empty flag; a
# dropped flag would fall back to the cozy- default.
- it: passes an empty platform namespace prefix through
set:
observabilityProxy.metrics.platformNamespacePrefix: ""
asserts:
- contains:
path: spec.template.spec.containers[0].args
content: --metrics-platform-namespace-prefix=
- notContains:
path: spec.template.spec.containers[0].args
content: --metrics-platform-namespace-prefix=cozy-
- it: falls back to cozy.local when the platform passes no cluster domain
asserts:
- contains:
@@ -17,6 +17,10 @@ observabilityProxy:
# How far before its window an application metrics query may read;
# one reading further back answers 422 on the application root.
appLookback: 720h
# Exporters in namespaces with this prefix may attribute a series to a
# tenant namespace through exported_namespace, and the series is then
# part of that namespace's metrics. Empty disables it.
platformNamespacePrefix: cozy-
resources:
requests:
cpu: 10m
@@ -49,6 +49,17 @@ vertical-pod-autoscaler:
app.kubernetes.io/name: vertical-pod-autoscaler
app.kubernetes.io/instance: "{{ .Release.Name }}"
app.kubernetes.io/component: admission-controller
extraArgs:
# The admission controller registers its mutating webhook for Pod CREATE in every
# namespace with failurePolicy Ignore and, by default, a 30s timeout. Ignore only
# applies after the timeout: when its pod is down and the Service IP blackholes (e.g. a
# cold start after a full cluster shutdown, before the CNI is back), the API server
# still waits the full 30s. The Pod CREATE webhooks are called one after another within
# one request, so their waits add up; past the request's deadline the creation fails
# with "context deadline exceeded", including for the CNI's own DaemonSet pods (#3125).
# A mutation takes milliseconds; 5s keeps slow responses working and keeps the total
# wait well inside the deadline.
webhook-timeout-seconds: 5
resources:
limits:
memory: 256Mi