test(tests): select resources in absence-only Helm checks (#5174)

## What this PR does

Select the intended resource by kind or name in 23 absence-only Helm
tests: the 21 cases catalogued in #4665, a newer MariaDB backup-script
case, and Kafka's JVM-options case. The requested
kubernetes-nodes/nodegroup and OpenSearch TLS audit found no remaining
unanchored negative-only cases.

Add a diff-based CI guard against new positional selectors in
first-party suites on templates with multiple literal resource blocks.
It preserves single-document uses and unchanged indices, including pure
suite renames; renamed suites are still checked for added index lines.
It follows Helm's template and index precedence, and its fixtures use
portable file writes. The guard runs after the test steps so a guard
failure cannot suppress their results.

The guard is intentionally static: it counts `---`-separated blocks with
a top-level `kind`, not resources produced by helpers or ranges, and
does not evaluate whether whole-document branches are mutually
exclusive. It does not audit vendored suites or existing indices when a
template later gains documents.

Validation:
- `make helm-unit-tests` passes: 106 package test targets, 556 Helm
suite executions, and 4,067 Helm tests, including the dashboard
package’s additional Go checks.
- All 51 focused BATS checks pass. The full BATS sweep initially passed
2,435 of 2,500 cases; six graph-selection failures were then resolved by
using the repository-pinned yq v4.53.3 (127/127 targeted rerun cases
pass). The remaining 56 Docker-dependent and three envsubst-dependent
cases are blocked by missing host prerequisites; no full BATS pass is
claimed. The six yq-related failures also reproduce on the unchanged
base with the older tool and pass with the pinned version.
- All 17 guard regression tests pass, including first-party scope,
inherited index precedence, pure/edited renames, template overrides,
nested default templates, conditional kinds, and multiline index values.
- Missing-target mutations cover all 23 selector changes: original cases
pass against unrelated resources; updated cases fail with `document not
found`.
- Reordered violating-target mutations cover all 22 formerly positional
cases: originals pass against leading unrelated resources; updated cases
find the intended target and fail their content assertion. Kafka's
updated case also passes when the reordered Job remains valid.
- Empty-render mutation confirms that the newer MariaDB backup-script
case previously passed without a document and now fails. Existing
positional cases already fail on an entirely empty render.
- The changed workflow passes the repository-pinned zizmor v1.26.1
offline audit; `git diff --check` is clean.

Closes #4665

### Screenshots

Not applicable (tests and CI only).

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Chores**
* Pull request checks now flag positional document selections in Helm
tests when they target multi-document templates.
* **Bug Fixes**
* Updated chart tests to select Kubernetes resources by kind or name
rather than document position, keeping assertions focused on the
intended resources.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Aleksei Sviridkin authored and GitHub committed 2026-11-15 05:59:52 +03:00
commit c1740e7631
11 files changed
+499 -24

No files matched your search

+5
View File
@@ -405,6 +405,11 @@ jobs:
- name: Run backport-batch tests
if: ${{ !cancelled() && needs.plan.outputs.backport_batch == 'true' }}
run: make test-backport-batch
- name: Check new Helm document indices
if: ${{ !cancelled() }}
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
run: python3 hack/check-helm-document-indices.py "$(git merge-base HEAD "origin/$BASE_REF")"
- name: Upload unit test report
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+334
View File
@@ -0,0 +1,334 @@
#!/usr/bin/env bats
load test_helper
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
setup() {
strict_setup
tmp=$(mktemp -d)
cd "$tmp"
git init -q
mkdir -p packages/apps/example/templates packages/apps/example/tests
cat > packages/apps/example/templates/multi.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
name: first
---
apiVersion: v1
kind: Secret
metadata:
name: second
EOF
head -n 4 packages/apps/example/templates/multi.yaml > packages/apps/example/templates/single.yaml
git add .
git -c user.name=Test -c user.email=test@example.org commit -qm fixture
base=$(git rev-parse HEAD)
}
teardown() {
cd "$REPO_ROOT"
rm -rf "$tmp"
}
write_suite() {
cat > packages/apps/example/tests/resource_test.yaml <<EOF
suite: resource selection
templates:
- templates/$1.yaml
tests:
- it: checks a resource
documentIndex: 0
asserts:
- notExists:
path: spec.missing
EOF
git add .
}
@test "rejects a new test-level index on a multi-document template" {
write_suite multi
if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then
echo "accepted a positional selector on a multi-document template" >&2
false
fi
grep -q 'resource_test.yaml:6: use documentSelector' output
}
@test "allows an index on a single-document template and ignores comments" {
write_suite single
printf '\n# documentIndex: 1\n' >> packages/apps/example/tests/resource_test.yaml
python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base"
}
@test "leaves existing indices alone when another assertion changes" {
write_suite multi
git -c user.name=Test -c user.email=test@example.org commit -qm existing
base=$(git rev-parse HEAD)
sed 's/spec.missing/spec.other/' packages/apps/example/tests/resource_test.yaml > changed.yaml
mv changed.yaml packages/apps/example/tests/resource_test.yaml
python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base"
}
@test "follows assertion template overrides in later YAML suites" {
write_suite single
cat >> packages/apps/example/tests/resource_test.yaml <<'EOF'
---
suite: overridden template
templates:
- templates/single.yaml
tests:
- it: checks another resource
asserts:
- notExists:
path: spec.missing
template: templates/multi.yaml
documentIndex: 1
EOF
if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then
echo "accepted an assertion-level positional selector" >&2
false
fi
grep -q 'use documentSelector.*templates/multi.yaml' output
}
@test "rejects an invalid base rather than passing an empty diff" {
if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" missing-base > output 2>&1; then
echo "accepted an invalid base" >&2
false
fi
}
@test "test-level indices follow each assertion's template override" {
write_suite multi
cat > packages/apps/example/tests/resource_test.yaml <<'EOF'
suite: assertion override
templates: [templates/single.yaml, templates/multi.yaml]
tests:
- it: checks only the single-document template
template: templates/multi.yaml
documentIndex: 0
asserts:
- template: templates/single.yaml
notExists:
path: spec.missing
EOF
python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base"
}
@test "conditional kinds inside one document are not multiple documents" {
write_suite single
cat > packages/apps/example/templates/single.yaml <<'EOF'
apiVersion: v1
{{ if .Values.secret }}
kind: Secret
{{ else }}
kind: ConfigMap
{{ end }}
metadata:
name: single
EOF
python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base"
}
@test "rejects a positional selector whose value is on a separate line" {
write_suite multi
cat > packages/apps/example/tests/resource_test.yaml <<'EOF'
suite: multiline index
templates: [templates/multi.yaml]
tests:
- it: checks a resource
documentIndex:
0
asserts:
- notExists:
path: spec.missing
EOF
if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then
echo "accepted a multiline positional selector" >&2
false
fi
grep -q 'resource_test.yaml:5: use documentSelector' output
}
@test "rejects a singular test template overriding a single-document suite" {
write_suite single
cat > packages/apps/example/tests/resource_test.yaml <<'EOF'
suite: test template override
templates: [templates/single.yaml]
tests:
- it: checks a resource
template: templates/multi.yaml
documentIndex: 0
asserts:
- notExists:
path: spec.missing
EOF
if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then
echo "accepted a positional selector on the test template override" >&2
false
fi
grep -q 'use documentSelector.*templates/multi.yaml' output
}
@test "allows a singular test template selecting one document" {
write_suite multi
cat > packages/apps/example/tests/resource_test.yaml <<'EOF'
suite: test template override
templates: [templates/multi.yaml]
tests:
- it: checks a resource
template: templates/single.yaml
documentIndex: 0
asserts:
- notExists:
path: spec.missing
EOF
python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base"
}
@test "plural test templates take precedence over the singular test template" {
write_suite multi
cat > packages/apps/example/tests/resource_test.yaml <<'EOF'
suite: test template override
templates: [templates/multi.yaml]
tests:
- it: checks a resource
templates: [templates/single.yaml]
template: templates/multi.yaml
documentIndex: 0
asserts:
- notExists:
path: spec.missing
EOF
python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base"
}
@test "an empty plural test override falls back to the singular template" {
write_suite single
cat > packages/apps/example/tests/resource_test.yaml <<'EOF'
suite: test template override
templates: [templates/single.yaml]
tests:
- it: checks a resource
templates: []
template: templates/multi.yaml
documentIndex: 0
asserts:
- notExists:
path: spec.missing
EOF
if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then
echo "accepted a positional selector after an empty plural override" >&2
false
fi
grep -q 'use documentSelector.*templates/multi.yaml' output
}
@test "does not check vendored chart suites" {
mkdir -p packages/apps/example/charts/sub/templates packages/apps/example/charts/sub/tests
cp packages/apps/example/templates/multi.yaml packages/apps/example/charts/sub/templates/
cat > packages/apps/example/charts/sub/tests/resource_test.yaml <<'EOF'
suite: upstream resource selection
templates: [templates/multi.yaml]
tests:
- it: checks an upstream resource
documentIndex: 0
asserts:
- notExists:
path: spec.missing
EOF
git add .
python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base"
}
@test "a new test-level index overrides existing assertion indices" {
cat > packages/apps/example/tests/resource_test.yaml <<'EOF'
suite: index inheritance
templates: [templates/multi.yaml]
tests:
- it: checks a resource
asserts:
- notExists:
path: spec.missing
documentIndex: 0
- notExists:
path: spec.other
documentIndex: 0
EOF
git add .
git -c user.name=Test -c user.email=test@example.org commit -qm existing
base=$(git rev-parse HEAD)
cat > packages/apps/example/tests/resource_test.yaml <<'EOF'
suite: index inheritance
templates: [templates/multi.yaml]
tests:
- it: checks a resource
documentIndex: 1
asserts:
- notExists:
path: spec.missing
documentIndex: 0
- notExists:
path: spec.other
documentIndex: 0
EOF
if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then
echo "accepted a new inherited positional selector" >&2
false
fi
grep -q 'resource_test.yaml:5: use documentSelector' output
}
@test "allows a pure suite rename with existing indices" {
write_suite multi
git -c user.name=Test -c user.email=test@example.org commit -qm existing
base=$(git rev-parse HEAD)
git mv packages/apps/example/tests/resource_test.yaml packages/apps/example/tests/renamed_test.yaml
python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base"
}
@test "checks added index lines in a renamed suite" {
write_suite multi
git -c user.name=Test -c user.email=test@example.org commit -qm existing
base=$(git rev-parse HEAD)
git mv packages/apps/example/tests/resource_test.yaml packages/apps/example/tests/renamed_test.yaml
cat > packages/apps/example/tests/renamed_test.yaml <<'EOF'
suite: resource selection
templates:
- templates/multi.yaml
tests:
- it: checks a resource
documentIndex: 1
asserts:
- notExists:
path: spec.missing
EOF
if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then
echo "accepted a new positional selector in a renamed suite" >&2
false
fi
grep -q 'renamed_test.yaml:6: use documentSelector' output
}
@test "an empty suite template list includes nested chart templates" {
mkdir packages/apps/example/templates/nested
mv packages/apps/example/templates/multi.yaml packages/apps/example/templates/nested/
cat > packages/apps/example/tests/resource_test.yaml <<'EOF'
suite: default templates
templates: []
tests:
- it: checks a resource
documentIndex: 0
asserts:
- notExists:
path: spec.missing
EOF
git add .
if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then
echo "accepted a positional selector with default templates" >&2
false
fi
grep -q 'use documentSelector.*templates/nested/multi.yaml' output
}
+87
View File
@@ -0,0 +1,87 @@
#!/usr/bin/env python3
"""Reject added positional selectors on multi-resource Helm templates.
Usage: python3 hack/check-helm-document-indices.py <base-ref>
Requires mikefarah/yq, like the chart test tooling.
"""
import json
from pathlib import Path
import re
import subprocess
import sys
def git(*args):
return subprocess.check_output(["git", *args], text=True)
def check(base):
"""Check added indices, leaving existing ones alone.
This static check counts --- separated blocks with a top-level kind; it does
not evaluate conditional documents or resources emitted by helpers/ranges.
"""
root = Path(git("rev-parse", "--show-toplevel").strip())
additions = {}
path = None
for line in git("diff", "--no-ext-diff", "-M", "--unified=0", base, "--",
":(glob)packages/*/*/tests/*_test.yaml").splitlines():
if line.startswith("diff --git "):
path = None
elif line.startswith("+++ b/"):
path = line[6:]
additions[path] = set()
elif line.startswith("@@"):
number = int(re.search(r"\+(\d+)", line).group(1))
elif path and line.startswith("+"):
if "documentIndex" in line:
additions[path].add(number)
number += 1
elif line.startswith(" "):
number += 1
errors = []
for path, added in additions.items():
if not added:
continue
# Preserve the key's source line while yq resolves multi-document YAML,
# flow mappings and anchors; a text-only suite parser misses those.
expression = ('[. | (.. | select(tag == "!!map" and has("documentIndex")) '
'| .documentIndex) |= {"line": key | line}]')
suites = json.loads(subprocess.check_output(
["yq", "eval-all", "-o=json", expression, str(root / path)], text=True))
chart = (root / path).parent.parent
for suite in suites:
if not suite:
continue
for test in suite.get("tests", []):
templates = test.get("templates") or (
[test["template"]] if test.get("template")
else suite.get("templates") or ["templates/**/*"])
for assertion in test.get("asserts", []):
index = test.get("documentIndex") or assertion.get("documentIndex", {})
number = index.get("line")
if number not in added:
continue
selected = [assertion["template"]] if "template" in assertion else templates
for pattern in selected:
matches = list(chart.glob(pattern))
if not matches:
matches = list((chart / "templates").glob(pattern))
for template in matches:
if not template.is_file():
continue
documents = re.split(r"(?m)^---[ \t]*(?:#.*)?$", template.read_text())
if sum(bool(re.search(r"(?m)^kind\s*:", doc)) for doc in documents) > 1:
errors.append(f"{path}:{number}: use documentSelector, not "
f"documentIndex, for {template.relative_to(root)}")
for error in sorted(set(errors)):
print(error, file=sys.stderr)
return bool(errors)
if __name__ == "__main__":
if len(sys.argv) != 2:
sys.exit("usage: python3 hack/check-helm-document-indices.py <base-ref>")
sys.exit(check(sys.argv[1]))
@@ -137,11 +137,13 @@ tests:
release:
name: test-kafka
namespace: tenant-test
documentSelector:
path: kind
value: Role
asserts:
- notContains:
path: rules[2].resourceNames
content: test-kafka-zookeeper
documentIndex: 0
# External bootstrap service absent when external=false, regardless of tls.enabled
- it: does not grant access to external bootstrap service when tls.enabled=true and external=false
@@ -27,7 +27,9 @@ tests:
# kafka-run-class.sh then drops its G1 settings.
- it: leaves the node pool JVM options to Strimzi and Kafka
template: templates/migration-hook.yaml
documentIndex: 3
documentSelector:
path: kind
value: Job
asserts:
- notMatchRegex:
path: spec.template.spec.containers[0].args[0]
@@ -241,18 +241,22 @@ tests:
errorMessage: 'controlPlane.apiServer.extraArgs contains "--secure-port=6444" — the tenant control plane sets --secure-port itself and drops every entry naming it before the kube-apiserver sees the list, so this entry has no effect, and written without =value it leaves its value behind as an argument the apiserver refuses. Remove the entry; the description of controlPlane.apiServer.extraArgs lists every flag the control plane owns.'
- it: leaves spec.admissionControllers out by default, so the control plane keeps its own default set
documentSelector:
path: kind
value: KamajiControlPlane
asserts:
- notExists:
path: spec.admissionControllers
documentIndex: 1
- it: leaves spec.admissionControllers out for an explicitly empty list
set:
controlPlane.apiServer.admissionControllers: []
documentSelector:
path: kind
value: KamajiControlPlane
asserts:
- notExists:
path: spec.admissionControllers
documentIndex: 1
- it: renders controlPlane.apiServer.admissionControllers into spec.admissionControllers verbatim
set:
+21 -6
View File
@@ -530,7 +530,9 @@ tests:
- templates/dashboard-resourcemap.yaml
set:
external: false
documentIndex: 0
documentSelector:
path: kind
value: Role
asserts:
- notContains:
path: rules[1].resourceNames
@@ -544,7 +546,9 @@ tests:
- templates/dashboard-resourcemap.yaml
set:
external: true
documentIndex: 0
documentSelector:
path: kind
value: Role
asserts:
- notContains:
path: rules[1].resourceNames
@@ -560,7 +564,9 @@ tests:
external: true
tls:
issuer: operator
documentIndex: 0
documentSelector:
path: kind
value: Role
asserts:
- notContains:
path: rules[1].resourceNames
@@ -576,7 +582,9 @@ tests:
external: false
tls:
issuer: cert-manager
documentIndex: 0
documentSelector:
path: kind
value: Role
asserts:
- notContains:
path: rules[1].resourceNames
@@ -695,7 +703,9 @@ tests:
set:
tls:
issuer: cert-manager
documentIndex: 3
documentSelector:
path: metadata.name
value: my-mariadb-tls
asserts:
- notContains:
path: spec.dnsNames
@@ -1187,7 +1197,9 @@ tests:
- it: "dashboard Role never grants the CA bundle even when TLS is unmanaged"
templates:
- templates/dashboard-resourcemap.yaml
documentIndex: 0
documentSelector:
path: kind
value: Role
set:
tls:
issuer: operator
@@ -1436,6 +1448,9 @@ tests:
enabled: true
s3Bucket: s3.example.org/test
cleanupStrategy: "--keep-last=3"
documentSelector:
path: kind
value: Secret
asserts:
- notMatchRegex:
path: stringData["backup.sh"]
+24 -8
View File
@@ -53,7 +53,9 @@ tests:
value: mg-test-ssl
- it: spec.tls has no issuerConf (the operator-managed secret is passed directly)
documentIndex: 0
documentSelector:
path: kind
value: PerconaServerMongoDB
asserts:
- notExists:
path: spec.tls.issuerConf
@@ -127,13 +129,17 @@ tests:
# test would pass even with the external check removed.
_namespace:
host: example.com
documentIndex: 0
documentSelector:
path: kind
value: PerconaServerMongoDB
asserts:
- notExists:
path: spec.replsets[0].splitHorizons
- it: chart does not set allowInvalidCertificates, leaving the operator default
documentIndex: 0
documentSelector:
path: kind
value: PerconaServerMongoDB
asserts:
# The README states the permissive default is inherited rather than
# chosen here. Pin that, so the claim stays falsifiable.
@@ -141,7 +147,9 @@ tests:
path: spec.tls.allowInvalidCertificates
- it: chart does not set certValidityDuration, leaving the operator default
documentIndex: 0
documentSelector:
path: kind
value: PerconaServerMongoDB
asserts:
# Same class as allowInvalidCertificates above: the leaf lifetime is the
# operator's to choose, not this chart's. Deliberately left unpinned.
@@ -177,7 +185,9 @@ tests:
# the CRD and notExists on it can never fail.
- it: no TLS-disabling escape hatches with a default replica count
documentIndex: 0
documentSelector:
path: kind
value: PerconaServerMongoDB
asserts:
- notExists:
path: spec.allowUnsafeConfigurations
@@ -235,7 +245,9 @@ tests:
- name: rs0
replicas: 3
size: 10Gi
documentIndex: 0
documentSelector:
path: kind
value: PerconaServerMongoDB
asserts:
# splitHorizons is replica-set scoped and rendered only for the
# non-sharded topology. In sharded mode external access goes through the
@@ -255,7 +267,9 @@ tests:
replicas: 0
_namespace:
host: example.com
documentIndex: 0
documentSelector:
path: kind
value: PerconaServerMongoDB
asserts:
# With replicas: 0 the range body renders nothing. Emitting the key with
# a null value would be rejected by the API server: the CRD types
@@ -269,7 +283,9 @@ tests:
external: true
_namespace:
host: ""
documentIndex: 0
documentSelector:
path: kind
value: PerconaServerMongoDB
asserts:
# The controller may create _namespace before populating host, so the
# bootstrap window presents host as an empty string rather than as an
+6 -2
View File
@@ -557,7 +557,9 @@ tests:
external: false
tls:
enabled: true
documentIndex: 3
documentSelector:
path: metadata.name
value: my-redis-tls
asserts:
- notContains:
path: spec.dnsNames
@@ -840,7 +842,9 @@ tests:
external: false
tls:
enabled: true
documentIndex: 3
documentSelector:
path: metadata.name
value: my-redis-tls
asserts:
- notContains:
path: spec.dnsNames
@@ -228,10 +228,14 @@ tests:
gateway-class-name: " cloudflare-tunnel "
gateway-edge-classes: cloudflare-tunnel
asserts:
- documentIndex: 0
- documentSelector:
path: metadata.name
value: letsencrypt-prod
notExists:
path: spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].sectionName
- documentIndex: 1
- documentSelector:
path: metadata.name
value: letsencrypt-stage
notExists:
path: spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].sectionName
@@ -156,9 +156,11 @@ tests:
resources:
requests:
storage: 10Gi
documentSelector:
path: kind
value: Job
asserts:
- documentIndex: 0
notMatchRegex:
- notMatchRegex:
path: spec.template.spec.containers[0].args[0]
pattern: kubectl patch pvc