The shell translator rewrites test bodies and loses Bats reporting.
The unit lane needs complete TAP results and JUnit reports without
changing how live-cluster suites run.
Sourced production helpers still need a dash pass, and unit test bodies
need nounset without exporting shell options into subprocesses.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The exit trap stopped the port-forward before running mc rm, so the
delete went to a closed port and failed every time, leaving the object
behind and printing a warning that said nothing about why. The delete
now runs first and the warning carries mc's own error, or says the
delete timed out when mc was killed before it could write one.
A PUT that timed out is now also treated as uploaded: it can land and
still outlive its time bound, and the trap never tried to remove such an
object. A PUT the server refused is not, so the common case of access
that never went live does not end in a delete doomed to the same refusal.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
mc alias set reaches the endpoint through the port-forward, so a tunnel
that came up and then broke is a likely cause of that failure, and its
log is the only record of it. The readiness and retry-timeout exits
already print it; this one threw it away.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Record each topic's ReplicationFactor in the backup manifest, off the
same --describe line the partition count already comes from, and
recreate the topic with it on restore. The value used to be a
BackupClass parameter defaulting to 1, and the Cozystack Kafka chart
sets the broker-level min.insync.replicas to 2 as soon as kafka.replicas
reaches 2: kafka-console-producer writes with acks=all, the broker
rejects every write into a topic with fewer replicas than that, and the
producer logs the rejection and exits 0. A restore onto any multi-broker
cluster therefore landed zero records and failed the post-replay count
with a message naming the wrong cause; in place, the documented delete
had already taken the live topic with it. replicationFactor stays as an
override for restoring into a smaller cluster, and a value the target
cannot host now fails at --create in Kafka's own words.
Hold the pre-replay emptiness check to the topic's partition count, as
the backup side already does. kafka-get-offsets skips a partition it
could not read and still exits 0, so a leader election in flight dropped
the populated partitions from the listing, the surviving empty ones
summed to zero, and the replay appended a full copy into a live topic -
the exact outcome the check was moved ahead of the replay to prevent.
Both restore-side counts now go through one helper that refuses a short
listing. The data-file-versus-manifest check runs both ways before
anything is created, and the replay walks the manifest topic by topic,
announcing each one, so a retried Pod's log says which topics landed.
The demo now runs two brokers with two-replica topics, which is the
shape the chart puts min.insync.replicas=2 on; the restore steps assert
the restored topic's replication factor, and step 06 submits a second
RestoreJob against the populated topic and requires it to be refused
with the topic untouched.
Smaller findings from the same round: the topics parameter is held to
Kafka's name charset (a literal "*" no longer path-expands) and refused
on a repeated name, which produced a Succeeded backup no restore could
balance; the strategy verifies S3 TLS against the seaweedfs CA projected
into the <app>-backup-s3 Secret instead of curl -k, following the
rabbitmq example, so the e2e endpoint moves to the .svc name the serving
cert covers; the image lookup keeps digest-pinned references; the
default namespace is tenant-root, the only one the demo ever ran in;
step 05 caches the resolved Backup name for the restore steps; the decoy
count subtracts begin offsets like the topic count; the bats test name
carries its count again; and the scenario walk-throughs the sibling
examples ship are added.
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-By: LLM
The manifest arrives inside the downloaded tarball but was consumed as
trusted input. A non-numeric partition field slipped through the numeric
test that computes a topic's partition count: an if condition is exempt
from errexit, so test exits 2, the term reads false, the count comes out
short, the topic is created smaller, and the shape check then compares
that short value against itself and passes. The per-topic total still
added up, so the restore reported Succeeded with every keyed record
re-placed. Validate each manifest field on read, refuse an empty manifest
whose verification loops would otherwise run zero times and report a
successful restore of nothing, and guard the restore-side PartitionCount
read the way the backup side already guards its own.
The delete-wait still matched the topic name as a regex; use grep -qxF.
The suite now pins that class of bug instead of relying on review: the
demo topic is orders.v1 and a decoy ordersXv1 sits beside it, matching the
first as a regex but not as a literal. The decoy is never named in the
BackupClass, so dropping any \Q...\E pin fails the run on its own - the
partition-set guard counts the decoy's partitions, or the in-place delete
takes the decoy with it.
Also: the host-side record count skipped a partition missing from the
begin-offset listing instead of defaulting it to 0, undercounting the
topic; the two preflight invariants now cover the kafka flow they already
describe; and the README names the disabled TLS verification.
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Assisted-by: LLM
The backup access preflight printed its port-forward capture only when
the port-forward never came up, and truncated its mc log at the top of
every attempt. On the path that fails in practice, a healthy
port-forward and an S3 round trip that never completes, the output was
the tail of the last attempt with nothing to say which stage it died
at, which reads as a successful upload followed by a hung download.
Print the port-forward capture alongside the mc log on the timeout
path, append the mc log across attempts, and record which of PUT, GET,
compare or DELETE each attempt reached.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.
The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.
Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.
Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.
The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.
Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.
One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.
Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Myasnikov Daniil