The backup round-trips drive examples/backups, whose bucket waits gave
up at 300s and whose postgres waits at 360s, under the 600s install
timeout cozystack-api gives each of those releases. The kafka example's
bucket wait passed no budget and took the helper's 300s default. A
failed install there reported only that the wait ran out.
Raise each to 660s, and grow the Chainsaw op around every walkthrough
by what its waits gained, so a failure at any later wait keeps the
headroom it had and is still reported by the script rather than cut
off by the op. The kafka and kafka-metadata Kafka waits stay: their
7h install timeout cannot be cleared inside the E2E job cap.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
cozystack-api gives every release it generates a 600s install timeout,
and these asserts gave up at 5-6m. A failed install was therefore
reported as a bare assert timeout while the release was still
installing and had not yet recorded why it failed.
Raise them to 11m, above that floor, so the release's own reason is on
record before Chainsaw gives up. The cost is up to six more minutes on
a red run of each suite; a green run is unchanged. vm-instance-test
stays: vm-instance-rd disables the install wait, so its release goes
Ready as soon as helm has applied.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
messaging-topology-operator 1.14.2 caches every Secret in the cluster: its
ClusterRole grants secrets list/watch cluster-wide with no namespace or label
restriction, so the controller-runtime cache holds the whole set. Against the
vendored 500Mi limit this OOMKills on any cluster that has grown enough
Secrets, and it never recovers, because the initial LIST alone no longer fits.
While it is down no topology CR reconciles, for any tenant.
Observed on a cluster with 8211 Secrets across 291 namespaces (293 MB, 87% of
it Helm release history): a flat 333Mi working set for days, then the ceiling,
then a permanent crashloop. The operator was caching 8211 Secrets to read 2.
Upstream fixed this by gating the Secret informer on a label, released in
v1.20.0, and by bypassing the cache for Service reads, released in v1.20.1.
Refresh the vendored manifest to v1.20.2, the current patch release carrying
both, through the package's existing make update pipeline.
The bump alone would break every install: the app chart's credentials Secret is
referenced by User.spec.importCredentialsSecret, and from 1.20 the admission
webhook rejects a user-provided Secret that does not carry
rabbitmq.com/topology-operator=true. Label it in the same change, pin the
label in the chart's unit tests, and give the e2e RabbitMQ a user so the
webhook and the User reconcile are exercised against the real operator.
The RabbitmqCluster default-user Secret needs no label, it is read uncached via
apiReader, and the Status.Binding contract is unchanged since 1.14.2, so the
vendored cluster-operator 2.9.0 stays compatible and does not move here.
Refs #4128
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
Those walkthroughs gave their database HelmReleases 300s, under the
600s install timeout the generated releases carry, so a failed install
surfaced as a bare timeout rather than its own reason. None of these
charts sets a startup probe, so 660s clears both floors, as it does
for mariadb.
Each Chainsaw op that runs one of the scripts grows by exactly what
the waits it reaches gained, so it keeps the margin it had. The
clickhouse op still sits above the sum of every inner ceiling. The
rabbitmq one covers a failed install at the source wait, while the
ceilings up to its target wait add up to more than the op, as they
did before. The mongodb round-trip runs with SKIP_RESTORE, so only its
source wait counts there.
postgres keeps its 360s: whether a suite pays the extra minutes on
every red run is a per-suite call, and this one is not made here.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The reworked rabbitmq backup roundtrip reached the upload step and failed
TLS verification: `curl (60) no alternative certificate subject name matches
target hostname 'seaweedfs-s3.tenant-root'`. The demo Bucket, in-cluster
endpoint resolution and the copied SeaweedFS CA all worked; the remaining
gap is the hostname. The seaweedfs filer serving cert's SANs are
`*.tenant-root.svc` and `*.tenant-root.svc.cluster.local`, which the 2-label
`seaweedfs-s3.tenant-root` matches neither; the 3-label `.svc` FQDN does (and
still resolves via search-domain expansion). Point S3_ENDPOINT at
`https://seaweedfs-s3.tenant-root.svc:8333`.
Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The rabbitmq backup roundtrip was the only backup e2e routing through the
platform cozy-default system bucket, whose advertised endpoint in the CI
sandbox is the unroutable s3.example.org placeholder, so the backup Job
failed with "Could not resolve host: s3.example.org". Mirror the
clickhouse/mariadb backup examples: provision a demo Bucket, project its
access keys into rabbitmq-backup-creds and copy the seaweedfs CA into
rabbitmq-backup-ca, and render a self-contained Rabbitmq strategy +
rabbitmq-default BackupClass whose curl uploads to the in-cluster seaweedfs
endpoint (S3_ENDPOINT override, CURL_CA_BUNDLE for the self-signed cert).
The in-Pod script is kept identical to the shipped cozy-default-rabbitmq
strategy, which is left unchanged; the demo only reuses its client image.
Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.
Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):
- reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
--aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
- A Go-read precondition gates each run on the RabbitmqCluster reporting
AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
precise Ready=False wait instead of a Job that starts and fails.
- The produced Backup records the object location on status.artifact.uri
(rendered from spec.artifactURITemplate; the Backup CRD has no status
subresource, so it persists on Create, as in the Etcd driver).
Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.
Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.
The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>