mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
ci(e2e): raise the E2E job limit from 215 to 265 minutes (#5288)
## What this PR does Full chainsaw run doesn't fit in 215 minutes anymore. Last four nightly runs were cancelled at the limit, and PRs that select full suite end same way while every suite that got to run passed (#5283 for example). This raises limit for all four jobs that run the suite (pull-requests, e2e-fork, e2e-tag, nightly) from 215 to 265, so they stay on one figure. Comments that state the cap follow it. Making e2e itself shorter is separate work. Part of #5261, `Collect report` ceiling in fork workflow from there is not covered here. ### Screenshots ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note ci(e2e): E2E job limit raised from 215 to 265 minutes. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Chores** * Increased the time limit for end-to-end test runs from 215 to 265 minutes across applicable workflows. This allows longer test runs to complete before the overall job is stopped. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
9 files changed
+15
-14
No files matched your search
@@ -543,7 +543,7 @@ jobs:
|
||||
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
|
||||
# running into its node-join deadline, so a red there can end as a
|
||||
# cancelled job. Held to one figure across every lane that runs the suite.
|
||||
timeout-minutes: 215
|
||||
timeout-minutes: 265
|
||||
permissions:
|
||||
contents: read
|
||||
packages: read
|
||||
|
||||
@@ -137,7 +137,7 @@ jobs:
|
||||
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
|
||||
# running into its node-join deadline, so a red there can end as a
|
||||
# cancelled job. Held to one figure across every lane that runs the suite.
|
||||
timeout-minutes: 215
|
||||
timeout-minutes: 265
|
||||
permissions:
|
||||
contents: read
|
||||
checks: write # Breakpoint action updates its dedicated "Breakpoint Open" Check Run.
|
||||
@@ -292,7 +292,7 @@ jobs:
|
||||
# collect-report packages cozyreport plus per-failed-test crust-gather
|
||||
# snapshots captured before Chainsaw cleanup.
|
||||
#
|
||||
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
|
||||
# A ceiling of its own, not the job's 265 minutes. `Upload cozyreport.tgz`
|
||||
# is the next step and the collector writes its tarball at the very end of
|
||||
# its run, so a read that never returns spends what is left of the job
|
||||
# budget, the job is cancelled, and the upload never runs -- the artifact
|
||||
|
||||
@@ -250,7 +250,7 @@ jobs:
|
||||
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
|
||||
# running into its node-join deadline, so a red there can end as a
|
||||
# cancelled job. Held to one figure across every lane that runs the suite.
|
||||
timeout-minutes: 215
|
||||
timeout-minutes: 265
|
||||
permissions:
|
||||
contents: read
|
||||
packages: read
|
||||
@@ -385,7 +385,7 @@ jobs:
|
||||
path: _out/chainsaw-report.xml
|
||||
if-no-files-found: ignore
|
||||
|
||||
# A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz`
|
||||
# A ceiling of its own, not the job's 265 minutes. `Upload cozyreport.tgz`
|
||||
# is the next step and the collector writes its tarball at the very end of
|
||||
# its run, so a read that never returns spends what is left of the job
|
||||
# budget, the job is cancelled, and the upload never runs -- the artifact
|
||||
|
||||
@@ -953,7 +953,7 @@ jobs:
|
||||
# Covers a green full suite. It does not cover a tenant-Kubernetes suite
|
||||
# running into its node-join deadline, so a red there can end as a
|
||||
# cancelled job. Held to one figure across every lane that runs the suite.
|
||||
timeout-minutes: 215
|
||||
timeout-minutes: 265
|
||||
permissions:
|
||||
contents: read
|
||||
packages: read
|
||||
@@ -1224,7 +1224,7 @@ jobs:
|
||||
|
||||
# ▸ Collect debug information (always runs)
|
||||
#
|
||||
# A ceiling of its own, not the job's 215 minutes. `Upload
|
||||
# A ceiling of its own, not the job's 265 minutes. `Upload
|
||||
# cozyreport.tgz` is the next step and the collector writes its tarball
|
||||
# at the very end of its run, so a read that never returns spends what
|
||||
# is left of the job budget, the job is cancelled, and the upload never
|
||||
@@ -1235,7 +1235,7 @@ jobs:
|
||||
# 30 minutes is measured against the step: it takes 3-42s across sampled
|
||||
# runs, on failed e2e jobs as well as green ones. It RESERVES nothing --
|
||||
# `timeout-minutes` caps a step, it does not carve budget out of the job,
|
||||
# so a suite that has already burned 210 of the 215 leaves this step five
|
||||
# so a suite that has already burned 260 of the 265 leaves this step five
|
||||
# minutes and the upload is cancelled with everything else. What the cap
|
||||
# buys is that a hung read cannot be the reason.
|
||||
# `continue-on-error` because the ceiling must not change what the job
|
||||
|
||||
+1
-1
@@ -121,7 +121,7 @@
|
||||
# the number; do not expect to find it written here.
|
||||
#
|
||||
# TWO ceilings sit outside it, and they fail differently. The job is capped at
|
||||
# `timeout-minutes: 215`, and reaching that cancels every step still to come --
|
||||
# `timeout-minutes: 265`, and reaching that cancels every step still to come --
|
||||
# the artifact upload among them. The step running this script is capped at 30 of
|
||||
# its own, and reaching THAT ends only this step: the uploads after it still run,
|
||||
# and `continue-on-error` on that step keeps the overrun out of the job's result,
|
||||
|
||||
+1
-1
@@ -208,7 +208,7 @@ _cozy_on_exit() {
|
||||
# capture would read as a complete one listing fewer crash-loops than
|
||||
# actually occurred. This handler's whole worst case is ~1350s (330 + 390
|
||||
# + 630, each leg including its 30s kill grace) against a job capped at
|
||||
# 215 minutes with no per-step timeout —
|
||||
# 265 minutes with no per-step timeout —
|
||||
# deliberate, for the same reason recorded in hack/e2e-chainsaw/.chainsaw.yaml:
|
||||
# a bounded, honest wait beats a kill that collects nothing.
|
||||
if [ "$_prev_rc" -ne 0 ]; then
|
||||
|
||||
@@ -66,14 +66,14 @@ spec:
|
||||
# script: it SIGKILLs the shell alone, orphaned children hold the pipe open,
|
||||
# and the op overruns (#4268 measured 606s under a 300s ceiling), so the
|
||||
# nominal ~5m was a figure a busy catch could exceed, not a cap. Tests run
|
||||
# sequentially, and the E2E job is capped at 215 timeout-minutes. A cascade
|
||||
# sequentially, and the E2E job is capped at 265 timeout-minutes. A cascade
|
||||
# reddening many suites can therefore spend a large share of the job in
|
||||
# diagnostics. That is deliberate — the old ceiling
|
||||
# bought its speed by collecting nothing at all — and it is bounded: both legs
|
||||
# only approach their ceilings on a genuinely degraded cluster, a systemic
|
||||
# failure usually reds the install gate and skips the E2E step entirely, and
|
||||
# the report-upload steps are `if: always()`. That last point does NOT extend
|
||||
# to a job timeout: a 215-minute cap ends the job `cancelled`, and the ~4m48s
|
||||
# to a job timeout: a 265-minute cap ends the job `cancelled`, and the ~4m48s
|
||||
# grace window is consumed by the earlier `Collect report` step, so both
|
||||
# cozyreport.tgz and the job log are lost (measured on runs 29557003429 and
|
||||
# 29568562337). If this ever does start truncating runs, bound the number of
|
||||
|
||||
@@ -35,7 +35,7 @@
|
||||
# and the teardown. This is the third suite in a full run that brings up a
|
||||
# tenant cluster, next to kubernetes-latest and kubernetes-previous. A failure
|
||||
# that stalls worker join everywhere reddens all three, and their red paths
|
||||
# together can exceed the E2E job's 215-minute cap, which cancels the job and
|
||||
# together can exceed the E2E job's 265-minute cap, which cancels the job and
|
||||
# loses its log and report. The cap is sized for the red path of the
|
||||
# kubernetes pair only; this suite is not counted in it, and that is an
|
||||
# accepted risk.
|
||||
|
||||
@@ -80,7 +80,8 @@ spec:
|
||||
# wait, all on the passing path. Raising the ceiling to cover them is not
|
||||
# available -- the guard in hack/run-kubernetes-serial-console_test.bats
|
||||
# holds COZY_OP_CEILING equal to BOTH suites' `timeout:`, and two suites
|
||||
# at 79m plus their 22m teardowns exceed the job's own 215-minute cap.
|
||||
# at 79m plus their 22m teardowns take 202m, which with the install and
|
||||
# the rest of a full run does not fit the job's own 265-minute cap.
|
||||
# What is available is a measurement: on the green run of this branch
|
||||
# (33724179053) the whole Test took 569.72s against this 4020s op, with
|
||||
# kubernetes-previous, which carries no OIDC fold, at 470.19s. The
|
||||
|
||||
Reference in new issue
Block a user