IvanHunters 857c883695 fix(monitoring-agents): relax fluent-bit probes for loaded installs (#3045)
## What this PR does

**Problem**

On a loaded install the `monitoring-agents` fluent-bit DaemonSet
crash-loops and its HelmRelease times out (`timeout waiting for:
[DaemonSet/.../monitoring-agents-fluent-bit status: 'InProgress']`),
cascading into velero → backupstrategy-controller `DependencyNotReady`
and failing the whole install.

Root cause: fluent-bit's HTTP server on `:2020` (the liveness/readiness
target) binds only AFTER the `filter:kubernetes` "testing connectivity
with API server" step. During a concurrent install the kube-API can be
unreachable for ~10s (observed `connection #144 timeout after 10 seconds
to: kubernetes.default.svc:443`), which delays the `:2020` bind. The
vendored subchart's default probes (`initialDelaySeconds 0,
timeoutSeconds 1, failureThreshold 3`) fire into a connection-refused
port immediately, and the single-threaded HTTP server can't always meet
a 1s timeout while the engine is busy — so liveness SIGTERMs fluent-bit
(exitCode 0, not OOM) into a restart loop.

**Fix**

Relax both probes in the first-party umbrella `values.yaml`
`fluent-bit:` block: `initialDelaySeconds 15, periodSeconds 10,
timeoutSeconds 5, failureThreshold 6` (~75s tolerance from container
start, ~60s steady-state). The `httpGet` target is unchanged — path `/`
on port `http` (`:2020`). The change is values-only; the vendored
subchart is not touched.

**Why load-triggered (not always-broken)**

A steady-state cluster runs the identical probe config with the
DaemonSet 3/3 Ready, 0 restarts, 26 days. The probe only fails when the
`:2020` bind is delayed by API unreachability under install-time load.
Steady-state-stable + flaky-only-under-load = probe-too-aggressive, not
a target or config error.

**Test**

Added a helm-unittest (`tests/fluent-bit_probes_test.yaml`) pinning both
the relaxed timing and the preserved `:2020` httpGet target (path `/`,
port `http`) on the rendered DaemonSet, so a future change that
re-tightens the probes or breaks the path/port is caught.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
fix(monitoring-agents): relax fluent-bit liveness/readiness probes so the DaemonSet no longer crash-loops on a loaded install when its :2020 HTTP server binds late behind a busy kube-API
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Adjusted Fluent Bit health probes to use more relaxed timing, reducing
the chance of restart loops on busy installations.
* Kept the readiness check pointed at the expected HTTP path and port
while making probe behavior more stable.
* **Tests**
* Added test coverage to verify the generated probe configuration stays
consistent.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-25 19:22:08 +03:00
2025-10-08 09:43:34 +05:00
2025-04-01 18:48:14 +02:00
2024-02-08 12:04:32 +01:00
2026-03-17 02:57:52 +05:00

Cozystack Cozystack

Open Source Apache-2.0 License Support Active GitHub Release GitHub Commit OpenSSF Best Practices

Cozystack

Cozystack is a free platform and framework for building clouds.

Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.

With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.

Use Cozystack to build your own cloud or provide a cost-effective development environment.

Cozystack user interface

Use-Cases

Documentation

The documentation is located on the cozystack.io website.

Read the Getting Started section for a quick start.

If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.

Versioning

Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.

Contributions

Contributions are highly appreciated and very welcomed!

In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.

You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.

If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.

Community

You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.

License

Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.

Commercial Support

A list of companies providing commercial support for this project can be found on official site.

Languages
Go 48%
Shell 37%
TypeScript 6.9%
Go Template 5.3%
Makefile 1.4%
Other 1.2%