Commit Graph
20 Commits
Author SHA1 Message Date
Yan Bondarenko 1d8bea039b test(harbor): check that the resized components reach the release
The case that resizes the other components looked only at nginx and
the portal, so it would keep passing if those overrides stopped
reaching the nested release, and then it could not catch nginx or the
portal following another component's values. It now checks the core,
registry and jobservice limits it sets and leaves trivy alone, as
nothing in the case reads it. The portal case also checks its memory
request, as the nginx case does.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-21 13:13:56 +03:00
Yan Bondarenko 7c0634586b test(harbor): pin the resources of the nginx proxy and the portal
Both containers run nginx with worker_processes auto, so their idle
memory follows the CPU count of the node, whatever their own CPU limit
is. In a tenant namespace a container without resources gets the
LimitRange default of 128Mi, which the proxy outgrows on a node with
many CPUs before it serves a request. The presets that keep both out
of that default have no test, so dropping them or tying them to
another component's values would go unnoticed.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-21 12:56:07 +03:00
Aleksei Sviridkin 889bb79236 feat(harbor): restrict the portal and API to listed source ranges (#4300)
## What this PR does

Adds `managementWhitelist` to the Harbor app, a list of IP addresses or
CIDR ranges allowed to reach the web portal and the Harbor API, and I
made it a typed list on purpose. The obvious alternative, a free-form
`ingress.annotations` map, gives tenants a way into the shared ingress
controller. The platform ingress-nginx runs with
`allowSnippetAnnotations: true` (the kubernetes app needs a
`configuration-snippet`), and v1.11.5 defaults `annotations-risk-level`
to `Critical`, so it accepts any annotation, `server-snippet` included.
Tenants can't create Ingress objects themselves, but they can write any
field of an app spec. An annotation map on the Harbor spec would let any
tenant put arbitrary nginx config into a controller that serves every
tenant. A typed list is also what the repo already does for this:
`filer.whitelist` in `extra/seaweedfs`, `whitelist` in `extra/ingress`
and `tcp-balancer`.

The use case is a registry that serves customers: pulls and pushes come
from anywhere, the admin UI and API should answer only to an office or
VPN range. Right now the portal, `/api/v2.0` and `/v2/` sit on one
Ingress and one public host, and a tenant can't restrict any of them.

With a non-empty list the existing Ingress gets
`nginx.ingress.kubernetes.io/whitelist-source-range`, and `/v2` and
`/service/token` move to a second Ingress, `<release>-registry-ingress`,
on the same host. ingress-nginx applies that annotation only to the
locations of the Ingress that carries it, so those two stay open. They
are what a registry client needs: Harbor answers an anonymous `/v2/`
request with a Bearer challenge whose realm is `/service/token` on the
same host. Everything else under `/service/` is
`/service/notifications/*`, internal job callbacks that the bundled
nginx already answers with 404, so it stays on the restricted side. The
second Ingress uses the same TLS secret and has no cert-manager
annotations, otherwise ingress-shim gets two owners for one Certificate.
The HTTP-01 solver is its own Ingress and is not restricted. With the
default empty list the chart renders exactly what it renders today.

An HTTPRoute can't express a source restriction, so in Gateway API mode
a non-empty list fails the render. Dropping it silently would leave the
portal public while the spec says it is restricted.

Each entry has to be an IP address or CIDR range, and a bad one fails
the render. ingress-nginx reads an empty `whitelist-source-range` as no
restriction and turns an unparseable one into a 503 for the whole
location, and the values schema generator can't validate list items.

Three things users should know. Harbor-to-Harbor replication that pulls
from this registry goes through its API, so the other registry's address
has to be on the list. The check uses the client address that
ingress-nginx sees: with `externalTrafficPolicy: Local` or PROXY
protocol that is the real client, behind another proxy it is the proxy.
And for the portal and API the list replaces the tenant ingress
`whitelist` rather than narrowing it.

Disabling the portal is not here. The vendored Harbor chart (1.19.1)
always renders the portal Deployment and its internal nginx always
proxies `/` to it, so there is no small switch to wire.

The new Ingress is added to the dashboard Role and to the ingress
selector in the ApplicationDefinition.

### Screenshots

No UI code change. The field reaches the dashboard form through the
regenerated schema and `keysOrder`.

### Downstream repositories

The terraform provider models Harbor by hand, so the new field needs a
follow-up there: cozystack/terraform-provider-cozystack#46. The website
reference page is generated from `README.md` on release. Nothing else in
the trigger map is touched: no package added or renamed, no platform,
installer or `hack/` change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/46
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(harbor): add `managementWhitelist`, a list of IP addresses or CIDR ranges allowed to reach the Harbor web portal and API. Only `/v2/` and `/service/token` stay reachable from any address. The default is an empty list, which keeps the current behaviour. Not supported with Gateway API publishing: a non-empty list fails the render there. Each entry must be an IP address or CIDR range; for the portal and API the list replaces the tenant ingress `whitelist`.
```
2026-10-01 17:39:56 +03:00
IvanHunters 2fa51bf587 test(charts): isolate guard coverage for every guarded template
Add per-template guard suites so each guarded hostname site is exercised
on its own: harbor harbor.yaml and httproute.yaml, bootbox machines.yaml,
seaweedfs cosi-deployment.yaml, bucket deployment.yaml and httproute.yaml,
monitoring alerta.yaml, grafana httproute.yaml and the grafanaHost helper
via oidc-keycloak.yaml. Each suite renders only its target template, so
removing that file's guard reddens it. Also add a postgres positive case
that renders the Certificate with an apex set (external=true).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 18:32:49 +03:00
IvanHunters 37f59369d3 fix(charts): fail fast on empty tenant apex in host composition
Charts that publish a service compose their hostname as
<sub>.<_namespace.host>. When the injected tenant apex is empty and no
per-app host override is set, they used to emit a malformed trailing-dot
hostname (e.g. "harbor.") into Ingresses, HTTPRoutes and Certificate
dnsNames silently. Source the apex through cozy-lib.ns-host-required at
each hostname-emitting site so the render fails with a clear message
instead, matching the gateway path.

Covered: harbor, postgres (external), qdrant (external), kubernetes
(control-plane endpoint), vpn, seaweedfs, bootbox, monitoring
(grafana, alerta, oidc client), and system/bucket (adds the cozy-lib
subchart symlink it lacked). The guard is placed at the point each
resource is actually emitted, so standalone renders, no-exposure
tenants and the kubernetes no-etcd beacon path are unaffected. The
post-delete cleanup hooks are left unguarded so an empty apex cannot
block uninstall.

Behavior change: a tenant with an empty injected apex and no host
override that enables one of these components now fails its render
(HelmRelease NotReady) instead of deploying a broken hostname. In
practice the apex is always injected (root has root-host, children
inherit it), so this only trips a genuinely misconfigured tenant.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 15:14:45 +03:00
Aleksei Sviridkin fd2a6a05e4 fix(harbor): refuse managementWhitelist entries that are not addresses
ingress-nginx reads an empty whitelist-source-range as no annotation
and falls back to the open default, so managementWhitelist: [""]
rendered a split that looked restricted while the portal and API
stayed public. One unparseable entry went the other way and turned
the whole portal location into a 503.

The values schema generator has no item-level constraints, so each
entry is checked at render time and a bad one fails the release with
the offending value in the message. IPv4 is matched strictly; IPv6 by
character set only, which still leaves a malformed IPv6 entry failing
closed in the controller.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 15:52:35 +03:00
Aleksei Sviridkin 5e3ffbe634 feat(harbor): restrict the portal and API to listed source ranges
Harbor serves its web portal and management API on the same host as
the registry endpoints, and a tenant had no way to limit who reaches
the portal and API without also cutting off image pulls.

managementWhitelist restricts every path except /v2 and
/service/token, which move to a second Ingress on the same host, so
docker login, pull and push keep working from any address. The field
is a list of ranges rather than a free-form annotation map: the
shared ingress-nginx accepts snippet annotations, so arbitrary
annotations from a tenant-writable spec would reach its nginx
configuration.

Gateway API publishing has no equivalent, so a non-empty list fails
the render there instead of being silently dropped.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 14:56:57 +03:00
Yan Bondarenko f886f26ad4 fix(apps): bound the wait the cleanup hooks now perform
With watch granted, kubectl delete waits for the claims to be gone, and
its default for that wait is a week. A claim held by a pod that will not
stop, or a stuck detach, would hold the post-delete hook until
helm-controller's 300s uninstall budget ran out and leave the
HelmRelease stuck uninstalling.

The PVC deletes in the clickhouse, harbor, mariadb and qdrant hooks now
wait 120s at most and report the claims still terminating by name, and
each Job carries activeDeadlineSeconds 270 with a 10s termination grace
period, the numbers packages/apps/tenant/templates/cleanup-job.yaml
documents against that budget, so a kubectl that never returns cannot
hold the hook either.

The mariadb Role no longer grants watch on the named Secrets: they carry
no finalizer, so the delete never reaches the watch, and a resourceNames
rule does not authorize a collection watch anyway.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
2026-09-17 21:49:51 +03:00
Yan Bondarenko 711fcc168c fix(apps): let the cleanup hooks watch the objects they delete
kubectl delete waits for the object to disappear unless told otherwise,
and that wait is a watch. The post-delete hook Roles of harbor,
clickhouse, mariadb and qdrant granted get, list and delete only, so
every uninstall logged a forbidden watch, the wait was skipped and the
hook reported success before the PVCs were actually gone.

The rules now carry watch for every resource the hooks delete, including
the named Secrets the mariadb hook removes.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-09-09 17:53:57 +03:00
Aleksei Sviridkin e9411d6b8c fix(apps): make the post-delete cleanup hooks report failure
The qdrant and harbor post-delete hooks guarded every delete with
`|| echo "WARNING: ..." >&2`, set no `-e`, and ended on an
unconditional completion message, so the container exited 0 on
every path. Helm read that as a successful hook and the
hook-succeeded delete policy removed the Job together with the pod
that carried the warning, leaving an uninstall that reclaimed
nothing indistinguishable from one that had nothing to reclaim.

Count the failures instead and set the exit code at the end, so a
failed delete does not skip the deletes that follow it. An object
that is already gone stays a success: every delete keeps
--ignore-not-found, which exits 0 in that case. This is the shape
packages/apps/mariadb already uses for the same problem.

Both hooks also consumed the output of `kubectl get` directly --
harbor by iterating it, qdrant by piping it into grep, whose status
the pipeline then reports. A listing that failed therefore arrived
as an empty list and the sweep was skipped rather than reported.
Each listing is now captured before it is used, so a listing that
failed and one that returned nothing are different events.

A hook that exits non-zero keeps its Job and pods, which is the
point: they hold the log. Give the Job one retry rather than the
Kubernetes default of six or none at all. Every delete is
idempotent, so a second attempt re-issues an API call rather than
re-running any logic, and one short backoff is what separates a
transient apiserver error from a real one -- these failures become
visible for the first time here, and a signal that fires on noise
stops being read, including on the day it is right. Six is the far
end of the same mistake: ten minutes of backoff across seven pods
outlasts the hook timeout, and Helm reports that timeout instead of
the hook's own error.

The qdrant Chainsaw suite explained its post-delete pod collector
by quoting the guard string removed here and concluding the pod is
always collected away. That is now backwards: a failed cleanup is
exactly when the pod survives and the collector is the evidence.
Rewrite the note, and the sentence in the harbor suite pointing at
it.

hack/app-cleanup-hooks_test.bats renders both scripts and runs them
against a fake kubectl. There are eight failure paths across the two
hooks -- six in harbor, two in qdrant -- and each has a case that
goes red when that one path alone is reverted to a swallowed
warning. Message assertions are anchored on the hook's own `ERROR: `
prefix, because the fake kubectl names the object in its own
diagnostic too and an unanchored match is satisfied by the fixture.
The helm-unittest suites pin the retry limit and the hook delete
policy: adding hook-failed to that annotation hands the Job back to
Helm to delete on failure and destroys the evidence this change
exists to keep, and until these asserts that reversal left both
suites green.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-12 18:43:02 +03:00
Aleksei Sviridkin 79ceaf1dc2 fix(harbor): retry the nested system install instead of uninstalling it (#3621)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Harbor's app chart renders a nested HelmRelease that owns the whole
workload: the Deployments, the inline RedisFailover and its volume. It
had no install strategy, so it got Flux's default, and install
remediation is an uninstall in every case Flux offers. A first install
that misses its budget gets torn down, and the next one provisions the
redis volume and runs the master election again from scratch.

That happens on a loaded runner. harbor-core waits on redis, redis waits
on its volume and the master election, and the nginx Deployment hits the
default `progressDeadlineSeconds` of 600s. Flux reads that as a stalled
resource and fails the install early. The e2e wait budget is 10m, the
same as the deadline, so after a teardown the second attempt has less
time than a first install needs.

`RetryOnFailure` keeps the manifests applied and retries as an upgrade.
Nothing is deleted, so the volume, the elected master and harbor-core's
startup all survive, and the retry costs 30s instead of a full install.
A Deployment clears `ProgressDeadlineExceeded` by itself once its pods
go ready, so the retry lands as soon as the stack converges.

This drops the teardown, not the deadline. The suite applies the release
in one step and waits for it in later ones, so part of the budget is
still left when the fail-early hits. That leftover is useless to a fresh
install and usable to a retry that resumes. Whether the deadline or the
budget also needs moving is a separate question, and an early deadline
only becomes safe once a failed install stops being torn down. In
production the win holds without any of that: a slow first install no
longer deletes the Redis volume and the database.

Both actions carry the strategy. The controller runs the retry of a
failed install as an upgrade, so with install alone the retry's own
failure falls through to upgrade remediation. The remediation blocks
stay: an absent release consults install, an out-of-sync one consults
upgrade, both by retry count rather than by strategy.

It goes on the nested release, not the parent. The parent is built from
the Application by the apiserver, which already stamps this strategy on
both actions. The nested one owns the workload and the volume a
remediation deletes.

Same shape as the tenant CNI and CSI releases in #3552 and #3581, and
the unit test follows theirs. Relates to #3445.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(harbor): retry a failed Harbor install instead of uninstalling the release, so a slow first install no longer deletes the Redis volume and starts over
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Harbor installations and upgrades now automatically retry every 30
seconds when failures occur.
* Existing unlimited remediation retries remain available for both
operations.

* **Tests**
* Added automated coverage to verify Harbor retry behavior and
remediation settings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-08 14:04:13 +03:00
Aleksei Sviridkin a3c1408d08 test(harbor): select hook and release documents by kind, not by index
Both suites addressed their subject by document index. An index is a
position, so a template that gains a document ahead of the one under test
moves every later assertion onto a different object, and what happens then
depends on the assertion: measured by inserting one document at the head of
each template, the cleanup suite reddens outright, while two of the four
jobservice tests keep passing because a `notExists` against an object that
never had the path is satisfied for the wrong reason.

Indices are also easy to get wrong by hand. helm-unittest numbers documents
in template source order, while `helm template` prints them sorted by kind,
so the rendered output is not the order being counted.

Every document these suites assert on is unique by kind within its template,
so each test now names its subject. A selector that matches nothing fails,
including on the `notExists` tests, so the anchor holds in the direction that
was silent before.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 03:09:46 +03:00
Aleksei Sviridkin f4dc5d4d3c fix(harbor): retry the nested system install instead of uninstalling it
The harbor app chart renders a nested HelmRelease that owns the whole
workload: the harbor Deployments, the inline RedisFailover and its
volume. It carried Flux's default install strategy, and install
remediation is an uninstall in every case Flux offers, so a first
install that went over budget was torn down between attempts and the
next one re-provisioned the redis volume and re-ran the master election
from zero.

That is reachable on a loaded runner. harbor-core blocks on redis while
redis waits for its volume and its master election, and the nginx
Deployment reaches the default progressDeadlineSeconds of 600s, which
Flux reads as a stalled resource and fails the install early. The e2e
wait budget for the release is the same 600s, so one teardown cycle
consumes all of it and the second attempt cannot finish however fast it
converges.

RetryOnFailure keeps the manifests applied and retries the failed
release, so the convergence already under way continues instead of
restarting. Both actions carry the strategy because the controller
performs the retry of a failed install as an upgrade, so on install
alone the retry's own failure falls through to upgrade remediation. The
cost is that a real upgrade failure retries rather than rolls back,
which gives up no stable safety net: with retries: -1 and the default
strategy, upgrade remediation already rolled back and retried without
end. The remediation blocks stay because an absent release consults
install and an out-of-sync one consults upgrade, both by retry count
rather than by strategy.

The change lands on the nested release rather than on the parent: the
parent is built from the Application by the apiserver, which already
stamps this strategy on both actions, while the nested release is the
one that owns the workload and the volume a remediation deletes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 00:12:53 +03:00
Алексей Артамонов a76c5929fd fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry
The post-delete cleanup Jobs in the harbor and mariadb charts hardcoded
`docker.io/clastix/kubectl:v1.32` — a moving tag pulled directly from
Docker Hub. This broke build reproducibility (the tag can be re-pushed),
was a supply-chain risk, and could not be satisfied on air-gapped /
mirrored clusters, where it failed at the worst moment: during uninstall,
when the post-delete hook runs and stalls release teardown.

Add a canonical `cozy-lib.image` helper that prefixes a registry-relative
image path with `cozy-lib.images-registry` when set and returns it
unchanged (no leading "/") when empty, so standard installs resolve from
the default registry and mirrored installs resolve internally. Consume it
from both cleanup hooks with the image digest-pinned to the immutable
multi-arch index sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c
(the same digest already pinned for postgres-operator's webhook hook).

Keep the pin fresh with a custom Renovate manager: the repo disables the
helm-values manager (enabledManagers), and the ref is built at render time
by the helper rather than being a Dockerfile/go.mod dependency, so the
built-in managers cannot see it. A regex manager over the hook templates
tracks clastix/kubectl by tag+digest — mirroring the existing extra/etcd
manager — so the supply-chain pin does not silently age.

Also harden `cozy-lib.images-registry` against a nil .Values._cluster so
the helper never crashes the render — relevant precisely because the hook
renders during teardown. Helm unit tests assert the rendered image (routed,
digest-pinned, no leading slash) by pattern rather than an exact digest, so
a Renovate bump keeps them green.

Closes #3088

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 17:44:59 +03:00
scooby87 bef42229d4 fix(harbor): clean up orphaned jobservice PVC and ACME solver resources on delete (#3061) (#3077)
## What this PR does

After deleting a `Harbor` app CR, two things were left orphaned in the
tenant
namespace:

- the **jobservice / trivy PVCs** (`<release>-jobservice`, and
`<release>-trivy`
when Trivy is enabled), because
`packages/apps/harbor/templates/harbor.yaml`
sets `persistence.resourcePolicy: "keep"`, which annotates the upstream
Harbor
subchart's PVCs with `helm.sh/resource-policy: keep` so Helm/Flux
deliberately
  never deletes them;
- cert-manager's **ACME http01-solver** resources
(`cm-acme-http-solver-*`), via
the same stuck-challenge mechanism as the Bucket issue (#3060) when
Harbor
  issues a non-wildcard cert.

This adds a single `post-delete` cleanup hook that:

- **deletes the upstream Harbor subchart's kept PVCs** — selecting by
`app.kubernetes.io/instance=<release>-system` (the subchart runs as the
  `<release>-system` HelmRelease, so its PVCs carry that label, **not**
`<release>`). A hook is required precisely because `resourcePolicy:
keep`
blocks normal reclaim. This intentionally covers both the jobservice and
trivy
  PVCs: both hold only regenerable data (jobservice file logs; trivy's
  vulnerability-DB cache), and registry images live in S3.
- **deletes this release's `Certificate`** (deterministic name
`<release>-ingress-tls`); cert-manager then cascade-deletes the `Order`,
`Challenge` and solver `Pod`/`Service`/`Ingress` objects it owns via
owner
  references.
- **host-scoped belt-and-braces sweep** for any ACME http01 solver
objects
cert-manager may have orphaned (e.g. the Certificate was already gone,
so
deleting it did not cascade). The sweep is scoped **strictly to this
release's
Harbor host** — Challenges filtered by `.spec.dnsName`, solver Ingresses
by
  `.spec.rules[0].host` — **not** a namespace-wide
`acme.cert-manager.io/http01-solver=true` label sweep, because a
Cozystack
tenant namespace can host many apps and a blanket label sweep would
delete
  sibling apps' in-flight challenge solvers.

The Job is hardened: non-root (uid 65534), read-only root fs, all caps
dropped,
`seccompProfile: RuntimeDefault`, with CPU/memory requests and limits
set. RBAC
is scoped to exactly the resources it deletes (`persistentvolumeclaims`,
`certificates`, `orders`, `challenges`, solver `ingresses`) and verified
against
the vendored cert-manager CRDs. Failures are logged but never block
teardown.

## Cleanup scope — what is and isn't removed

**Removed:** the upstream Harbor subchart's kept PVCs (jobservice +
trivy) and
this release's cert-manager ACME artifacts (Certificate, and host-scoped
Order/Challenge/solver-Ingress).

**Not touched (by design):**

- **CNPG database PVCs** (`<release>-db-N`) — owned by the CNPG
`Cluster` and
  governed by CNPG's own reclaim policy; they carry CNPG labels, not
`app.kubernetes.io/instance=<release>-system`. Re-installing Harbor with
a
different DB size into the same namespace can therefore leak the old DB
PVCs;
  this is left to CNPG's lifecycle on purpose (the data is the user's).
- **Redis PVCs** — managed by the redis-operator with their own labels,
likewise
  untouched.
- **Solver `Pod`/`Service`** — not deleted directly; they are removed by
cert-manager's owner-ref cascade when the Certificate/Challenge is
deleted, so
  the hook grants no RBAC for them.

> [!NOTE]
> The PVC sweep is eventually-consistent: while the inner
`<release>-system`
> HelmRelease is still being uninstalled by Flux, a PVC may briefly stay
in
> `Terminating` until its consumer Pods (jobservice/trivy) exit. The
Job's
> deletion request always succeeds; reclaim completes once the pods are
gone.

> [!WARNING]
> **This overrides the intentional `persistence.resourcePolicy: "keep"`
setting
> and force-deletes the jobservice/trivy PVCs on app deletion.** Those
PVCs hold
> only jobservice file logs and the trivy vuln-DB cache (registry images
live in
> S3), so data-loss risk is low — but if retaining them across
re-installs is
> desired, this is a conscious product decision to revisit. The ACME
cleanup
> affects only cert-manager's transient artifacts.

Fixes #3061.

> Follow-up tracked in #3088: digest-pin the cleanup `kubectl` image and
route it
> through `cozy-lib.images-registry` (pre-existing debt shared with the
MariaDB
> cleanup hook), out of scope for this PR.

### Release note

```release-note
fix(harbor): clean up the orphaned jobservice/trivy PVCs and cert-manager ACME http01-solver resources left behind after a Harbor app is deleted. NOTE: this force-deletes the jobservice/trivy PVCs, overriding persistence.resourcePolicy=keep (jobservice file logs and trivy vuln-DB cache only; registry images are in S3). CNPG database PVCs are not touched.
```
2026-06-30 15:55:52 +03:00
Алексей АртамоновandClaude Opus 4.8 b25f294d46 fix(harbor): select kept PVCs by release label so the trivy PVC is also cleaned
The post-delete cleanup hook selected PVCs by
app.kubernetes.io/instance=<release>-system, a label only the jobservice PVC
carries (via the chart's "harbor.labels" helper). The trivy PVC is a
StatefulSet volumeClaimTemplate labeled via "harbor.legacy.labels" with a
selector using "harbor.matchLabels" — neither sets
app.kubernetes.io/instance — so the sweep silently missed it. Trivy is enabled
by default, so a default Harbor delete left the trivy PVC orphaned,
contradicting the PR's own claim.

Select on release=<release>-system instead: both kept PVCs carry it, while
CNPG (db) and Redis PVCs use different label schemes and the registry is on S3,
so only the two intended PVCs match. The unit test now asserts the corrected
selector and guards against regressing to the instance label.

Also drop the by-name delete of Orders/Challenges (their hash-suffixed names
make a by-name delete a no-op; the Certificate owner-ref cascade plus the
host-scoped sweep handle them), narrow the hook RBAC to challenges only, and
note the Gateway-mode no-op.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-06-30 14:28:31 +03:00
Myasnikov Daniil bdf520bf74 fix(harbor): preserve jobservice and trivy storageClass on upgrade
The cozy-harbor wrapper never passed storageClass to the jobservice
jobLog PVC, so it was created without a storageClassName and the
default-StorageClass admission stamped the cluster default. Because PVC
storageClassName is immutable, this caused rendered-vs-live drift on
upgrade. Preserve an existing PVC's class via lookup and fall back to the
configured storageClass for fresh installs.

The trivy StatefulSet's volumeClaimTemplate had the same latent drift --
.Values.storageClass was forwarded blindly into a VCT that cannot be
patched at all -- so it gets the same lookup-preserve treatment, keyed
off data-<release>-trivy-0.

A live object pinned to an explicit "" (no dynamic provisioning) now
round-trips through upstream's "-" sentinel instead of collapsing to an
omitted storageClassName, and the spec lookup is guarded with
`| default dict` so a PVC carrying no storageClassName key stays omitted.

Adds a helm-unittest regression suite and wires a test target so
make unit-tests stops skipping packages/apps/harbor.

Fixes #2368

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-29 13:16:34 +05:00
Алексей Артамонов d113273aa9 fix(harbor): clean up orphaned jobservice PVC and ACME solver resources on delete (#3061)
The jobservice PVC (resourcePolicy: keep) and cert-manager ACME
http01-solver objects were left orphaned after a Harbor app CR was deleted.
Add a post-delete hook that deletes the release PVCs (selecting by
app.kubernetes.io/instance=<release>-system, the label the upstream subchart
stamps) and the release Certificate/Order/Challenge plus the http01-solver
sweep. A hook is required because resourcePolicy: keep blocks normal reclaim.
The cleanup Job runs non-root with a read-only root fs and dropped
capabilities; failures are logged but never block teardown.

Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
2026-06-26 13:02:58 +03:00
Aleksei Sviridkin 9e3ae7447b fix(ingress): keep tls section in operator wildcard mode
Dropping the whole tls block in wildcard mode disabled the forced
HTTP-to-HTTPS redirect: ingress-nginx only issues the 308 redirect when
TLS is enabled for the Ingress, while a host without a tls section is
still served on 443 with the default certificate but stays reachable
over plain HTTP. Keep the tls section with the host list and omit only
the per-host secretName, so ingress-nginx falls back to the operator
wildcard configured as the default SSL certificate and the redirect
stays enforced, matching the non-wildcard behaviour.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-10 03:18:42 +03:00
Aleksei Sviridkin b1f5d9579c feat(ingress): serve operator wildcard via default SSL certificate
When publishing.certificates.wildcardSecretName is set, the root
tenant's ingress-nginx controller now serves the operator-supplied
wildcard Secret as its --default-ssl-certificate, and the platform
service Ingresses stop requesting their own per-host ACME certificates.

The flag is set only on the publishing controller (Release.Namespace ==
expose-ingress) because ingress-nginx loads the Secret from its own
namespace; child-tenant controllers cannot reach a Secret in another
namespace, so they keep the existing ACME path.

Each service Ingress (dashboard, keycloak, grafana, alerta, bucket,
linstor-gui, harbor, seaweedfs S3, bootbox) drops its cert-manager
cluster-issuer annotation, the http01 challenge-class annotation, and
its tls block in wildcard mode; nginx then falls back to the default
certificate for those hosts. Other annotations and routing are
unchanged. Each file keeps its existing render guard intact, and the
annotations block collapses cleanly where the cert annotations were its
only content.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-10 03:08:41 +03:00