## What this PR does After deleting a `Harbor` app CR, two things were left orphaned in the tenant namespace: - the **jobservice / trivy PVCs** (`<release>-jobservice`, and `<release>-trivy` when Trivy is enabled), because `packages/apps/harbor/templates/harbor.yaml` sets `persistence.resourcePolicy: "keep"`, which annotates the upstream Harbor subchart's PVCs with `helm.sh/resource-policy: keep` so Helm/Flux deliberately never deletes them; - cert-manager's **ACME http01-solver** resources (`cm-acme-http-solver-*`), via the same stuck-challenge mechanism as the Bucket issue (#3060) when Harbor issues a non-wildcard cert. This adds a single `post-delete` cleanup hook that: - **deletes the upstream Harbor subchart's kept PVCs** — selecting by `app.kubernetes.io/instance=<release>-system` (the subchart runs as the `<release>-system` HelmRelease, so its PVCs carry that label, **not** `<release>`). A hook is required precisely because `resourcePolicy: keep` blocks normal reclaim. This intentionally covers both the jobservice and trivy PVCs: both hold only regenerable data (jobservice file logs; trivy's vulnerability-DB cache), and registry images live in S3. - **deletes this release's `Certificate`** (deterministic name `<release>-ingress-tls`); cert-manager then cascade-deletes the `Order`, `Challenge` and solver `Pod`/`Service`/`Ingress` objects it owns via owner references. - **host-scoped belt-and-braces sweep** for any ACME http01 solver objects cert-manager may have orphaned (e.g. the Certificate was already gone, so deleting it did not cascade). The sweep is scoped **strictly to this release's Harbor host** — Challenges filtered by `.spec.dnsName`, solver Ingresses by `.spec.rules[0].host` — **not** a namespace-wide `acme.cert-manager.io/http01-solver=true` label sweep, because a Cozystack tenant namespace can host many apps and a blanket label sweep would delete sibling apps' in-flight challenge solvers. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, with CPU/memory requests and limits set. RBAC is scoped to exactly the resources it deletes (`persistentvolumeclaims`, `certificates`, `orders`, `challenges`, solver `ingresses`) and verified against the vendored cert-manager CRDs. Failures are logged but never block teardown. ## Cleanup scope — what is and isn't removed **Removed:** the upstream Harbor subchart's kept PVCs (jobservice + trivy) and this release's cert-manager ACME artifacts (Certificate, and host-scoped Order/Challenge/solver-Ingress). **Not touched (by design):** - **CNPG database PVCs** (`<release>-db-N`) — owned by the CNPG `Cluster` and governed by CNPG's own reclaim policy; they carry CNPG labels, not `app.kubernetes.io/instance=<release>-system`. Re-installing Harbor with a different DB size into the same namespace can therefore leak the old DB PVCs; this is left to CNPG's lifecycle on purpose (the data is the user's). - **Redis PVCs** — managed by the redis-operator with their own labels, likewise untouched. - **Solver `Pod`/`Service`** — not deleted directly; they are removed by cert-manager's owner-ref cascade when the Certificate/Challenge is deleted, so the hook grants no RBAC for them. > [!NOTE] > The PVC sweep is eventually-consistent: while the inner `<release>-system` > HelmRelease is still being uninstalled by Flux, a PVC may briefly stay in > `Terminating` until its consumer Pods (jobservice/trivy) exit. The Job's > deletion request always succeeds; reclaim completes once the pods are gone. > [!WARNING] > **This overrides the intentional `persistence.resourcePolicy: "keep"` setting > and force-deletes the jobservice/trivy PVCs on app deletion.** Those PVCs hold > only jobservice file logs and the trivy vuln-DB cache (registry images live in > S3), so data-loss risk is low — but if retaining them across re-installs is > desired, this is a conscious product decision to revisit. The ACME cleanup > affects only cert-manager's transient artifacts. Fixes #3061. > Follow-up tracked in #3088: digest-pin the cleanup `kubectl` image and route it > through `cozy-lib.images-registry` (pre-existing debt shared with the MariaDB > cleanup hook), out of scope for this PR. ### Release note ```release-note fix(harbor): clean up the orphaned jobservice/trivy PVCs and cert-manager ACME http01-solver resources left behind after a Harbor app is deleted. NOTE: this force-deletes the jobservice/trivy PVCs, overriding persistence.resourcePolicy=keep (jobservice file logs and trivy vuln-DB cache only; registry images are in S3). CNPG database PVCs are not touched. ```
Managed Harbor Container Registry
Harbor is an open-source trusted cloud-native registry project that stores, signs, and scans content.
Prerequisites
The Cozystack Harbor app stores its registry data exclusively in S3-compatible object storage: the chart pins the registry backend to S3 and exposes no filesystem option. That bucket is provisioned through COSI (objectstorage.k8s.io) from a SeaweedFS deployment, so before deploying Harbor the tenant must have SeaweedFS available — enabled on the same tenant or inherited from a parent tenant (the resolved class is propagated down the tenant tree, surfaced as the namespace.cozystack.io/seaweedfs namespace annotation).
Enable it by setting seaweedfs: true on the tenant (or a parent tenant):
seaweedfs: true
Without object storage in the tenant chain, Harbor cannot provision its registry bucket: the <release>-registry BucketClaim/BucketAccess never produces the <release>-registry-bucket credentials secret, so the Harbor HelmRelease stays unreconciled, waiting on BucketInfo.
storageClassis annotated as immutable in the chart schema — seedocs/storage-immutability.mdfor the contract and which consumers enforce it.
Parameters
Common parameters
| Name | Description | Type | Value |
|---|---|---|---|
host |
Hostname for external access to Harbor (defaults to 'harbor' subdomain for the tenant host). | string |
"" |
storageClass |
StorageClass used to store the data. | string |
"" |
Component configuration
| Name | Description | Type | Value |
|---|---|---|---|
core |
Core API server configuration. | object |
{} |
core.resources |
Explicit CPU and memory configuration. When omitted, the preset defined in resourcesPreset is applied. |
object |
{} |
core.resources.cpu |
Number of CPU cores allocated. | quantity |
"" |
core.resources.memory |
Amount of memory allocated. | quantity |
"" |
core.resourcesPreset |
Default sizing preset used when resources is omitted. |
string |
t1.small |
registry |
Container image registry configuration. | object |
{} |
registry.resources |
Explicit CPU and memory configuration. When omitted, the preset defined in resourcesPreset is applied. |
object |
{} |
registry.resources.cpu |
Number of CPU cores allocated. | quantity |
"" |
registry.resources.memory |
Amount of memory allocated. | quantity |
"" |
registry.resourcesPreset |
Default sizing preset used when resources is omitted. |
string |
t1.small |
jobservice |
Background job service configuration. | object |
{} |
jobservice.resources |
Explicit CPU and memory configuration. When omitted, the preset defined in resourcesPreset is applied. |
object |
{} |
jobservice.resources.cpu |
Number of CPU cores allocated. | quantity |
"" |
jobservice.resources.memory |
Amount of memory allocated. | quantity |
"" |
jobservice.resourcesPreset |
Default sizing preset used when resources is omitted. |
string |
t1.nano |
trivy |
Trivy vulnerability scanner configuration. | object |
{} |
trivy.enabled |
Enable or disable the vulnerability scanner. | bool |
true |
trivy.size |
Persistent Volume size for vulnerability database cache. | quantity |
5Gi |
trivy.resources |
Explicit CPU and memory configuration. When omitted, the preset defined in resourcesPreset is applied. |
object |
{} |
trivy.resources.cpu |
Number of CPU cores allocated. | quantity |
"" |
trivy.resources.memory |
Amount of memory allocated. | quantity |
"" |
trivy.resourcesPreset |
Default sizing preset used when resources is omitted. |
string |
t1.nano |
database |
PostgreSQL database configuration. | object |
{} |
database.size |
Persistent Volume size for database storage. | quantity |
5Gi |
database.replicas |
Number of database instances. | int |
2 |
redis |
Redis cache configuration. | object |
{} |
redis.size |
Persistent Volume size for cache storage. | quantity |
1Gi |
redis.replicas |
Number of Redis replicas. | int |
2 |