Commit Graph
26 Commits
Author SHA1 Message Date
Alexey Artamonov 3c78c0be56 feat(clickhouse): add configurable version parameter
Add a `version` value (v25.8, v25.3, v24.9; default v24.9) to the
ClickHouse chart, backed by files/versions.yaml and resolved by
templates/_versions.tpl for both the ClickHouse server and Keeper
images, so the Cozystack API returns the engine version (spec.version)
like the other managed-DB charts. Default v24.9 -> 24.9.2.42 keeps
existing installations byte-for-byte identical.

The maintainer-run hack/update-versions.sh generator intersects the
server/keeper Docker Hub tags byte-collated (LC_ALL=C) so comm keeps
tags across the X.9 -> X.10 boundary; freezes the default major to the
tag it already ships (read from versions.yaml, so the freeze follows the
default); errors on a dropped default or an unresolvable major; writes
atomically; and runs on stock macOS bash 3.2 / BSD. Covered by
hack/clickhouse-update-versions_test.bats (10 cases) and the
tests/version_test.yaml helm-unittest suite.

Refs: #1246
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-08-24 23:41:27 +03:00
Andrey KolkovandClaude Opus 4.8 9379729b31 fix(clickhouse): address review notes on endpointCA and the CA copy
Review notes from IvanHunters on #3385 (LGTM with non-blocking findings).

1. The backup-s3-ca volume carries no `optional: true` while the adjacent
   useSystemBucket credential refs deliberately do, and the asymmetry was
   undocumented — an operator naming a Secret that does not exist gets the whole
   ClickHouse Pod stuck in ContainerCreating, not just the sidecar. Kept the
   fail-closed behaviour and wrote down why the two cases differ: credentials are
   projected by the platform after install, so a strict ref would wedge every
   fresh release until the first BackupJob, whereas this CA is named by the
   tenant, must already exist, and is trust material — optional would mount an
   empty dir, the Pod would look healthy, and the first backup would fail deep
   inside clickhouse-backup with an x509 error naming neither the Secret nor the
   chart. The comment also states the blast radius plainly.

2. The CA probe swallowed stderr, so an RBAC denial on the cross-namespace read
   was reported as "CA not found" followed by advice (set S3_CA_SECRET) that
   cannot fix a permissions problem. It now separates present / absent /
   not-allowed-to-look and says so, and the README no longer labels step 03
   simply "tenant": copying the CA reads a Secret in tenant-root, which is a
   platform capability. Added a section spelling out the three ways to satisfy
   that (admin credentials, a one-off admin copy, or skipping the CA entirely on
   a publicly-trusted endpoint).

3. Caveat, and a usability regression this branch introduced: on a cluster with
   a publicly-trusted endpoint and no seaweedfs, step 03 hard-exited on a default
   the user never chose. It now distinguishes an explicitly named Secret (still
   an error when missing — a typo should not be guessed past) from the unset
   default, where it warns and continues with backup.endpointCA unset. Verified
   all four paths against a fake kubectl: Forbidden, absent+default,
   absent+explicit, and present.

4. Caveat: endpointCA reaches the clickhouse-backup sidecar only. The legacy
   `schedule` CronJob writes to S3 through restic and does not consume it, so a
   private-CA endpoint combined with `schedule` still fails TLS. Documented on
   the field itself, so it lands in values.schema.json, the README and the
   cozyrds openAPISchema.

The PR body's claim about select-e2e.sh/select-install.sh mappings was stale
after the restructure that removed them; corrected there rather than in code.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-03 15:40:41 +04:00
Andrey KolkovandClaude Opus 4.8 2687e095f4 feat(clickhouse): let the backup sidecar trust a private S3 CA
The clickhouse-backup sidecar had no TLS/CA surface at all: bucket coordinates
and credentials arrive as S3_* env, and nothing let it verify an endpoint whose
certificate is signed by a private CA. Cozystack's own in-cluster SeaweedFS is
exactly that case — it serves :8333 behind the self-signed "SeaweedFS CA" — so a
ClickHouse release could only ever back up to a publicly-trusted endpoint, while
MariaDB (storage.s3.tls.caSecretKeyRef) and Postgres (barman endpointCA) have had
a CA field all along.

Add backup.endpointCA {name, key}. When set, the chart mounts that Secret into
the sidecar and points SSL_CERT_DIR at it. Empty (the default) renders no env, no
mount and no volume, so existing releases are untouched.

SSL_CERT_DIR rather than AWS_CA_BUNDLE, measured against
altinity/clickhouse-backup:2.7.4 with a self-signed endpoint and public AWS S3 in
turn: Go reads SSL_CERT_DIR in addition to the system bundle, so both verify;
AWS_CA_BUNDLE replaces the SDK trust pool, so the private endpoint verifies and
public S3 starts failing `x509: certificate signed by unknown authority`. That
would silently break a release that also talks to a public endpoint.
S3_DISABLE_CERT_VERIFICATION was rejected outright — it drops verification
instead of trusting one more CA.

Covered by helm-unittest: the unset default, the mount/env/volume wiring, a
custom key still projected as ca.crt, composition with useSystemBucket, and that
backup.enabled=false renders nothing.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-30 16:04:03 +04:00
Andrei KvapilandClaude c2f3e1f379 feat(apps): annotate schema fields with x-cozystack-options
Mark fields that reference cluster resources with the x-cozystack-options
vendor keyword so the dashboard renders them as dynamic dropdowns:
GPU, instanceType, instanceProfile, network (vm-instance), GPU/
instanceType (kubernetes nodeGroups), image (vm-disk), storagePool
(bucket), storageClass (all apps) and disks[].name (vm-instance). Drop
the stale static instanceProfile enum.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-06-02 16:33:57 +03:00
Andrey Kolkov f17ead3d26 feat(platform): add default backupclass
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-28 13:49:59 +04:00
Aleksei Sviridkin 5af5962edf feat(apps): mark stateful-app storageClass fields as immutable
Annotate storageClass as immutable across every stateful app whose
chart exposes one — clickhouse, foundationdb, harbor, http-cache,
kafka (Kafka + ZooKeeper), kubernetes (top-level only), mariadb,
mongodb, nats, openbao, opensearch, postgres, qdrant, rabbitmq,
redis, vm-disk. Without coverage across the catalogue the UI would
render storageClass read-only on some apps and editable on others —
a worse mental model than uniform behaviour.

The annotation lands at the schema layer (CEL XValidation rule
'self == oldSelf' on values.schema.json / types.go / embedded
openAPISchema in cozyrds/<app>.yaml). The cozystack aggregated
apiserver does not yet evaluate CEL on application-backed resources;
that is tracked at cozystack/cozystack#2657. The UI
(cozystack/cozystack-ui#6) consumes the marker today.

The PVC-pinning argument is identical for every chart: Kubernetes
fixes a PVC's storageClassName at creation time, so editing this
field on an existing resource never migrated data. The schema
annotation makes that contract explicit.

kubernetes.nodeGroups[name].storageClass is intentionally NOT
annotated: the field is optional and undefaulted, so a strict
'self == oldSelf' rule would block any future attempt to set it on
an existing node group. A richer ratcheting form is needed there —
see docs/storage-immutability.md.

A refactor in pkg/registry/apps/application/rest.go extracts the
inline schema-build path into buildSpecSchema so it can be
unit-tested. Three regression tests in rest_spec_schema_test.go pin
the contract that x-kubernetes-validations survives the
v1->internal->structural conversion (loaded from the actual MariaDB
ApplicationDefinition fixture on disk, not a hand-edited mock).

Each per-app README gains a one-line blockquote pointing to a new
docs/storage-immutability.md that explains the contract and which
consumers enforce it today.

Requires cozystack/cozyvalues-gen with @immutable support — bump the
pre-commit pin in .github/workflows/pre-commit.yml once the release
exists.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-26 23:51:11 +03:00
Aleksei SviridkinandMatthieu ROBIN fdffedbd4b feat(apps): instance-type resourcesPreset across apps and extras
Rewrite resourcesPreset values in every apps/* and extras/* chart from
the legacy flat names to the new instance-type form, using the 1:1
mapping:

  nano    -> t1.nano       micro   -> t1.micro
  small   -> t1.small      medium  -> c1.small
  large   -> c1.medium     xlarge  -> c1.large
  2xlarge -> c1.xlarge

Each affected values.yaml carries the canonical 47-value enum (40
instance-type names plus 7 legacy aliases) so the schema accepts both
forms during the deprecation window. cozyvalues-gen regenerates the
matching api/apps/v1alpha1/*/types.go kubebuilder enum and defaults,
the chart values.schema.json, the README parameter table, and the
ApplicationDefinition openAPISchema in packages/system/*-rd/cozyrds.

Three hardcoded legacy literals in chart templates migrated:
  - packages/apps/harbor/templates/harbor.yaml:121 (portal)
  - packages/apps/opensearch/templates/opensearch.yaml:43 (dashboards)
  - packages/system/keycloak/templates/sts.yaml:115 (heap fallback)

packages/apps/foundationdb/Makefile gains the missing hack/update-crd.sh
call. Without it, every regen left foundationdb-rd cozyrds stale and
new FoundationDB app CRs through cozystack-api were rejected at
validation for any non-legacy resourcesPreset value.

Every default-rendered chart produces identical CPU and memory to the
value applied before the rename, courtesy of the legacy alias table in
cozy-lib and the 1:1 default mapping.

Co-authored-by: Matthieu ROBIN <matthieu@hidora.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-11 21:08:41 +03:00
Andrey Kolkov 250e1b3376 feat(platform): implement backup strategy for clickhouse application
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-11 14:46:48 +04:00
Myasnikov Daniil 9e55552910 [docs] Updated app go types
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-03-25 15:57:25 +05:00
Andrei Kvapil 9632772337 [dx] JSDoc compatible syntax for values.yaml
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-10-29 08:57:26 +05:00
Andrei Kvapil 8c6fc68367 [cozystack-api] Update defaulting API schemas
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-09-18 04:49:50 +02:00
Nick Volynkin 2ef11ff513 [apps] Update application READMEs
- Remove duplicate values from rabbitmq README
- Use placeholders for passwords and secrets
- Fix copy-pasted postgres reference in mysql
- Fix links to cloud-init docs
- Explain CPU and memory consistently

Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
2025-08-18 10:29:31 +03:00
Nick Volynkin f864b40a85 [apps] Use new OpenAPI schema and README generator for packages/apps
- clickhouse
- ferretdb
- http-cache
- kafka
- kubernetes
- mysql
- nats
- rabbitmq
- redis
- tcp-balancer
- vm-disk
- vm-instance
- vpn

Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-08-11 12:59:50 +03:00
kklinch0 896209a004 [clickhouse] add clickhouse keeper
Signed-off-by: kklinch0 <kklinch0@gmail.com>
2025-08-07 14:07:05 +03:00
Andrei Kvapil 72e7b5e0b5 Get rid of bitnami's readme-generator
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-07-18 00:40:31 +02:00
Nick Volynkin bb907e5e7d [applications] Reorder values.yaml for better readability
Use the same order for values in all applications:

1. Common configuration parameters in the specified order, if exist:
   - replicas
   - shards
   - resources
   - resourcesPreset
   - size
   - storageClass
   - external (goes last, because we don't want to promote this practice)

2. Application-specific parameters, such as database and users
3. Component-specific, each component under its own section
4. Backup
5. Bootstrap (recovery)

Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
2025-07-17 19:36:20 +03:00
Nick Volynkin 55cfdb3a38 [apps] Remove preset 'none' from app charts and README
Preset 'none' is in fact disallowed since cozystack/cozystack#1156

Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
2025-07-14 19:31:10 +03:00
Andrei Kvapil 946fad8bb8 [apps] Give examples of new resources in managed app README's
- Change wording for `resources` and `resourcesPreset` variables.
- Explain and give exampls of other object-type variables,
  if their child fields are not annotated.
- Fix a few typos, improve wording.
- Bump all application charts to ensure that new texts are shown
  immediately after updating Cozystack.

Co-authored-by: Andrei Kvapil <kvapss@gmail.com>
Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-07-03 22:58:06 +03:00
Nick Volynkin cfc8c269f3 [apps] Add enum of allowed values to resourcePreset in all applications
It was present in some apps, such as managed kubernetes, but missing in others.

bitnami/readme-generator removes enums after re-generating README,
so now we patch them back using `yq` in Makefiles.

Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
2025-06-25 16:48:20 +03:00
Andrei Kvapil e8b83fbbda [clickhouse][kafka] fix openapispec generation
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-06-24 17:10:55 +02:00
Nick Volynkin 980185ca2b [docs] Review the Clickhouse app docs
Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
2025-06-16 08:40:46 +03:00
kklinch0 ea3b092128 feature/add-presets 2025-03-13 17:03:00 +03:00
klinch0andAndrei Kvapil 18bbdb675f delete extra logs, fix ch for cozy (#431)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced a local hook to generate a versions map and check for
changes.
- Added new configuration options for ClickHouse, including enhanced
logging, user management, and structured templates.
- New parameters for persistent volume size and Docker image
specifications in ClickHouse configuration.
- Added hostname matching capabilities in ClickHouse operator
configuration.

- **Bug Fixes**
	- Updated exclusion patterns for YAML files in pre-commit hooks.

- **Version Updates**
	- Incremented ClickHouse chart version from 0.5.0 to 0.6.0.
- Updated versioning for several packages, including `clickhouse` and
`tenant`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Andrei Kvapil <kvapss@gmail.com>
2024-10-21 11:54:08 +02:00
Andrei Kvapil 4b84798f00 Add backups for ClickHouse
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-09-15 06:52:21 +02:00
Andrei Kvapil c07c4bbdab Introduce stroageClass option for all applications (#290)
Provide the oportunity to specify StroageClass in applications

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-08-20 17:19:10 +02:00
Andrei Kvapil 4da8ac3b77 Add schema generation and remove default values (#110)
* Add schema generation and remove default values

* fix monitoring schema generation

* fix default values


Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-05-09 10:45:57 +02:00