Skip to main content
hiloop charts package the deployable services and expose their configuration through values files. The generated Helm reference is the source of truth for exact values.

​
Charts

​
Custom resource definitions

Helm does not install or upgrade CRDs, so the platform CRDs ship as a versioned crds.yaml alongside the chart. Apply it server-side before the first install and again before every upgrade (upgrades need --force-conflicts, because the running controller owns fields on the installed CRDs):
Re-applying at every upgrade keeps the CRDs and the controllers reconciling them at the same version. On uninstall, leave the CRDs in place unless you intend to delete every sandbox custom resource with them.

​
Install pattern

The timeout deliberately exceeds the hook Jobs’ own 15-minute deadline: Helm’s --timeout bounds each Kubernetes operation including hooks, and it must not expire before the Job’s own budget or Helm wedges the release while the Job was still within it. Use your deployment’s registry, chart version, and values file. For air-gapped deployments, mirror the chart and image artifacts into your internal registry first. Avoid --atomic on a first install: hook resources are not release-managed, and a failed hook plus --atomic uninstalls the fresh release instead of leaving the failed hook Job and its logs in place to diagnose. The umbrella chart’s own README (shipped in the chart, helm show readme) documents the full hook semantics, including the reserved hook-weight ranges each subchart owns.

​
Migrations and upgrades

Database migrations run as a blocking pre-install,pre-upgrade hook Job, always first (it holds the lowest hook weight in the release), before any workload updates. Migrations are expand/contract across the whole release: a release’s schema changes stay compatible with the previous release’s binaries, which is also what makes helm rollback safe (rollback never runs migrations down). A failed migration Job fails the release and stays in the namespace with its logs; fix the cause and re-run the same command: the Job is idempotent and re-created on the next attempt. If an upgrade wedges in pending-upgrade (for example a hook timeout), recover with helm rollback; the umbrella chart’s README (shipped in the chart; helm show readme) carries the exact recovery recipe alongside the full hook semantics.

​
First API key

Self-hosted installs mint the deployment’s first organization and API key at install time via a post-install hook Job, writing the key material only into the hiloop-first-api-key Secret, never to logs. helm get notes hiloop -n hiloop prints the retrieval command. Back that Secret up with your cluster backups: the cluster holds the only copy, and re-running install or upgrade never re-mints or rotates it. Bring your own key material instead by setting api.firstKey.existingSecret.

​
Shared configuration

Keep shared identity and routing values consistent across subcharts:
  • session issuer;
  • session audience;
  • API and telemetry hostnames;
  • JWKS location;
  • image registry and digests;
  • database and object storage references.
Use the Helm reference for exact value names, types, and defaults.

​
Telemetry storage and sizing

The telemetry gateway keeps its durable datasets — the canonical event store and the raw-record archive — in an object store, configured with telemetry-gateway.objectStore.url. This value is required: an install or upgrade without it fails at render time with
There is deliberately no silent local-disk fallback — a local volume puts a hard capacity ceiling on telemetry durability. Point it at a bucket (s3://…; S3 credentials come from the pod’s ambient identity, not the chart), or set file:///data on a single-node install that knowingly keeps telemetry on the local volume — and size that volume for your full retention if you do. The gateway runs as a StatefulSet, and each replica keeps its own small persistent volume (persistence.size, default 5Gi, one claim per pod) holding only the write-ahead log. Its footprint tracks the un-flushed ingest backlog, not lifetime ingest, so the default is enough unless you expect sustained multi-gigabyte ingest bursts between flushes. Plan total WAL storage as persistence.size × replicaCount.

​
High availability (active/standby)

The sizing presets run one replica — the shape every install can run. Set telemetry-gateway.replicaCount: 2 to run an active/standby pair instead: one active writer plus one warm standby, coordinated through a Kubernetes lease. Two replicas require a shared object store (objectStore.url pointing at a bucket, not file://) and the Postgres catalog (catalog.enabled: true) — the chart refuses to render without them. What you see with two replicas:
  • Two Ready pods, each with its own write-ahead-log volume — per-pod claims named data-<name>-0 and data-<name>-1, not one shared volume. Each replica also gets the full sizing preset’s CPU and memory.
  • Clients reach only the active writer, through the Service. The lease holder labels its own pod and the Service routes on that label — kubectl get pods -l telemetry.hiloop.ai/writer=active shows who is serving. A standby addressed directly answers every data request with a retryable “standing by” error (gRPC UNAVAILABLE, HTTP 503); that is by design — retry against the Service.
  • Rollouts hand off instead of gapping. The outgoing writer releases the lease on shutdown and keeps serving through a short drain window (shutdownDrainWindowSeconds) while the standby takes over, so the ingest interruption during an upgrade is on the order of a second rather than a full restart gap. If a node is lost, the standby takes over when the lease expires, within seconds; the dead pod’s un-flushed write-ahead log is replayed from its own volume once that pod is rescheduled, so nothing acknowledged is dropped.
Upgrading an install from the earlier single-volume topology converts the workload in place: the old release-named volume is removed by the upgrade itself, and the new pods create fresh per-pod volumes. The outgoing pod drains its write-ahead log to the object store before the volume is released — snapshot the volume first if you want a rollback point. Expect one final restart-style ingest gap while the first converted pod boots; rollouts after that use the handoff.

​
Memory sizing

Two application budgets bound gateway memory, and the sizing preset’s memory limit must clear both with headroom:
  • hotBufferBudgetMiB (default 384) caps in-memory buffered ingest;
  • each in-flight query runs under a ~256 MiB engine budget.
One full ingest buffer plus one worst-case query is therefore ~640 MiB before process overhead — the default medium preset (1Gi limit) is sized for exactly that. Raise hotBufferBudgetMiB in step with a larger preset; small (512Mi) fits light traffic only.

​
Backpressure

When buffered ingest reaches the budget (for example while the object store is unreachable), the gateway rejects new ingest with a retryable backpressure error — gRPC RESOURCE_EXHAUSTED, HTTP 429 — instead of growing memory without bound. Nothing already accepted is affected, and a rejected batch is never half-written. The capture wrapper and SDKs treat this error as retryable automatically: events are spooled and redelivered with backoff when the gateway recovers, so a backpressure window shows up as delayed delivery, not data loss. Sustained backpressure is an operator signal that flushing is stalled (object-store health) or the deployment is undersized.

​
Request and payload limits

  • Ingest and other write requests cap at 32 MiB per message; send larger batches through the streaming ingest RPC, which chunks frames well under the limit.
  • Query and read responses cap at 64 MiB per message.
  • The run-events read (GET /v1/telemetry/runs/{run_id}/events) is paginated: each response returns at most one page (limit capped server-side) plus a next cursor to resume from, so arbitrarily large runs are read page by page.
Telemetry retention is infinite by default; set telemetry-gateway.retentionDays to age out events older than a cutoff.

​
Telemetry export sink

The telemetry-gateway chart can continuously forward telemetry to an object store you own, as an opt-in audit/SIEM/warehouse feed. It is off by default; set telemetry-gateway.export.url to an s3:// (or S3-compatible) bucket to enable it, and optionally point export.credentialsSecretName at a Secret carrying the sink’s write-only credentials. Events are forwarded as newline-delimited JSON, partitioned by organization and date; delivery is best-effort and never blocks ingest. See the export guide for behavior and the telemetry-gateway Helm reference for the exact export.* values.

​
Validate

Before applying to a production environment:
Then smoke-test the edge path, telemetry ingest, telemetry query, and a bounded sandbox create, exec, and delete.