> ## Documentation Index > Fetch the complete documentation index at: https://docs.hiloop.ai/llms.txt > Use this file to discover all available pages before exploring further. # Helm deployment > Install hiloop services with Helm charts. hiloop charts package the deployable services and expose their configuration through values files. The generated Helm reference is the source of truth for exact values. ## Charts | Chart | Purpose | Enabled by default | | ------------------- | -------------------------------------------------------------------------- | ------------------ | | `hiloop` | Umbrella chart that composes the full platform below as one release. | (composition) | | `api` | Control-plane API. | Yes | | `authz` | Edge authorization service. | Yes | | `telemetry-gateway` | Telemetry ingest and query gateway. | Yes | | `web` | Web console. | Yes | | `model-gateway` | Operator-only legacy Rust proxy; not the supported sandbox gateway target. | Yes | | `operator` | Sandbox lifecycle controller. | Yes | | `agent-sandbox` | Sandbox workload controller (requires the platform CRDs below). | Yes | ## Custom resource definitions Helm does not install or upgrade CRDs, so the platform CRDs ship as a versioned `crds.yaml` alongside the chart. Apply it server-side before the first install and again before every upgrade (upgrades need `--force-conflicts`, because the running controller owns fields on the installed CRDs): ```sh theme={null} kubectl apply --server-side -f crds.yaml # first install kubectl apply --server-side --force-conflicts -f crds.yaml # before every upgrade ``` Re-applying at every upgrade keeps the CRDs and the controllers reconciling them at the same version. On uninstall, leave the CRDs in place unless you intend to delete every sandbox custom resource with them. ## Install pattern ```sh theme={null} helm install hiloop \ oci://REGISTRY.example.com/charts/hiloop \ --namespace hiloop \ --create-namespace \ -f values.yaml \ --wait --timeout 16m ``` The timeout deliberately exceeds the hook Jobs' own 15-minute deadline: Helm's `--timeout` bounds each Kubernetes operation including hooks, and it must not expire before the Job's own budget or Helm wedges the release while the Job was still within it. Use your deployment's registry, chart version, and values file. For air-gapped deployments, mirror the chart and image artifacts into your internal registry first. Avoid `--atomic` on a first install: hook resources are not release-managed, and a failed hook plus `--atomic` uninstalls the fresh release instead of leaving the failed hook Job and its logs in place to diagnose. The umbrella chart's own README (shipped in the chart, `helm show readme`) documents the full hook semantics, including the reserved hook-weight ranges each subchart owns. ## Migrations and upgrades Database migrations run as a blocking `pre-install,pre-upgrade` hook Job, always first (it holds the lowest hook weight in the release), before any workload updates. Migrations are expand/contract across the whole release: a release's schema changes stay compatible with the previous release's binaries, which is also what makes `helm rollback` safe (rollback never runs migrations down). A failed migration Job fails the release and stays in the namespace with its logs; fix the cause and re-run the same command: the Job is idempotent and re-created on the next attempt. If an upgrade wedges in `pending-upgrade` (for example a hook timeout), recover with `helm rollback`; the umbrella chart's README (shipped in the chart; `helm show readme`) carries the exact recovery recipe alongside the full hook semantics. ## First API key Self-hosted installs mint the deployment's first organization and API key at install time via a `post-install` hook Job, writing the key material only into the `hiloop-first-api-key` Secret, never to logs. `helm get notes hiloop -n hiloop` prints the retrieval command. Back that Secret up with your cluster backups: the cluster holds the only copy, and re-running install or upgrade never re-mints or rotates it. Bring your own key material instead by setting `api.firstKey.existingSecret`. ## Shared configuration Keep shared identity and routing values consistent across subcharts: * session issuer; * session audience; * API and telemetry hostnames; * JWKS location; * image registry and digests; * database and object storage references. Use the [Helm reference](/reference/helm/hiloop) for exact value names, types, and defaults. ## Telemetry storage and sizing The telemetry gateway keeps its durable datasets — the canonical event store and the raw-record archive — in an **object store**, configured with `telemetry-gateway.objectStore.url`. This value is **required**: an install or upgrade without it fails at render time with ``` objectStore.url is required: set the telemetry object-store URL, e.g. s3:///telemetry (or file:///data to keep telemetry on the local volume of a single-node install) ``` There is deliberately no silent local-disk fallback — a local volume puts a hard capacity ceiling on telemetry durability. Point it at a bucket (`s3://…`; S3 credentials come from the pod's ambient identity, not the chart), or set `file:///data` on a single-node install that knowingly keeps telemetry on the local volume — and size that volume for your full retention if you do. The gateway runs as a StatefulSet, and each replica keeps its own small persistent volume (`persistence.size`, default 5Gi, one claim per pod) holding **only the write-ahead log**. Its footprint tracks the un-flushed ingest backlog, not lifetime ingest, so the default is enough unless you expect sustained multi-gigabyte ingest bursts between flushes. Plan total WAL storage as `persistence.size` × `replicaCount`. ### High availability (active/standby) The sizing presets run one replica — the shape every install can run. Set `telemetry-gateway.replicaCount: 2` to run an **active/standby pair** instead: one active writer plus one warm standby, coordinated through a Kubernetes lease. Two replicas require a shared object store (`objectStore.url` pointing at a bucket, not `file://`) and the Postgres catalog (`catalog.enabled: true`) — the chart refuses to render without them. What you see with two replicas: * **Two Ready pods, each with its own write-ahead-log volume** — per-pod claims named `data--0` and `data--1`, not one shared volume. Each replica also gets the full sizing preset's CPU and memory. * **Clients reach only the active writer, through the Service.** The lease holder labels its own pod and the Service routes on that label — `kubectl get pods -l telemetry.hiloop.ai/writer=active` shows who is serving. A standby addressed directly answers every data request with a retryable "standing by" error (gRPC `UNAVAILABLE`, HTTP 503); that is by design — retry against the Service. * **Rollouts hand off instead of gapping.** The outgoing writer releases the lease on shutdown and keeps serving through a short drain window (`shutdownDrainWindowSeconds`) while the standby takes over, so the ingest interruption during an upgrade is on the order of a second rather than a full restart gap. If a node is lost, the standby takes over when the lease expires, within seconds; the dead pod's un-flushed write-ahead log is replayed from its own volume once that pod is rescheduled, so nothing acknowledged is dropped. Upgrading an install from the earlier single-volume topology converts the workload in place: the old release-named volume is removed by the upgrade itself, and the new pods create fresh per-pod volumes. The outgoing pod drains its write-ahead log to the object store before the volume is released — snapshot the volume first if you want a rollback point. Expect one final restart-style ingest gap while the first converted pod boots; rollouts after that use the handoff. ### Memory sizing Two application budgets bound gateway memory, and the sizing preset's memory limit must clear both with headroom: * `hotBufferBudgetMiB` (default 384) caps in-memory buffered ingest; * each in-flight query runs under a \~256 MiB engine budget. One full ingest buffer plus one worst-case query is therefore \~640 MiB before process overhead — the default `medium` preset (1Gi limit) is sized for exactly that. Raise `hotBufferBudgetMiB` in step with a larger preset; `small` (512Mi) fits light traffic only. ### Backpressure When buffered ingest reaches the budget (for example while the object store is unreachable), the gateway rejects **new** ingest with a retryable backpressure error — gRPC `RESOURCE_EXHAUSTED`, HTTP 429 — instead of growing memory without bound. Nothing already accepted is affected, and a rejected batch is never half-written. The capture wrapper and SDKs treat this error as retryable automatically: events are spooled and redelivered with backoff when the gateway recovers, so a backpressure window shows up as delayed delivery, not data loss. Sustained backpressure is an operator signal that flushing is stalled (object-store health) or the deployment is undersized. ### Request and payload limits * Ingest and other write requests cap at **32 MiB** per message; send larger batches through the streaming ingest RPC, which chunks frames well under the limit. * Query and read responses cap at **64 MiB** per message. * The run-events read (`GET /v1/telemetry/runs/{run_id}/events`) is **paginated**: each response returns at most one page (`limit` capped server-side) plus a `next` cursor to resume from, so arbitrarily large runs are read page by page. Telemetry retention is infinite by default; set `telemetry-gateway.retentionDays` to age out events older than a cutoff. ## Telemetry export sink The `telemetry-gateway` chart can continuously forward telemetry to an object store you own, as an opt-in audit/SIEM/warehouse feed. It is **off by default**; set `telemetry-gateway.export.url` to an `s3://` (or S3-compatible) bucket to enable it, and optionally point `export.credentialsSecretName` at a Secret carrying the sink's write-only credentials. Events are forwarded as newline-delimited JSON, partitioned by organization and date; delivery is best-effort and never blocks ingest. See the [export guide](/observability/export#continuous-forward-sink) for behavior and the [telemetry-gateway Helm reference](/reference/helm/telemetry-gateway) for the exact `export.*` values. ## Validate Before applying to a production environment: ```sh theme={null} helm template hiloop oci://REGISTRY.example.com/charts/hiloop -f values.yaml helm lint oci://REGISTRY.example.com/charts/hiloop -f values.yaml ``` Then smoke-test the edge path, telemetry ingest, telemetry query, and a bounded sandbox create, exec, and delete.