> ## Documentation Index > Fetch the complete documentation index at: https://docs.hiloop.ai/llms.txt > Use this file to discover all available pages before exploring further. # telemetry-gateway Helm chart > hiloop telemetry ingest + query gateway hiloop telemetry ingest + query gateway. Source chart version `0.5.0`. Published OCI artifacts use the immutable version from the deployment release record: ```bash theme={null} helm install telemetry-gateway oci:///charts/telemetry-gateway \ --version "${PUBLISHED_CHART_VERSION:?set to the immutable OCI version for this release}" ``` ## Values Keys marked `*` are required. Defaults are the chart's `values.yaml`; per-environment overlays (`values-onprem.yaml`, `values-saas.yaml`) and the deployment set the rest. | Key | Type | Default | Description | | ---------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `affinity` | `object` | `{"podAntiAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": [{"topologyKey": "kubernetes.io/hostname", "labelSelector": {"matchLabels": {"app.kubernetes.io/name": "telemetry-gateway"}}}]}}` | Pod affinity/anti-affinity scheduling rules. | | `catalog` | `object` | — | Postgres catalog (FDAP parquet-file index). When enabled, the gateway uses the database catalog pool (selected by env) instead of the in-memory catalog. Default off so non-SaaS installs keep the in-memory catalog. | | `catalog.auth` | `password` \| `iam` | `"password"` | Credential mechanism. password = static DSN from a Secret (local/on-prem); iam = RDS IAM SigV4 token per connection (cloud). | | `catalog.databaseUrlSecretName` | `string` | `"telemetry-catalog-dsn"` | password auth only: name of the Secret holding the catalog DSN under key TELEMETRY\_DATABASE\_URL (synced by ESO from Secrets Manager). (non-empty) | | `catalog.enabled` | `boolean` | `false` | Use the Postgres catalog instead of the in-memory catalog. | | `containerSecurityContext` | `object` | `{"allowPrivilegeEscalation": false, "readOnlyRootFilesystem": true, "capabilities": {"drop": ["ALL"]}, "seccompProfile": {"type": "RuntimeDefault"}}` | Container-level securityContext (Kubernetes SecurityContext). Hardening such as readOnlyRootFilesystem and dropped capabilities. | | `coreDb` | `object` | — | Core-database access: a second, least-privilege connection distinct from the telemetry catalog. It validates annotation payloads, resolves fresh run anchors, and publishes durable sandbox-capture completion records. When disabled, annotation validation is structural-only and sandbox capture cannot produce a durable terminal receipt. | | `coreDb.auth` | `password` \| `iam` | `"password"` | Credential mechanism. password = static least-privilege DSN from a Secret (local/on-prem); iam = RDS IAM SigV4 token per connection (cloud). | | `coreDb.databaseUrlSecretName` | `string` | `"telemetry-core-dsn"` | password auth only: name of the Secret holding the least-privilege core DSN under key TELEMETRY\_CORE\_DATABASE\_URL. (non-empty) | | `coreDb.enabled` | `boolean` | `false` | Open the least-privilege core-database connection for annotation validation, run-anchor resolution, and sandbox-capture completion publication. | | `disruptionClass` | `edge-critical` \| `data-path` \| `singleton-controller` \| `node-infra` \| `disposable` | `"singleton-controller"` | Availability class this workload declares; the chart derives the PodDisruptionBudget and scheduling priority from it. The gateway is a lease-elected single writer, so it ships as singleton-controller: no disruption budget at 1 replica (it could only wedge node drains), at most one pod down at a time for the 2-replica active/standby pair. | | `disruptionOverrides` | `object` | — | Explicit, justified deviations from the declared disruption class. | | `disruptionOverrides.justification` | `string` | `""` | Reason for deviating from the class's availability posture. Non-empty allows the reduced posture and is stamped on the workload as the hiloop.ai/disruption-override annotation. | | `env` | `local` \| `test_e2e` \| `dev` \| `staging` \| `production` \| `onprem` | `"local"` | Deployment profile (HILOOP\_ENV): selects the catalog credential mechanism (offline DSN vs. cloud RDS IAM) and the provider seam. | | `export` | `object` | — | Opt-in audit/SIEM forward sink (HILOOP\_TELEMETRY\_EXPORT\_URL). When url is set, every batch of canonical events the gateway flushes to cold storage is also written as newline-delimited JSON to this customer-owned object store, partitioned by organization + date, for your own audit/SIEM/warehouse pipelines (point it at an Object-Lock bucket for tamper-evident retention). Off by default. Delivery is best-effort off the flush path — a sink outage never blocks ingest. The sink adds no redaction and the gateway has no redaction stage of its own, so forwarded events carry only whatever scrubbing their producer applied: capture-side body and environment scrubbing for runs captured by hiloop run, and none at all for telemetry sent directly to the OTLP, annotation, or bulk ingest endpoints. | | `export.allowHttp` | `boolean` | `false` | Allow plaintext HTTP to the endpoint (MinIO / non-TLS dev only). Keep false for real S3 over TLS. | | `export.credentialsSecretName` | `string` | `""` | Optional Secret holding the sink's S3 credentials under keys AWS\_ACCESS\_KEY\_ID / AWS\_SECRET\_ACCESS\_KEY. Empty = use the pod's ambient AWS credentials. | | `export.endpoint` | `string` | `""` | S3-compatible endpoint for the sink bucket (e.g. a MinIO URL). Empty = AWS S3. | | `export.region` | `string` | `""` | Region of the sink bucket (e.g. us-east-1). Empty = inferred from the ambient AWS config. | | `export.url` | `string` | `""` | Object-store URL for the forward sink (e.g. s3://\/\). Empty disables the sink. Point at an Object-Lock bucket for tamper-evident retention. | | `global` | `object` | — | Cross-chart settings shared by every image in this release. | | `global.imagePullSecrets` | `array` | `[]` | imagePullSecrets applied to the pod for private/mirrored registries, e.g. \[\{ name: acme-registry-creds }]. | | `global.imageRegistry` | `string` | `""` | Registry host prepended to every image (e.g. registry.internal.acme.corp). Empty = use the registry baked into image.repository. Set once for air-gapped relocation. | | `hotBufferBudgetMiB` | `integer` | `384` | In-memory ingest buffer budget (MiB). Once this many MiB of events are buffered awaiting flush to storage, new ingest requests are rejected with a retryable backpressure error (gRPC RESOURCE\_EXHAUSTED / HTTP 429) instead of growing memory without bound; already-accepted events are unaffected. Size it so budget + the \~256MiB per-query engine budget fits under the pod's memory limit with headroom — the 384MiB default matches the medium preset's 1Gi limit; raise it in step with a larger sizing preset. (min 1) | | `image` | `object` | — | Container image for the telemetry-gateway. | | `image.digest` | `string` | `""` | Image digest (sha256:...). Preferred in prod for immutability; takes precedence over tag. | | `image.pullPolicy` | `Always` \| `IfNotPresent` \| `Never` | `"IfNotPresent"` | Kubelet image pull policy. | | `image.repository` \* | `string` | `"ACCOUNT.dkr.ecr.REGION.amazonaws.com/hiloop/telemetry-gateway"` | Image repository (without registry host when global.imageRegistry is set). (non-empty) | | `image.tag` | `string` | `""` | Image tag. Fallback when digest is empty; CI sets one of tag/digest. | | `internalRoute` | `object` | — | East-west ingest exposure via an internal (ClusterIP-only) Gateway listener: a second GRPCRoute for in-cluster callers, authenticated by the same SecurityPolicy as the public route, on a data plane that forwards W3C trace context so caller and ingest spans share one trace. Default off. | | `internalRoute.enabled` | `boolean` | `false` | Render the internal GRPCRoute attaching the ingest surface to the internal Gateway. Requires hostname. | | `internalRoute.hostname` | `string` | `""` | In-cluster Host header the internal listener matches (e.g. telemetry-internal.envoy-gateway-system.svc.cluster.local); required when enabled. Not a public DNS name. | | `internalRoute.parentRef` | `object` | — | Internal Gateway listener the GRPCRoute attaches to. | | `internalRoute.parentRef.name` | `string` | `"eg-internal"` | Internal Gateway resource name (e.g. eg-internal). (non-empty) | | `internalRoute.parentRef.namespace` | `string` | `"envoy-gateway-system"` | Namespace of the internal Gateway. (non-empty) | | `internalRoute.parentRef.sectionName` | `string` | `"http"` | Internal Gateway listener (section) name to attach to. (non-empty) | | `networkPolicy` | `object` | — | Direct-service trust fence. The chart always renders an ingress NetworkPolicy restricting ClusterIP access to the ingest and query listeners, because the gateway derives the caller's organization from identity headers its edge stamps and cannot tell a forged one from a stamped one. | | `networkPolicy.ingressFrom` | `array` | `[]` | Additional peers permitted to reach the ingest and query listeners, beyond the Envoy data planes of the enabled routes and the exact API workload. Deployments whose edge this chart does not manage (self-hosted installs behind the operator's own ingress controller) must declare that edge here; the onprem profile fails to render while it is empty. | | `nodeSelector` | `object` | `{}` | Node selector constraining pod scheduling to matching nodes. | | `objectStore` | `object` | — | Object store for the telemetry datasets — canonical events (Parquet) and the bronze raw-record archive. AWS S3 can use ambient workload identity; customer-operated S3-compatible stores use an explicit endpoint plus Secret-backed credentials and a private CA. | | `objectStore.allowHttp` | `boolean` | `false` | Allow a plaintext HTTP endpoint for local development. Customer-operated on-prem installs require HTTPS. | | `objectStore.credentialsSecretName` | `string` | `""` | Secret containing ACCESS\_KEY\_ID, SECRET\_ACCESS\_KEY, and optional SESSION\_TOKEN keys. The chart projects them with the AWS\_ prefix; empty uses ambient credentials. | | `objectStore.endpoint` | `string` | `""` | S3-compatible API authority with no bucket or path (for example, [https://minio.storage.svc:9000](https://minio.storage.svc:9000)). Empty uses AWS S3. | | `objectStore.region` | `string` | `""` | Signing region for S3-compatible requests. Required when endpoint is set. | | `objectStore.trustBundle` | `object` | — | Private CA bundle for an HTTPS S3-compatible endpoint. | | `objectStore.trustBundle.existingSecret` | `string` | `""` | Existing versioned Secret containing the endpoint CA. | | `objectStore.trustBundle.key` | `string` | `"ca.crt"` | Key in the trust-bundle Secret containing the PEM CA certificate. (non-empty) | | `objectStore.url` | `string` | `""` | Object-store URL for the telemetry datasets (e.g. s3://\/telemetry, or file:///data to keep telemetry on the local volume of a single-node install). Required: install/upgrade fails when unset. Pairs with catalog.\*. | | `observability` | `object` | — | OTLP telemetry export settings for the service binary. | | `observability.enabled` | `boolean` | `true` | Emit OTLP telemetry and set the OTEL\_\* env vars. | | `observability.environment` | `string` | `"local"` | Deployment environment label (HILOOP\_ENVIRONMENT), e.g. production. (non-empty) | | `observability.otlpEndpoint` | `string` | `"http://otel-collector-agent.monitoring.svc.cluster.local:4317"` | OTLP gRPC collector endpoint (OTEL\_EXPORTER\_OTLP\_ENDPOINT). (non-empty) | | `observability.resourceAttributes` | `string` | `"service.namespace=hiloop"` | Extra OTEL resource attributes (OTEL\_RESOURCE\_ATTRIBUTES), e.g. service.namespace=hiloop. | | `observability.rustLog` | `string` | `"info"` | RUST\_LOG filter directive controlling log verbosity. (non-empty) | | `observability.serviceName` | `string` | `"telemetry-gateway"` | OTEL service name (OTEL\_SERVICE\_NAME). (non-empty) | | `otlpExport` | `object` | — | Opt-in OTLP forward exporter (HILOOP\_TELEMETRY\_OTLP\_EXPORT\_ENDPOINT). When endpoint is set, every batch of canonical events the gateway flushes to cold storage is also pushed to this OTLP/gRPC collector as log records carrying the event/run/lineage/organization correlation attributes — independent of, and combinable with, the object-store forward sink. Off by default. Delivery is best-effort off the flush path — a collector outage never blocks ingest, and there is no retry queue (front the endpoint with a buffering OTel collector if needed). The exporter adds no redaction and the gateway has no redaction stage of its own, so exported events carry only whatever scrubbing their producer applied: capture-side body and environment scrubbing for runs captured by hiloop run, and none at all for telemetry sent directly to the OTLP, annotation, or bulk ingest endpoints. | | `otlpExport.endpoint` | `string` | `""` | The collector's OTLP/gRPC endpoint (e.g. [https://collector.example:4317](https://collector.example:4317), or http\:// inside a private network). Empty disables the exporter. | | `otlpExport.headersSecretName` | `string` | `""` | Optional Secret holding collector auth headers under the key OTLP\_EXPORT\_HEADERS, as comma-separated name=value pairs (e.g. authorization=Bearer \). Attached to every export call as gRPC metadata; never logged. Empty = no auth headers. | | `persistence` | `object` | — | Local PVC for the data dir (HILOOP\_TELEMETRY\_DATA\_DIR), which holds only the write-ahead log. One ReadWriteOnce volume per pod (StatefulSet volumeClaimTemplates): each replica owns its WAL and a replacement pod remounts and replays its own volume. | | `persistence.enabled` | `boolean` | `true` | Provision a per-pod PVC for the data dir. When false, /data uses an ephemeral emptyDir. | | `persistence.size` | `string` | `"5Gi"` | Requested volume size (e.g. 5Gi). The write-ahead log's footprint tracks the un-flushed ingest backlog, not lifetime ingest. | | `persistence.storageClass` | `string` | `""` | StorageClass for the PVC. Empty = cluster default. | | `podSecurityContext` | `object` | `{"runAsNonRoot": true, "runAsUser": 65532, "fsGroup": 65532}` | Pod-level securityContext (Kubernetes PodSecurityContext). Keeps the pod non-root for the distroless image. | | `probes` | `object` | — | Startup/readiness/liveness probe tuning. Readiness is an HTTP GET /health on the query port; startup and liveness are tcpSocket checks on the gRPC ingest port. | | `probes.enabled` | `boolean` | `true` | Render startup, readiness, and liveness probes. | | `probes.initialDelaySeconds` | `integer` | `5` | Delay before the first readiness/liveness probe after startup succeeds. (min 0) | | `probes.periodSeconds` | `integer` | `10` | How often (seconds) to run the readiness and liveness probes. (min 1) | | `probes.startup` | `object` | — | Startup probe budget. The gRPC port binds only after boot completes — including full write-ahead-log replay, which can take minutes after heavy traffic — and liveness/readiness begin only once this probe succeeds, so a long replay is never restart-killed mid-recovery. | | `probes.startup.failureThreshold` | `integer` | `60` | Startup probe failures tolerated before the container is restarted. The total startup budget is periodSeconds x failureThreshold; keep it generous enough for a worst-case write-ahead-log replay. (min 1) | | `probes.startup.periodSeconds` | `integer` | `10` | How often (seconds) to run the startup probe. (min 1) | | `replicaCount` | `integer` | — | Override the pod count from the sizing preset (presets default to 1). Set 2 for the highly-available pair — one active writer plus one warm standby behind the writer lease (extra replicas are additional standbys); the hosted profile runs 2. More than one replica requires a shared object store (not file://) and the Postgres catalog — the chart refuses to render otherwise. (min 1) | | `resources` | `object` | — | Override the container resource requests/limits from the sizing preset. Set this for full control (e.g. very large deployments). | | `resources.limits` | `object` | — | Hard CPU/memory caps; exceeding memory triggers an OOM kill. | | `resources.requests` | `object` | — | Minimum CPU/memory the scheduler reserves. | | `retentionDays` | `integer` \| `null` | `null` | Age out telemetry older than this many days: a file whose newest event has passed the cutoff is reclaimed after the compactor grace window. null / unset = infinite retention (events are never aged out). (min 1) | | `route` | `object` | — | Gateway API GRPCRoute (SaaS exposure on clusters running Envoy Gateway). When enabled, attaches to the shared Gateway on hostname. Pair with securityPolicy.\* to authenticate it. | | `route.enabled` | `boolean` | `false` | Render a GRPCRoute. Requires hostname. Default off so non-mesh/on-prem installs stay internal. | | `route.hostname` | `string` | `""` | External hostname the route serves (e.g. telemetry.staging.hiloop.ai); external-dns mints the DNS record from it. Required when enabled. | | `route.parentRef` | `object` | — | The Gateway listener this route attaches to. | | `route.parentRef.name` | `string` | `"eg"` | Gateway resource name (e.g. eg). (non-empty) | | `route.parentRef.namespace` | `string` | `"envoy-gateway-system"` | Gateway namespace (e.g. envoy-gateway-system). (non-empty) | | `route.parentRef.sectionName` | `string` | `"https"` | Gateway listener section to bind (e.g. https). (non-empty) | | `route.timeouts` | `object` | — | Edge timeout budgets, rendered as one BackendTrafficPolicy per route rule (GRPCRoute itself has no timeouts field). | | `route.timeouts.request` | `string` | `"35s"` | Request timeout granted to the bounded unary RPCs. Size it strictly above the gateway's read-class budget so the server's typed deadline error reaches the client before the edge cancels. | | `route.timeouts.streamIdle` | `string` | `"0s"` | Stream idle timeout over the long-lived streaming RPCs. 0s disables it; dead peers are reaped by TCP keepalives and HTTP/2 pings instead. | | `route.timeouts.streamRequest` | `string` | `"0s"` | Request timeout over the long-lived streaming RPCs (IngestStream, UploadBlob, Tail). 0s disables it: these streams are legitimately open for hours and carry no in-stream heartbeat. | | `securityPolicy` | `object` | — | Envoy Gateway SecurityPolicy (ext\_authz) on the GRPCRoutes: authz injects trusted identity before routing. | | `securityPolicy.apiPeer` | `object` | — | Exact API workload identity permitted to call the telemetry read/query listeners directly. Read by the always-rendered trusted-ingress NetworkPolicy regardless of securityPolicy.enabled. | | `securityPolicy.apiPeer.instance` \* | `string` | `""` | Exact app.kubernetes.io/instance label of the API pods. Empty resolves to the current Helm release; override when API is a separate release. | | `securityPolicy.apiPeer.name` \* | `string` | `"api"` | Exact app.kubernetes.io/name label of the API pods. (non-empty) | | `securityPolicy.apiPeer.namespace` \* | `string` | `""` | Namespace containing the API pods. Empty resolves to the telemetry release namespace. | | `securityPolicy.authz` | `object` | — | The ext\_authz HTTP target (the authz service). | | `securityPolicy.authz.namespace` | `string` | `""` | authz Service namespace. Empty = the release namespace. | | `securityPolicy.authz.port` | `integer` | `8080` | authz ext\_authz HTTP port. (min 1, max 65535) | | `securityPolicy.authz.service` | `string` | `"authz-authz"` | authz Service name (e.g. authz-authz). (non-empty) | | `securityPolicy.enabled` | `boolean` | `false` | Render the SecurityPolicy and run the gateway with --trust-mesh-org. Fails closed (no/invalid token yields 401). | | `service` | `object` | — | Kubernetes Service fronting the gRPC ingest port and the HTTP query surface. | | `service.port` | `integer` | `50051` | gRPC ingest port; the container binds and the Service exposes this. (min 1, max 65535) | | `service.queryHttpPort` | `integer` | `50052` | Safe-SQL HTTP query surface port; the container binds 0.0.0.0:\ and the Service exposes it cluster-internally for the API edge reverse-proxy. (min 1, max 65535) | | `service.type` | `ClusterIP` \| `NodePort` \| `LoadBalancer` | `"ClusterIP"` | Service type. ClusterIP for in-cluster/Gateway-fronted; LoadBalancer/NodePort for direct external exposure. | | `serviceAccount` | `object` | — | ServiceAccount for the pod. | | `serviceAccount.annotations` | `object` | `{}` | Annotations on the ServiceAccount (e.g. EKS Pod Identity association). Usually empty on-prem. | | `serviceAccount.create` | `boolean` | `true` | Create a ServiceAccount. When false, serviceAccount.name is required — the chart binds its Roles to that name. | | `serviceAccount.name` | `string` | `""` | Name of the ServiceAccount. With create=true it overrides the generated name; with create=false it is REQUIRED and names the existing ServiceAccount to reuse, because the chart's RoleBindings are rendered against it. | | `shutdownDrainWindowSeconds` | `integer` | `5` | Rollout drain window (seconds). On SIGTERM the gateway fails its readiness probe immediately but keeps serving new and in-flight requests for this long, so the fronting proxy removes the endpoint before the listeners stop accepting; it then finishes in-flight requests and flushes buffered telemetry before exit. (min 0) | | `sizing` | `string` | `""` | T-shirt size for this service: small \| medium \| large \| xlarge. Usually set once for the whole platform via the umbrella's global.sizing; set here only to size this service differently. Empty inherits global.sizing (default medium). | | `terminationGracePeriodSeconds` | `integer` | `60` | Graceful-shutdown budget (seconds). On SIGTERM the gateway serves through the drain window, lets in-flight requests finish, then drains its hot buffer and checkpoints the write-ahead log before exit; keep this above shutdownDrainWindowSeconds plus worst-case flush-to-store latency. (min 1) | | `test` | `object` | — | Post-reconcile smoke run by `helm test`. When enabled, renders a Helm test-hook Job that drives an authenticated round trip (identity echo then a telemetry query) through the public API edge as the configured test organization. The Job reads an existing in-cluster Secret directly. Default off: needs a reachable public edge and a provisioned test-tenant key. | | `test.apiBaseUrl` | `string` | `""` | Public base URL of the API edge the smoke calls (e.g. [https://api.example.com](https://api.example.com)). Required when enabled. | | `test.apiKeySecret` | `object` | — | Existing in-cluster Secret containing the organization-scoped API key. | | `test.apiKeySecret.key` \* | `string` | `"api-key"` | Data key containing the API key. (non-empty) | | `test.apiKeySecret.name` \* | `string` | `""` | Secret name. | | `test.enabled` | `boolean` | `false` | Render the test-hook Job. Requires apiBaseUrl and apiKeySecret. | | `test.image` | `object` | — | Container image for the test Job (a digest-pinned curl image). | | `test.image.digest` | `string` | `"sha256:7c12af72ceb38b7432ab85e1a265cff6ae58e06f95539d539b654f2cfa64bb13"` | Image digest (sha256:...). Pinned by manifest-list digest so the image can't drift; bump to upgrade curl. (non-empty) | | `test.image.repository` | `string` | `"curlimages/curl"` | Image repository (a public, multi-arch curl image). (non-empty) | | `test.resources` | `object` | — | Resource requests/limits for the test Job container. | | `test.resources.limits` | `object` | `{"memory": "64Mi"}` | Hard CPU/memory caps. | | `test.resources.requests` | `object` | `{"cpu": "10m", "memory": "16Mi"}` | Minimum CPU/memory the scheduler reserves. | | `tolerations` | `array` | `[]` | Pod tolerations allowing scheduling onto tainted nodes. | | `workerThreadStackBytes` | `integer` | `16777216` | Stack size in bytes for the gateway's runtime worker threads (exported as RUST\_MIN\_STACK). 0 uses the runtime's built-in default. (min 0) |