> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hiloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Export telemetry

> Export filtered telemetry for offline analysis, audits, and warehouse workflows.

Use query output when you need to analyze runs outside hiloop, archive experiment evidence, or join
events with another system. Self-hosted deployments can additionally turn on a continuous
[forward sink](#continuous-forward-sink) that streams every event to an object store you own, an
[OTLP forward exporter](#otlp-collector-forward) that pushes the same stream to an OTLP collector,
or both.

There is no separate bulk export endpoint in the public API today. For public surfaces, export by
querying the rows you need and writing the response to a controlled destination.

## Continuous forward sink

Self-hosted deployments can continuously forward telemetry to an object store you control — for audit
retention, SIEM ingestion, or a data warehouse — without querying. When enabled, every batch of events
the gateway flushes to its own cold storage is **also** written to your bucket as newline-delimited
JSON (JSONL): one event per line.

* **Off by default, opt-in.** Set the forward URL to enable it (see the
  [Helm reference](/reference/helm/telemetry-gateway) `export.*` values); leave it empty to disable.
* **What lands.** One JSONL object per flushed batch, laid out by organization and date so you can retrieve a
  window without a full scan:

  ```text theme={null}
  <your-prefix>/org_id=<org>/dt=<YYYY-MM-DD>/events-<timestamp>-<seq>.jsonl
  ```

  Each line is one event carrying its `event_id`, `org_id`, `project_id`, `run_id`, `root_run_id`,
  `lineage_path`, and trace/span IDs — so you can correlate runs, sandboxes, and lineage on your side
  outside hiloop.
* **Redaction.** The sink applies no transformation and runs no redaction of its own. Events forwarded
  here carry exactly whatever scrubbing happened upstream, which for runs captured by `hiloop run` is
  the narrow capture-side pattern set, and for telemetry sent straight to the ingest endpoints is
  none at all. See [redaction](/observability/redaction) before pointing this at a destination with
  its own handling requirements.
* **Delivery is best-effort.** Forwarding runs off the flush path and never blocks ingest: if your
  bucket is briefly unreachable or slow, that batch is dropped (and logged) rather than retried, and
  ingest continues uninterrupted. Treat the sink as a near-real-time archive, not a guaranteed,
  gap-free ledger.
* **Tamper-evidence.** Point the sink at a bucket with **Object Lock** (WORM) enabled to keep an
  append-only, immutable audit trail; grant the sink least-privilege, write-only credentials.

<Note>
  The sink writes to any S3-compatible object store (AWS S3 or MinIO). Credentials for the sink bucket
  are configured separately from the events store, so the audit sink can use its own write-only
  credentials. See the [Helm reference](/reference/helm/telemetry-gateway) `export.*` values.
</Note>

## OTLP collector forward

Self-hosted deployments can also push the same continuous stream to an **OTLP collector** — for
SIEM and observability pipelines that speak OTLP (Splunk, Chronosphere, Grafana, or any
OpenTelemetry Collector). When enabled, every batch of events the gateway flushes to its own cold
storage is **also** exported over OTLP/gRPC as log records.

* **Off by default, opt-in.** Set the collector endpoint to enable it (see the
  [Helm reference](/reference/helm/telemetry-gateway) `otlpExport.*` values); leave it empty to
  disable. It is independent of the object-store sink — enable either or both.
* **What lands.** One `LogsService.Export` call per flushed batch. Each event becomes one log
  record whose timestamp is the event's causal wall clock and whose `event_name` is the event name;
  the record body carries the event's attributes as JSON, and the record attributes carry the
  correlation identity you join on: `hiloop.event.id`, `hiloop.signal`, `hiloop.project.id`, and —
  when the event belongs to a run — `hiloop.run.id` and `hiloop.lineage.path`. Events captured with
  trace context export their trace and span IDs natively, so trace-aware backends correlate without
  custom joins. The batch's resource carries `service.name` and `hiloop.org.id`.
* **Authentication.** Optional collector headers (for example `authorization=Bearer <token>`) are
  supplied from a secret and attached to every export call; they are never logged.
* **Redaction.** The exporter applies no transformation and runs no redaction of its own. Exported
  events carry exactly whatever scrubbing happened upstream, which for runs captured by `hiloop run`
  is the narrow capture-side pattern set, and for telemetry sent straight to the ingest endpoints is
  none at all. See [redaction](/observability/redaction) before pointing this at a destination with
  its own handling requirements.
* **Delivery is best-effort.** Exporting runs off the flush path and never blocks ingest: if the
  collector is unreachable, slow, or rejects records, that batch is dropped (and logged) rather
  than retried, and ingest continues uninterrupted. If you need buffering or retries, front the
  endpoint with an OpenTelemetry Collector you operate — the standard deployment shape for OTLP
  pipelines.

## Query API export

For small exports, run a `SELECT` and save the rows. The CLI builds the SQL from scoping flags and
`--output json` writes the raw `{"rows": […]}` body:

```sh theme={null}
hiloop query \
  --run-id "$RUN_ID" \
  --limit 1000 \
  --output json > telemetry.json
```

For an arbitrary projection, pass the SQL yourself with `--sql`, or `POST /v1/telemetry/sql` with a
`{"sql": "<SELECT …>"}` body.

For aggregate output, group in the `SELECT` — here over the `otel_genai_calls`
[data view](/observability/data-views), which derives one row per LLM exchange from the raw
captured payloads:

```sh theme={null}
hiloop query --sql "
  SELECT gen_ai_request_model,
         SUM(input_tokens)  AS sum_input_tokens,
         SUM(output_tokens) AS sum_output_tokens
  FROM otel_genai_calls
  WHERE run_id = '$RUN_ID'
  GROUP BY gen_ai_request_model
  ORDER BY sum_input_tokens DESC" \
  --output json > model-tokens.json
```

## Reproducibility bundle

A useful manual export bundle includes:

```text theme={null}
bundle/
  manifest.json
  sandbox.json
  operations.json
  executions.json
  artifacts-manifest.json
  telemetry.json
  model-tokens.json
```

`manifest.json` should record:

```json theme={null}
{
  "project_id": "67e55044-10b1-426f-9247-bb680e5fe0c8",
  "run_id": "01K6Z000000000000000000000",
  "sandbox_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "created_at": "2026-06-26T00:00:00Z",
  "image": {
    "reference": "ghcr.io/acme/agent-env:2026-06-26",
    "digest": "sha256:..."
  }
}
```

## Governance

Before export:

* confirm the destination is approved;
* apply your redaction policy;
* prefer digests and metadata when full payloads are not needed;
* record who requested the export;
* set retention on the exported object;
* encrypt exported files at rest and in transit.
