> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hiloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Telemetry reliability

> How hiloop keeps accepted telemetry durable, the ingest acknowledgement contract, and how ingest behaves under overload, restarts, and failures.

hiloop treats telemetry as data you cannot afford to lose. When an event is accepted, it is
recoverable — an accepted event becomes queryable and stays queryable, across restarts, upgrades,
and unclean failures. This page explains the guarantees so you know what an acknowledgement means
and what to do when ingest pushes back.

## The acknowledgement contract

Every ingest path — [`hiloop run`](/guides/capturing-telemetry) capture, direct event ingest, and
[`hiloop annotations add`](/observability/annotations) — returns an `event_id` only **after** the event is
durably recorded. The acknowledgement is the guarantee, not a hint:

* **Accepted means durable.** By the time you hold an `event_id`, the event is on durable storage.
  A crash, restart, or upgrade immediately afterward does not lose it.
* **Accepted means queryable.** An accepted event is visible to [SQL queries](/observability/query-telemetry)
  and [live tail](/observability/live-tail) right away —
  you can read your own writes without waiting for a flush.
* **No acknowledgement means not accepted.** If a request fails, the event was not committed. It is
  safe to retry.

Because acknowledgement follows durability, "acked ⇒ eventually queryable" holds even in the worst
cases below.

## Retries are always safe

Every event carries a stable `event_id`. Ingest deduplicates on it, so re-sending an event that may
or may not have been accepted never creates a duplicate. Retry freely — the query surface shows each
event once.

## Backpressure, not silent loss

Under sustained overload hiloop protects durability by pushing back, never by quietly dropping
accepted data. When an ingest buffer is saturated, new requests are rejected **before** they are
acknowledged, with a retryable error:

| Surface     | What you see            |
| ----------- | ----------------------- |
| gRPC ingest | `RESOURCE_EXHAUSTED`    |
| HTTP ingest | `429 Too Many Requests` |

These are transient. The `hiloop` CLI and the capture interceptor treat them as spool-and-retry
signals and resend automatically, so a burst becomes a brief slowdown rather than a gap. A rejected
request was never accepted — nothing is lost, and nothing is acknowledged that later disappears.

## Surviving restarts, upgrades, and crashes

Accepted events are protected by two layers of durable storage: a write-ahead log the event lands in
before it is acknowledged, and a durable raw archive that holds every accepted record. Together they
make recovery robust across the failure modes that matter:

| Situation                                                     | What happens                                                                                                                                                                                                                                               |
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Planned upgrade or restart**                                | In-memory events are flushed to long-term storage and the log is checkpointed before shutdown, within the shutdown budget. The next process starts with nothing outstanding.                                                                               |
| **Unclean stop** (node loss, forced kill)                     | On restart, the write-ahead log is replayed so accepted-but-not-yet-flushed events are restored before the service accepts reads.                                                                                                                          |
| **Format change across an upgrade, or a damaged log segment** | If any log segment cannot be replayed, its records are recovered from the durable raw archive and re-added to the query surface — deduplicated by `event_id`. An accepted `event_id` stays queryable even across an upgrade combined with an unclean stop. |

The result: whatever accepted your event, and whatever happens to the process next, the `event_id`
you were handed remains queryable.

## Availability during rollouts

Deployments hand off between processes rather than taking ingest down. A rollout is a brief
handoff measured in seconds or less, not an outage, and events sent across the handoff are
deduplicated on `event_id` — so a retry during a deploy is safe and does not duplicate.

## What this means for you

* **Trust the `event_id`.** If you got one, the event is safe. Read-back verification will find it.
* **Retry on error.** A failed or throttled request was not accepted; resend it. Deduplication makes
  retries free of duplicates.
* **Let the client spool.** The CLI and interceptor already back off and resend on throttling — you
  rarely need to handle it yourself.

For where these events land and how to query them, see the [observability overview](/observability/overview)
and [query telemetry](/observability/query-telemetry). For backup and disaster-recovery ownership by
deployment model, see [backup and restore](/deployment/backup-restore).