Annotations - hiloop

An annotation is a structured judgment about a run: an eval score, a pass/fail label, a reviewer’s note. Annotations are events like any other — they live on the same run identity and are queryable alongside the LLM calls, tool calls, and logs they describe. That means you can filter and aggregate on a human or model judgment exactly the way you query the rest of a run. An annotation’s payload is entirely yours — there are no built-in fields. You decide what an annotation carries (a score, a verdict, an annotator, a free-text note, a nested object) by registering a schema for it. You write annotations from the CLI or SDK (shown below) and read them back with a telemetry query. You annotate two things:

A point — a single event (target_event_id), or the run as a whole when you omit the target.
A range — a [start, end] wall-clock window within a run.

Every annotation names a schema that its payload is validated against at ingest, so the data stays well-shaped enough to query.

Register a schema first

A schema is a named, versioned JSON Schema (draft 2020-12) that ingest validates annotation payloads against. Register one before you annotate against it:

hiloop annotation-schema register eval.quality \
  --json-schema @eval-quality.schema.json

Schemas are immutable and versioned per organization:

Names are organization-global, not project-local — every project and workstream in the organization shares one namespace. Namespace your schema names per workstream (kaggle.experiment.v1, not experiment.v1): re-registering a name another workstream already uses doesn’t error — it silently creates that schema’s next version for everyone reading it.
An unseen name starts at version 1.
Re-registering an existing name creates the next version — after a backward-compatibility check. An incompatible change is rejected rather than silently breaking existing annotations.
hiloop annotation-schema archive eval.quality retires a version without deleting it; annotations already stamped against it stay valid.
hiloop annotation-schema list shows the latest live version per name (add --include-archived for every version); get <name> [--version N] fetches one.

Promote the fields you query

Any field stays queryable from the JSON payload. To make a field fast to filter, sort, or join on across many runs, promote it into a typed column at register time with --promote field:type[:identity][:bloom] (type is str / f64 / i64 / bool):

hiloop annotation-schema register eval.quality \
  --json-schema @eval-quality.schema.json \
  --promote score:f64 \
  --promote outcome:str \
  --promote annotator:str:identity

A promoted field gets full columnar acceleration (statistics, page-index pruning, predicate pushdown) for the “compute once, filter forever” query — without baking any field into the platform. Unpromoted fields stay queryable from the payload, just without the column speedup.
:identity marks a field part of the latest-wins key (see Re-scoring). Mark an annotator field identity to keep the latest write per annotator. Distinct records need an identity field: without one, every write to the same target supersedes the last, so rows you meant to coexist (one per annotator, lane, or segment) collapse to the newest on the default read. Registration warns when a schema promotes fields but declares no identity field.
:bloom adds a point-lookup index (string fields only) — useful for a high-cardinality promoted id you look up by exact match.
Registration also auto-creates the schema-named views (here ann_eval_quality, plus its full-history sibling ann_eval_quality_history) that surface each promoted field under its declared name — see Query annotations back.

Annotate a run or an event

Attach a judgment to a run — or to one event within it with --target-event. There are two ways in, and they write the same annotation event:

The CLI (hiloop annotations add) sends through the telemetry ingest path — the ergonomic option.
The Annotate API is the typed call you make from your code.

Everything you want the annotation to carry goes in --data; there are no special value flags. The payload is used verbatim — nested objects and arrays are preserved — and --data accepts inline JSON, @file, or - for stdin:

hiloop annotations add \
  --run 01K6Z… \
  --schema eval.quality \
  --target-event 01K71… \
  --data '{"score":0.2,"outcome":"fail","annotator":"human","note":"hallucinated the API name"}'

A structured payload — say a result with a nested metric{} block and an array of evidence event ids — goes through the same flag:

hiloop annotations add \
  --run 01K6Z… \
  --schema experiment.idea \
  --data '{
    "annotator": "llm",
    "score": 52.4,
    "outcome": "worked",
    "headline": "Add interaction + ratio features",
    "hypothesis": "Pairwise ratios expose nonlinearity a linear model can use directly.",
    "metric": { "name": "rmse", "value": 52.4, "direction": "lower_better", "valid": true },
    "evidence_event_ids": ["01HQZX1FEAT0CHANGE", "01HQZX1FEAT0METRIC"]
  }'

If the schema promotes score/outcome/annotator, those values lift into typed columns for fast filter/sort while staying in the payload. Both examples anchor the annotation at the run you pass as --run — an annotation always belongs to exactly one run and inherits that run’s logical lineage position. The sandbox runtime does not inject a sandbox credential or HILOOP_RUN_ID; annotate from an authenticated external client. --output json prints the minted {"event_id": …}. By default the server mints a fresh event_id per invocation, so re-running the command writes a new annotation. To make a retry safe after an ambiguous failure (a 5xx, a lost response), mint the id yourself with --event-id <ULID>: re-running with the same id returns the stored annotation instead of writing a duplicate. The id names this logical annotation — never reuse it for different content. The same annotation through the API, which anchors by run_id:

import { annotationServiceAnnotate } from "@hiloopai/sdk";

const { data } = await annotationServiceAnnotate({
  client,
  body: {
    run_id: "01K6Z…",
    schema_name: "eval.quality",
    target_event_id: "01K71…",
    payload_json: JSON.stringify({
      score: 0.2,
      outcome: "fail",
      annotator: "human",
      note: "hallucinated the API name",
    }),
  },
});
console.log(data?.event_id);

import json
from hiloop.api.annotation_service import annotation_service_annotate
from hiloop.models import AnnotateRequest

resp = annotation_service_annotate.sync(
    client=client,
    body=AnnotateRequest(
        run_id="01K6Z…",
        schema_name="eval.quality",
        target_event_id="01K71…",
        payload_json=json.dumps(
            {"score": 0.2, "outcome": "fail", "annotator": "human", "note": "hallucinated the API name"}
        ),
    ),
)
print(resp.event_id)

The response carries the minted event_id — the annotation’s stable dedup and lookup key. The Annotate request body is:

Field	Meaning
`run_id`	The run the annotation belongs to. Exactly one of `run_id` / `project_id`.
`project_id`	The project a run-less annotation belongs to (see Annotate a project).
`schema_name`	The registered schema the payload validates against; becomes the event `name`.
`target_event_id`	The event this annotation is about. Omit for a run/session-level annotation. Run targets only.
`payload_json`	The annotation body — a JSON object (scalars, nested objects, and arrays), stored verbatim as the event’s `attributes_json` and validated against the schema. Entirely yours.

Annotate a range

A range annotation targets a window instead of a single event — useful for “this whole retry loop was slow” or “the agent went off-task between these timestamps”. Pass --range <start>..<end>, where both endpoints are times — an RFC 3339 timestamp or a raw wall-clock nanosecond value (exactly what a query’s ts_wall_ns column returns) — or both are event ids, in which case the window spans those two events’ recorded timestamps:

hiloop annotations add \
  --run 01K6Z… \
  --schema review.span \
  --range 2026-07-03T10:14:22Z..2026-07-03T10:15:40Z \
  --data '{"label":"off-task"}'

# The same window, bounded by the first and last event of the loop under review:
hiloop annotations add \
  --run 01K6Z… \
  --schema review.span \
  --range 01K71DXPZ3TQJ8FJQ4RG9BC2NM..01K71E2M4YV0S8BW3T0V0V5H8K \
  --data '{"label":"off-task"}'

An event-bounded range keeps the endpoint events linked on the annotation (so a reader can jump to them) and stores the resolved timestamps alongside, so time-window queries work the same either way. Both endpoint events must belong to the annotated run, in start-before-end order. The AnnotateRange API is the same call, with range_start_ns / range_end_ns (or range_start_event_id / range_end_event_id) in place of target_event_id.

Annotate a project

A project-scoped annotation has no run at all: it belongs to the project itself. It is the home for cross-run knowledge — promotion decisions, negative results, claims about the problem — that must survive every sandbox and run. Pass --project (slug or id) instead of --run:

hiloop annotations add \
  --project nano \
  --schema knowledge.v1 \
  --data '{"kind":"negative","claim":"GeLU hurts at this width","evidence_runs":["01K6Z…"]}'

Project annotations carry no lineage and no target event (--target-event/--range don’t apply); they are queryable like everything else, including through their schema-named view. Through the API, set project_id instead of run_id.

Re-scoring

Annotations are append-only and immutable: re-scoring writes a new event with a later timestamp. By default a read returns the latest annotation per target and schema — the target is what you annotated: the run, the event, or the range (its exact bounds) — so a re-score of the same target supersedes the prior one, while annotations on different targets (say, two different ranges of the same run) each stay current. Declaring an identity field (e.g. annotator) refines that key so the latest write per annotator is kept — a human correction and an LLM judge on the same event both survive, each at their latest. Identity is keyed by the field’s value in each record’s payload, so it applies across schema versions: records written before a version bump that declared the field fold into the same latest-wins chain as post-bump writes. Nothing is mutated or hidden: every version stays readable.

Distinct records need an identity field — or every write supersedes the last. With no identity field declared, the supersession key is only the annotated target, so two same-schema writes to the same run (or event, or range) are treated as versions of one record and the default read serves just the newest. If your rows are separate records — per-worker results, append-only measurements — declare the field(s) that tell them apart as identity (a lane, a segment, an annotator) so they coexist. Superseded rows are never lost: the default listing reports how many it hid, and --history returns them all — in SQL, through the ann_<schema>_history view.

List annotations

hiloop annotations list (or GET /v1/telemetry/annotations) is the served read implementing those semantics:

# The run's own current annotations (a re-score shows once, at its latest):
hiloop annotations list --run 01K6Z…

# Include every stored version, newest first, with superseded ones marked:
hiloop annotations list --run 01K6Z… --history

# One schema only (declared identity fields refine the latest-wins key either way):
hiloop annotations list --run 01K6Z… --schema eval.quality

# A project's annotations — the rollup: run-less project annotations plus run-anchored
# annotations from the project's runs, with a SCOPE column naming each row's anchor:
hiloop annotations list --project nano

Each row carries the annotation’s anchor (run_id, its lineage position; both absent on a run-less project annotation), the schema name, the target (target_event_id, or the range bounds plus the endpoint event ids for an event-bounded range), the writing principal, ts_wall_ns, and your payload under payload_json. On the HTTP API the payload is a raw JSON object string — the exact bytes you annotated, so payload values of every JSON type (including 64-bit integers, which plain JSON numbers cannot carry past 2^53) read back unchanged. The CLI’s --output json prints the full {"annotations": […]} envelope with each payload spliced back in as a real nested JSON object (same bytes, same fidelity — no double-decoding needed):

hiloop annotations list --run 01K6Z… --output json | jq '.annotations[].payload_json.score'

With --history, each JSON row also carries a status field — "current" or "superseded" — the same marking the table’s STATUS column shows, so scripts never re-derive latest-wins themselves.

status is per supersession key, not per payload value. A row reads superseded when a newer write shares its whole key: the anchor scope, the schema, and the target, refined by the schema’s declared identity fields. If the schema declares no identity field, every same-scope write belongs to one chain, so a standing row can read superseded even though no newer row carries its distinguishing payload value (a fingerprint, a lane): the write that superseded it is simply a different record in the same chain. Before treating status as a per-record fact (for example, verifying a standing registry row is still current before acting on it), make sure the field that names the record is declared :identity on the schema. The declaration applies retroactively: identity is keyed by each row’s stored payload value, so existing rows re-read under the refined key without a rewrite.

When the current view hides superseded versions, the listing says so rather than letting a write silently disappear: the CLI prints note: N superseded annotation row(s) hidden … use --history on stderr, and the API response carries the count as superseded_count.

Change a few fields of a record: read-modify-resend

A re-annotation replaces the payload wholesale — there is no partial update. When a record moves through states (a queue item going queued → submitting → scored, say) and a transition changes only a couple of fields, read the record’s current version, merge your changes into it, and resend the whole payload, so every untouched field carries forward instead of silently dropping:

# 1. Read the current version (filter on the field that identifies your record):
current=$(hiloop annotations list --project nano --schema submission.v1 --output json \
  | jq -c '.annotations[] | select(.payload_json.candidate_id == "cand-42") | .payload_json')

# 2. Merge only the fields that change (shallow merge; everything else carries forward):
next=$(jq -c '. + {"state": "scored", "lb": 0.712}' <<<"$current")

# 3. Resend — the merged payload becomes the record's current version:
hiloop annotations add --project nano --schema submission.v1 --data "$next"

The same pattern works run-anchored (--run in both commands). Because annotations are append-only, the pre-transition version stays in --history — a bad merge is always recoverable from the prior version.

Query annotations back

Registering a schema auto-creates a schema-named view: eval.quality is queryable as the table ann_eval_quality (the name is the schema name lowercased, with every non-alphanumeric run mapped to _, behind an ann_ prefix). The view serves the current annotations — the same latest-wins read as annotations list: the newest version per annotated target, refined by the schema’s declared identity fields — so SQL and the served listing agree about what the registry contains. Every stored version, superseded ones included, stays queryable through the full-history sibling ann_eval_quality_history — the SQL counterpart of --history. Registration alone creates both; a new schema version refreshes them, and archiving the schema retires them. A view’s payload-derived columns are the promoted fields — each under its declared name, backed by its typed column, so filters and aggregates get the full columnar acceleration. Alongside them every view carries the annotation’s context columns (event_id, run_id, root_run_id, lineage_path, project_id, principal, ts_wall_ns, and the point/range target columns) and a payload_json column holding the whole payload as JSON text. A field you didn’t promote is not a column — selecting it is an unknown-column error — but it stays reachable through the payload (promote it if you filter or aggregate on it often):

hiloop query --sql "
  SELECT lineage_path, outcome, AVG(score) AS avg_score
  FROM ann_eval_quality
  WHERE run_id = '01K6Z…'
  GROUP BY lineage_path, outcome"

# A non-promoted field, through the payload column:
hiloop query --sql "
  SELECT hiloop_json_get(payload_json, 'note') AS note
  FROM ann_eval_quality
  WHERE run_id = '01K6Z…'"

A declared field name that collides with a SQL keyword works quoted (GROUP BY \"group\"), and one that collides with a context column (or payload_json) is surfaced as field_<name>. The ann_ prefix is reserved: you can’t store your own data view under it. Annotations are still ordinary events, so the raw form works too — filter events by signal = 'annotation' and the schema name. The raw attributes_json carries your payload plus platform-stamped metadata; reach into it with hiloop_json_get(attributes_json, '$.path.to.key'):

hiloop query --sql "
  SELECT hiloop_json_get(attributes_json, '\$.note') AS note
  FROM events
  WHERE run_id = '01K6Z…' AND signal = 'annotation' AND name = 'eval.quality'"

Query telemetry — the SQL query that reads annotations back.
Compare runs — compare annotated outcomes with SQL.
Saved & data views — save an annotation query as a reusable view.

​Register a schema first

​Promote the fields you query

​Annotate a run or an event

​Annotate a range

​Annotate a project

​Re-scoring

​List annotations

​Change a few fields of a record: read-modify-resend

​Query annotations back

​Next