Reproduce a failed run

When an agent run fails, the goal is not only to see the error. The goal is to recreate the state that produced it. hiloop gives you the pieces:

run id;
lineage path;
sandbox id;
command stdout, stderr, and exit code;
image reference and digest;
workspace storage class and snapshot source when one was used;
telemetry events.

Collect identifiers

run_id: 01K6Z000000000000000000000
lineage_path: 01H8A.01H8B
sandbox_id: 7c9e6679-7425-40de-944b-e07fc1f90ae7

If you only have a run id, start with telemetry:

hiloop query --run-id "$RUN_ID" --limit 100 --output json

Inspect failure signals

Look for process output, logs, or failed network responses:

hiloop query --sql "
  SELECT ts_wall_ns, signal, name, http_status_code, http_target
  FROM events
  WHERE run_id = '${RUN_ID}'
    AND (lineage_path = '${LINEAGE_PATH}' OR lineage_path LIKE '${LINEAGE_PATH}.%')
    AND http_status_code >= 400
  ORDER BY ts_wall_ns
  LIMIT 100"

$RUN_ID / $LINEAGE_PATH here are hiloop-generated identifiers (a ULID and a dotted path of run ULIDs), so they contain no quotes and are safe to interpolate. When you embed a user-supplied string in --sql, single-quote-escape it (double any ') — or use the --run-id / --signal flags, which build the SELECT and quote values for you.

For process output:

hiloop query --run-id "$RUN_ID" --signal log

Scope that to one lineage subtree by writing the same prefix predicate in --sql.

Ask the platform what happened

The sandbox’s run also carries the platform’s own record of the failure — runtime lifecycle events flow independently of workload capture. operation.failed names a stable error.code, and sandbox.state_changed / sandbox.preempted show whether the sandbox itself moved underneath the run:

hiloop query --run-id "$RUN_ID" --signal runtime --output json

The execution response is the source of truth for the sandbox command and exit status; the ambient run is its correlated, best-effort telemetry record.

Inspect the command’s outcome

There is no separate execution resource to fetch. hiloop sandbox exec returns the outcome directly, with stdout and stderr separated and the real exit code:

hiloop sandbox exec "$SANDBOX_NAME" --output json -- ./reproduce.sh

For a command wrapped locally by hiloop run, or run through a sandbox’s managed exec boundary, the captured exec signal carries the same ground truth after the fact:

hiloop query --run-id "$RUN_ID" --signal exec --output json

Inspect sandbox state

hiloop sandbox get "$SANDBOX_NAME" --output json

Record:

state, and state_reason when the sandbox failed or was quarantined;
image (or snapshot_id) as resolved at admission;
storage_class;
resources, the compute shape resolved at admission;
metadata;
lineage_path.

Re-run with diagnostics

Run the smallest command that reproduces the failure:

hiloop api "/v1/sandboxes/${SANDBOX_ID}:execute" \
  -X post \
  -d '{
    "sandbox_id": "'"${SANDBOX_ID}"'",
    "command": {
      "program": "bash",
      "args": ["-lc", "set -euxo pipefail; python -m agent.run --debug"],
      "working_dir": "/workspace",
      "timeout_secs": "300"
    }
  }'

Compare runs

Compare the failed run to a known-good run. Events unique to the failed run are an anti-join on signal, name, and attributes:

hiloop query --sql "
  SELECT a.event_id, a.ts_wall_ns, a.signal, a.name
  FROM events a
  LEFT JOIN events b
    ON  b.run_id = '${KNOWN_GOOD_RUN_ID}'
    AND b.signal = a.signal
    AND b.name = a.name
    AND b.attributes_json = a.attributes_json
  WHERE a.run_id = '${FAILED_RUN_ID}' AND a.signal = 'exec' AND b.event_id IS NULL
  ORDER BY a.ts_wall_ns"

Then compare model calls by swapping a.signal = 'exec' for a.signal = 'llm'. See Compare two runs for the reverse difference.

Checklist

Same image digest.
Same source commit.
Same snapshot source when applicable.
Same command arguments.
Same environment variables, excluding rotated secrets.
Same model configuration unless you are testing model sensitivity.
Same input artifacts.
Same resource class unless you are testing resource sensitivity.

Get started

How-to guides

Reproduce a failed run

Collect identifiers

Inspect failure signals

Ask the platform what happened

Inspect the command’s outcome

Inspect sandbox state

Re-run with diagnostics

Compare runs

Checklist

​Collect identifiers

​Inspect failure signals

​Ask the platform what happened

​Inspect the command’s outcome

​Inspect sandbox state

​Re-run with diagnostics

​Compare runs

​Checklist

Collect identifiers

Inspect failure signals

Ask the platform what happened

Inspect the command’s outcome

Inspect sandbox state

Re-run with diagnostics

Compare runs

Checklist