> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hiloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Reproduce a failed run

> Trace a failed agent run back to sandbox state, commands, image, and workspace storage.

When an agent run fails, the goal is not only to see the error. The goal is to recreate the state
that produced it.

hiloop gives you the pieces:

* run id;
* lineage path;
* sandbox id;
* command stdout, stderr, and exit code;
* image reference and digest;
* workspace storage class and snapshot source when one was used;
* telemetry events.

## Collect identifiers

```text theme={null}
run_id: 01K6Z000000000000000000000
lineage_path: 01H8A.01H8B
sandbox_id: 7c9e6679-7425-40de-944b-e07fc1f90ae7
```

If you only have a run id, start with telemetry:

```sh theme={null}
hiloop query --run-id "$RUN_ID" --limit 100 --output json
```

## Inspect failure signals

Look for process output, logs, or failed network responses:

```sh theme={null}
hiloop query --sql "
  SELECT ts_wall_ns, signal, name, http_status_code, http_target
  FROM events
  WHERE run_id = '${RUN_ID}'
    AND (lineage_path = '${LINEAGE_PATH}' OR lineage_path LIKE '${LINEAGE_PATH}.%')
    AND http_status_code >= 400
  ORDER BY ts_wall_ns
  LIMIT 100"
```

<Note>
  `$RUN_ID` / `$LINEAGE_PATH` here are hiloop-generated identifiers (a ULID and a dotted path of run
  ULIDs), so they contain no quotes and are safe to interpolate. When you embed a **user-supplied**
  string in `--sql`, single-quote-escape it (double any `'`) — or use the `--run-id` / `--signal`
  flags, which build the `SELECT` and quote values for you.
</Note>

For process output:

```sh theme={null}
hiloop query --run-id "$RUN_ID" --signal log
```

Scope that to one lineage subtree by writing the same prefix predicate in `--sql`.

## Ask the platform what happened

The sandbox's run also carries the platform's own record of the failure —
[`runtime` lifecycle events](/observability/event-model#runtime-platform-lifecycle) flow
independently of workload capture. `operation.failed` names a stable `error.code`, and `sandbox.state_changed` /
`sandbox.preempted` show whether the sandbox itself moved underneath the run:

```sh theme={null}
hiloop query --run-id "$RUN_ID" --signal runtime --output json
```

The execution response is the source of truth for the sandbox command and exit status; the ambient
run is its correlated, best-effort telemetry record.

## Inspect the command's outcome

There is no separate execution resource to fetch. `hiloop sandbox exec` returns the outcome
directly, with stdout and stderr separated and the real exit code:

```sh theme={null}
hiloop sandbox exec "$SANDBOX_NAME" --output json -- ./reproduce.sh
```

For a command wrapped locally by `hiloop run`, or run through a sandbox's managed exec boundary,
the captured `exec` signal carries the same ground truth after the fact:

```sh theme={null}
hiloop query --run-id "$RUN_ID" --signal exec --output json
```

## Inspect sandbox state

```sh theme={null}
hiloop sandbox get "$SANDBOX_NAME" --output json
```

Record:

* `state`, and `state_reason` when the sandbox failed or was quarantined;
* `image` (or `snapshot_id`) as resolved at admission;
* `storage_class`;
* `resources`, the compute shape resolved at admission;
* `metadata`;
* `lineage_path`.

## Re-run with diagnostics

Run the smallest command that reproduces the failure:

```sh theme={null}
hiloop api "/v1/sandboxes/${SANDBOX_ID}:execute" \
  -X post \
  -d '{
    "sandbox_id": "'"${SANDBOX_ID}"'",
    "command": {
      "program": "bash",
      "args": ["-lc", "set -euxo pipefail; python -m agent.run --debug"],
      "working_dir": "/workspace",
      "timeout_secs": "300"
    }
  }'
```

## Compare runs

Compare the failed run to a known-good run. Events unique to the failed run are an anti-join on
signal, name, and attributes:

```sh theme={null}
hiloop query --sql "
  SELECT a.event_id, a.ts_wall_ns, a.signal, a.name
  FROM events a
  LEFT JOIN events b
    ON  b.run_id = '${KNOWN_GOOD_RUN_ID}'
    AND b.signal = a.signal
    AND b.name = a.name
    AND b.attributes_json = a.attributes_json
  WHERE a.run_id = '${FAILED_RUN_ID}' AND a.signal = 'exec' AND b.event_id IS NULL
  ORDER BY a.ts_wall_ns"
```

Then compare model calls by swapping `a.signal = 'exec'` for `a.signal = 'llm'`. See
[Compare two runs](/guides/querying-telemetry#compare-two-runs) for the reverse difference.

## Checklist

* Same image digest.
* Same source commit.
* Same snapshot source when applicable.
* Same command arguments.
* Same environment variables, excluding rotated secrets.
* Same model configuration unless you are testing model sensitivity.
* Same input artifacts.
* Same resource class unless you are testing resource sensitivity.