> ## Documentation Index > Fetch the complete documentation index at: https://docs.hiloop.ai/llms.txt > Use this file to discover all available pages before exploring further. # Sandbox reliability > Observed state, safe retries, fail-closed admission, and what is durable today. Sandbox reliability starts with the record: the `state` field reports what the runtime has been observed doing, never what a request intended. A sandbox is `running` because its workload is running, and out-of-band runtime death converges the record to a terminal state with a stable `state_reason` instead of leaving it green. ## Safe retries Mutations are asynchronous and designed for ambiguous-failure retries: * Create and snapshot accept an `Idempotency-Key`; reusing the key with the same request replays the original resource instead of creating a duplicate. A reused key with a different body returns `409 idempotency_conflict`. * Exec accepts an optional `Idempotency-Key` so a retried command is not run twice. * Delete is idempotent by outcome: it is accepted once for any sandbox that still holds resources, `failed` included, so an ended sandbox never strands its storage; a repeat returns `404 not_found`, and only a `quarantined` sandbox refuses. * Reads long-poll: `GET /v1/sandboxes/{id}` can wait (bounded, up to 25 seconds) for a target state instead of hot-polling. Quota and capacity errors are explicit, and they name which limit was reached. Create admission returns `429 quota_exceeded` with `details.quota.metric` set to `sandboxes.running` when the organization already has as many sandboxes running as it may, or `sandboxes.total` when it holds as many as it may in total, running and stopped together. The first clears when a sandbox stops or ends; the second only when one is deleted. Honor the retry-after with capped backoff and keep the same idempotency key. ## Fail-closed admission A request the deployment cannot enforce exactly is refused, never weakened: * an unavailable storage class, volume transport, or secret-injection path fails with `unsupported_capability` naming what is missing; * a verb this deployment has no executor for is refused rather than accepted as intent nothing acts on (an admitted request nothing can complete would otherwise wedge the sandbox); * a snapshot request that cannot reach durability before its deadline fails rather than returning an unusable receipt. The current status of each lane is [what's live today](/sandboxes/overview#whats-live-today). ## What is durable today Placement and storage are independent. Standard placement is spot-first with on-demand fallback. Non-preemptible placement excludes provider capacity reclaim but not host failure, health events, drains, or operator action; it therefore reduces one interruption cause rather than replacing the filesystem and snapshot guarantees below. Losing a node splits a durable sandbox's state in two: files in the durable workspace survive, and running processes do not. The sandbox recovers `/workspace` from the deployment's configured durable backing, and the workload starts again from its image. * **Sandbox filesystems** use node-bound standard storage by default: writes not yet captured by a snapshot can be lost if the node is lost. * **`--storage-class durable`** mounts `/workspace` from a deployment-provided StorageClass with a lifetime of its own, instead of using a directory inside the sandbox's own filesystem. Lose the node and the sandbox recovers `/workspace` from that class's backing storage; everything outside `/workspace` stays node-bound and is lost with the node. The workspace is released when the sandbox is deleted, whatever state the sandbox reached first. The capability is available by default; a deployment whose cluster has no such storage class withdraws it, and there the request is refused with `unsupported_capability` rather than quietly downgraded. Two limits worth knowing before you rely on it. Recovery replaces the workload rather than resuming it: processes, their memory, and anything they held open do not survive the interruption, so a program that must pick up where it left off has to rebuild that from what it wrote to `/workspace`. The backing class determines whether recovery reattaches a volume or hydrates object-backed data, and whether it has any topology or recovery-staleness constraints. On the platform default image `/workspace` is also the home directory, so an installed toolchain, your shell history, and the configuration and session state a coding-agent CLI keeps are durable on the same terms as your files, and a sandbox created from a snapshot has them too. That includes any credential such a tool stores after you sign in to it: it is kept, and it is carried into anything created from a snapshot taken afterwards, so treat a snapshot of a sandbox you have signed in to as something that can act as you. A custom image keeps whatever `HOME` it sets, and gets this only if that path is under `/workspace`. * **Deployment-registered shared filesystems** are available only when an operator has bound a logical filesystem id to an organization and an existing claim. A create may mount an existing subdirectory at `/workspace` or below `/mnt`; no storage credential or backend coordinate enters the sandbox. When shared storage owns `/workspace`, the sandbox's own durable workspace moves to `/private`. Hiloop leaves the image's `HOME` and all other environment untouched, so an agent harness should place its private session, credential, and checkpoint state under `/private`. Shared files are reattached on stop/start and workload replacement, but are deliberately outside sandbox snapshots. A snapshot-derived child must request the shared attachment again. * **Snapshots** capture state that must outlive the sandbox and its zone. A receipt reports `local` when the storage snapshot exists and `replicated` only once its remote upload is confirmed; treat `replicated` as the bar for state you cannot afford to lose. * **Control-plane records** (sandbox identity, lineage, snapshot metadata, and telemetry) live in replicated stores independent of any node. Failed and terminated sandbox records stay readable for a retention window so post-mortems can see what happened, then return `404 not_found`. ## Blast-radius bounds Per-organization quotas bound active sandboxes at create admission, and the deployment carries a fleet-wide ceiling enforced transactionally at the cluster, so a runaway create loop cannot exhaust the platform. TTLs bound individual sandbox lifetimes; expiry is terminal for the runtime. See [lifecycle](/sandboxes/lifecycle) for states and [troubleshooting](/sandboxes/troubleshooting) for diagnosis.