> ## Documentation Index > Fetch the complete documentation index at: https://docs.hiloop.ai/llms.txt > Use this file to discover all available pages before exploring further. # Sandbox architecture > How hiloop sandboxes are built: Kubernetes objects, gVisor isolation, and a workspace served from node-local storage and replicated to your object storage. hiloop runs agent sandboxes on Kubernetes, in a cluster you operate. A sandbox is a Kubernetes object. Creating one writes a custom resource; a controller turns that into a pod running under gVisor in a dedicated namespace. The sandbox's `/workspace` is served from node-local storage by a per-node daemon, and its durability comes from continuous replication to your object storage rather than from a network volume attached to the pod. That last choice is the one that shapes everything else, so it is worth stating plainly: because the filesystem is described by a manifest of content-addressed chunks, forking a prepared workspace copies a manifest instead of copying data. The cost of a fork does not grow with the size of the workspace. ## The shape at a glance ```mermaid theme={null} flowchart TB CLI[CLI / SDK / API] --> EDGE[Authenticating edge] EDGE --> API[Control-plane API] API -->|writes| CR[Sandbox resource] CTRL[Sandbox controller] -->|reconciles| CR CTRL -->|creates| POD[Sandbox pod under gVisor] MIRROR[Status mirror] -->|observes| POD MIRROR -->|converges state| REC[(Product record)] API -->|reads| REC ``` 1. **One authenticating edge.** It resolves your credential to an organization, user, and scope, and stamps that identity. Backend services trust only edge-stamped identity, never a request body. 2. **The control-plane API** validates the request, owns the product record, and writes the sandbox resource. 3. **The controller** reconciles that resource into exactly one pod. 4. **A status mirror** watches the cluster and converges what it observed into the product record. The state you read is evidence of what the runtime is doing, not a record of what someone asked for. Two consequences worth naming. Sandbox identity stays opaque, so a sandbox id never exposes a pod name, container id, or node name and your callers cannot couple to the runtime. And admission fails closed: a request a deployment cannot enforce exactly is refused with a named `unsupported_capability` rather than quietly weakened. ## Design trade-offs Every choice below buys something and costs something. The costs are the useful part of this section, so they are stated as plainly as the benefits. ### Kubernetes objects, rather than owning the host Most sandbox platforms run their own orchestrator over VMs or bare metal, and some Kubernetes-based ones run workloads as privileged containers with host paths, driven by a node agent outside the pod model. Either way the platform controls the host outright. We run sandboxes as ordinary Kubernetes objects in a cluster you operate. **It costs us** the freedom to do whatever is convenient on the node. We have to satisfy your cluster's admission and pod-security rules rather than working around them, and we inherit Kubernetes pod-start cost: the kubelet, the CNI, and the pod sandbox all sit on the path to a running sandbox, which a platform driving containers directly on the node does not pay. That is a real startup and density tax, and it is the clearest thing we give up. **We took it** because it puts sandboxes in your cluster under your policy, your tooling, and your observability, with nothing phoning home, and because the isolation controls are then enforced by your API server rather than by our good intentions. It also means you can inspect and debug a sandbox with `kubectl` rather than through our dashboard. ### gVisor for every sandbox, rather than the host kernel Every sandbox pod runs under a user-space kernel that intercepts system calls. **It costs us** syscall performance. Independent published measurements put simple syscalls at roughly twice the cost of a conventional container runtime, and metadata-heavy filesystem work considerably worse. Some workloads notice, and a build over a large dependency tree is the usual case. **We took it** because the isolation boundary then does not depend on our application code being correct. A bug in our control plane, or a stolen control-plane credential, cannot land a customer workload on the host kernel, because the requirement is checked by cluster admission rather than by us. For a platform whose entire job is running untrusted code, that is a better trade than the performance. ### Local disk plus blob durability, rather than a network volume per sandbox Giving each sandbox its own network volume is the obvious design. Its costs are properties of network block storage rather than of any particular implementation: | | Volume per sandbox | Blob-backed workspace | | ------------------ | ----------------------------------------------- | ------------------------------------------- | | Sandboxes per node | Bounded by the node's device-attachment ceiling | Bounded by pods per node and memory | | Placement | Pinned for life to the zone its volume lives in | Free to come back on any node in any zone | | Idle cost | Billed per volume, whether or not it is in use | Billed as object storage for the bytes kept | | Fork | Copies the filesystem | Copies a manifest | The attachment ceiling is the one that bites: it is a fixed per-instance budget, shared with network interfaces, and it caps durable sandboxes per node low enough to dominate scheduling. Across the providers we surveyed, the field has largely moved away from this shape for that reason. **It costs us** cold reads and a replication window. Data not yet on the node faults in from object storage, so the first touch of cold data is slower than reading an attached disk, and against a cloud store that gap is substantial rather than marginal. And a write is durable once replicated rather than immediately, so data written inside the replication window is at risk if the node is lost. That window is bounded rather than open-ended; [what survives what](#what-survives-what) states what it means. **We took it** because those three costs are structural rather than fixable, and because the fan-out an agent workload actually performs is cheap in this shape and expensive in the other. ### A manifest copy for fork, rather than copying a filesystem Forking publishes a description of the filesystem under a new identity and shares the underlying data, so a fork's cost is governed by the size of that description rather than the size of the workspace. **It costs us** real complexity in the storage engine: content-addressed chunks, a replication log with exactly-one-writer semantics, and a cache with its own eviction behavior. That is harder than calling a volume-clone API, and it is why we copied a hardened open-source implementation of the block layer instead of inventing one. **We took it** because branching one prepared environment into many is the operation an agent or evaluation workload performs constantly, and this is the difference between fan-out being free and fan-out being the bottleneck. ### Continuous replication, rather than periodic snapshots Writes are captured behind the live device and streamed to object storage in order, instead of the filesystem being quiesced on a timer. **It costs us** bookkeeping we have to get exactly right, and a bound we have to enforce: there is a backlog of not-yet-replicated data, and at the bound the workload feels backpressure instead of the backlog growing without limit. A workload writing faster than your object store absorbs will be slowed down. **We took it** because the alternative stops guest I/O to take a copy, and because recovery then reads current state rather than replaying history, so a restore does not get slower the more that was written. ### Memory capture as an accelerator, not the correctness path Filesystem continuity is what the product guarantees. Memory capture sits on top of it: a stop checkpoints the process tree and records that image as the sandbox's restore point, and a start brings back the filesystem and starts the workload from its image rather than replaying the capture. **It costs us** the thing people most want to hear, which is that a sandbox resumes exactly where it left off. It does not, and a workload that needs to continue mid-flight has to rebuild that from what it wrote to disk. **We took it** because a memory image is only valid against the exact filesystem generation it was captured over, and a capture that restores successfully with subtly divergent memory is worse than one that fails outright. Treating the filesystem as the guarantee and memory as an optimization keeps that failure mode visible instead of silent. See [memory capture](#memory-capture). ## What you install | Piece | What it does | Where it runs | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | | Sandbox resource definitions | The API object the controllers reconcile, from the open-source agent-sandbox project. Applied with `kubectl` because Helm does not manage definition lifecycle. | Cluster-scoped | | One umbrella Helm release | The control-plane API, edge authorization, telemetry gateway, web console, the operator-only Rust model proxy, and the sandbox controllers. | Your platform namespace | | Workspace daemon | Serves each sandbox's `/workspace` from a node-local chunk cache and replicates writes to your object storage. One per node. | Sandbox nodes | | Memory capture daemon | Drives process-state checkpoints against the sandbox runtime. One per node. | Sandbox nodes | | Sandbox pods | Your images. One pod per sandbox, gVisor runtime class, no Kubernetes service-account token, egress deny floor. | A dedicated workload namespace | You provide Kubernetes and nodes, PostgreSQL, object storage, a registry mirror, TLS, an identity provider, and a collector for telemetry. See [deploy on Kubernetes](/deployment/kubernetes) for the full cluster contract. ## Isolation Sandboxes run arbitrary, untrusted images, so every control sits outside the workload and is pinned by cluster policy rather than by application code. * **gVisor for every sandbox pod.** The `runsc` runtime class puts a user-space kernel between the workload and the host. * **Admission pins it.** A cluster policy refuses any sandbox whose pod template does not request the gVisor runtime class, or which asks to share the host's network, process, or IPC namespaces. The check runs at the Kubernetes API server, so neither a control-plane bug nor a stolen control-plane credential can place a workload on the host kernel. * **A dedicated namespace** enforcing the Kubernetes `baseline` pod security standard as defense in depth, with no Kubernetes service-account token in sandbox pods. * **An egress deny floor.** DNS, the public internet, and the API's dedicated capture listener are network-reachable. The workload has no capture proof, so that listener rejects it; all other cluster services, private address ranges, and the cloud metadata endpoint are unreachable. * **The workspace is mounted `nosuid` and `nodev`**, so an image cannot smuggle setuid-root binaries or device nodes in through its own workspace. `noexec` is deliberately not set, because building and running your own code in `/workspace` is the primary flow. * **No platform-managed provider credential in a workload.** The approved gateway target uses an external Envoy data plane plus a Hiloop control service to verify live-workload proof, policy, and budget and select a credential outside the guest. That sandbox path is not deployed; the existing Rust proxy is operator-only. Direct provider access is not a supported fallback, so clean sandbox model access remains unavailable. ## The workspace layer ```mermaid theme={null} flowchart LR POD[Sandbox pod] -->|/workspace| DEV[Block device] DEV --> DAEMON[Per-node workspace daemon] DAEMON <--> CACHE[(Node-local cache)] DAEMON <-->|replicate and fetch| STORE[(Your object storage)] ``` Each sandbox's `/workspace` is a block device served by the per-node daemon, with a manifest describing where every block lives. **Node-local storage is the read-write path.** A copy-on-write cache on the node holds fetched chunks and dirty blocks. Cold blocks fault in from your object storage in the background while the workload runs. **Reads are verified before they are served.** Blocks are 4 KiB; data is fetched in larger compressed, checksummed chunks, and the checksum is verified before a single byte reaches the kernel. A corrupt or truncated fetch leaves no partial state to serve later. **Chunk boundaries follow file boundaries, not fixed offsets.** This is what makes sharing real. Recompiling a project shifts symbol addresses throughout a filesystem image, so fixed-offset chunking finds almost nothing in common between two builds of the same image, while file-aligned chunking finds most of it. Deduplication is what makes fork and fan-out cheap, so the layout is chosen for it. **Durability is continuous replication, not a periodic snapshot.** Writes are captured point-in-time behind the live device, so guest I/O is never stopped to take a copy. Batches land on node disk, then upload in strict order. Each payload is confirmed in storage before its log record publishes, and that record publishes through a create-only conditional write, so exactly one writer can ever win a sequence number. A lost race is treated as split brain and stops the lane rather than retrying into corruption. **Restore is a read, not a replay.** Current state is the newest consolidated manifest plus its log tail. A restore composes those and starts serving immediately, faulting blocks in lazily. There is no byte replay on the start path, so starting does not get slower the more that was written. **Fork is a manifest copy.** Forking publishes the source's effective manifest under the child's own identity. The child then diverges through its own log and its own write cache. Chunks stay shared because the objects the parent's history wrote are immutable and uniquely keyed: nothing is rewritten and nothing is re-keyed. Forking one source into many children never serializes on the source. **Backpressure is bounded and explicit.** The un-replicated backlog is bounded by how much can drain inside a spot interruption grace window. Below that bound writes never block; at it, the lane applies backpressure rather than growing an unbounded local backlog. ### Portability One policy layer of our own, holding a single timeout ladder, retry policy, verify-before-serve rule, and error taxonomy, sits over maintained blob drivers for S3, Google Cloud Storage, and local files. Any S3-compatible endpoint works, and the S3 lane is tested continuously against MinIO. The conditional-write primitive the replication log depends on is implemented natively by each driver, with no vendor escape hatches, which is why the same code path is correct on your object store as on ours. ## What survives what | Event | `/workspace` files | Processes and memory | | ---------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A process exits | Survive | Gone | | Stop and start | Survive | Not resumed. A stop captures the process tree and records that image as the sandbox's restore point; a start brings back the filesystem and starts the workload from its image rather than resuming the capture | | The node is lost | Everything replicated survives. The tail still draining is at risk, bounded by the interruption grace window | Gone; the workload starts again from its image | | The sandbox is deleted | Destroyed. State you explicitly kept outlives it | Gone | **What a durability receipt proves.** A receipt reports exactly what has been proven and nothing more: `local` means the capture exists on the node, `replicated` means its upload to your object storage is confirmed. The API does not report `replicated` on the strength of a local write, and it does not hold your request open waiting for replication. If you need the stronger guarantee before proceeding, ask for it and wait. Control-plane records, including sandbox identity, lineage, metadata, and telemetry, live in your replicated stores independent of any node. ### Memory capture Filesystem continuity is the correctness path. Memory capture sits on top of it, for a long-lived session you would rather not warm up again. A stop captures: while the pod is still running, the per-node daemon pauses the process tree and checkpoints the sandbox against the container runtime's state directory, then uploads the image to your object storage. The captured image is recorded as the sandbox's **restore point**. **A restore point is a record, not something a start consumes.** Starting a stopped sandbox brings back its `/workspace` and starts the workload from its image. It does not replay the captured memory, so treat process state as something to rebuild from what you wrote to disk, and treat the restore point as the artifact a resume path would read rather than as a resume that already happens. Two properties of the capture are treated as load-bearing: * **Restorability is verified, not assumed.** A checkpoint can report success and still be unrestorable, so only an actual restore proves an image is good. * **A memory image is bound to the exact filesystem generation it was captured over.** A small drift in a library underneath a checkpoint would restore successfully with divergent memory, which is worse than failing outright. ## Where your data lives Control-plane metadata, every workspace byte and manifest, event payloads, sandbox compute, and your credentials stay in your environment. The platform can run with no outbound internet route at all: mirror the images and charts into your registry, host the identity provider inside the boundary, and point telemetry at your own collector. One thing to be clear about, because it is a property of the product rather than of your configuration: a sandbox's default egress posture allows the public internet, with private ranges and the metadata endpoint denied. A workload cannot reach your internal infrastructure, but a workload that wants to send data to a public endpoint can. If your requirement is that nothing leaves the network at all, that is a policy you impose at your own perimeter. ## Related pages * [Kubernetes architecture](/concepts/kubernetes-architecture): the object model, the reconcile loop, and the isolation boundary in detail. * [Deploy on Kubernetes](/deployment/kubernetes): what to install and what your cluster must provide. * [Sandbox reliability](/sandboxes/reliability): retries, fail-closed admission, and durability. * [Snapshots and branching](/concepts/workspaces): the persistence and fan-out verbs. *** Parts of the workspace block storage layer derive from E2B's Apache-2.0 licensed implementation, with provenance and license recorded in our source tree.