> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hiloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Kubernetes architecture

> How a sandbox is represented, reconciled, isolated, and reached inside a Kubernetes cluster.

A sandbox is a Kubernetes object. Creating one through the API writes a custom resource; a
controller reconciles that resource into a pod; a status mirror converges what the cluster actually
observed back into the product record. Nothing in the model hides a pod behind a private protocol,
so an operator can inspect the whole lifecycle with `kubectl`.

```mermaid theme={null}
flowchart TB
  C[CLI / SDK / API] --> A[Control-plane API]
  A -->|writes| CR[Sandbox custom resource]
  CTL[Sandbox controller] -->|watches| CR
  CTL -->|creates| POD[Sandbox pod, gVisor]
  MIR[Status mirror] -->|watches| CR
  MIR -->|watches| POD
  MIR -->|converges observed state| DB[(Product record)]
  A -->|reads| DB
```

## The sandbox resource

Sandboxes are represented by the `Sandbox` kind in the `agents.x-k8s.io` API group, served at
`v1beta1` and scoped to a namespace. The type comes from the open-source
[agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) project maintained under
Kubernetes SIG Apps, licensed Apache-2.0. hiloop runs the upstream controller from a pinned image
and consumes its API types as a versioned module rather than forking them.

The spec fields that carry a sandbox's shape:

| Field                  | Meaning                                                                                                                            |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `podTemplate`          | The pod that *is* the sandbox: its containers, resources, and volumes. Required.                                                   |
| `operatingMode`        | `Running` or `Suspended`. Defaults to `Running`. This field is the whole of stop and start.                                        |
| `shutdownTime`         | The instant the sandbox expires, set from the requested time to live.                                                              |
| `shutdownPolicy`       | `Retain` or `Delete`: whether expiry leaves the record or removes it. Defaults to `Retain`.                                        |
| `volumeClaimTemplates` | Claims created per sandbox and owned by it, rather than by its pod.                                                                |
| `service`              | Whether to publish a headless Service. hiloop does not request one, so no sandbox gets in-cluster DNS or a stable cluster address. |

Status is observation, not intent: `conditions` (`Ready`, `Suspended`, `Finished`), plus
`nodeName` and `podIPs` once scheduled. The controller is the only writer of status.

## The reconcile loop

For each sandbox resource that is not being deleted, the controller:

1. **Creates exactly one pod** from `podTemplate`, with an owner reference back to the sandbox. One
   sandbox is always one pod. Observing more than one live child pod for a sandbox is a
   contradiction, and the product record reports it as needing attention rather than picking a
   winner.
2. **Re-propagates pod-template metadata onto the live pod** on every reconcile, including scrubbing
   keys that were removed. This is how a control-plane annotation edit reaches a running pod without
   another controller in the path.
3. **Creates one claim per `volumeClaimTemplates` entry**, owned by the sandbox rather than the pod.
   Because the claim's owner is the sandbox, it outlives any single pod and is garbage-collected with
   the sandbox itself.
4. **Writes status** from pod readiness.

Stop and start are the same mechanism: setting `operatingMode: Suspended` makes the controller
delete the sandbox's pod, and setting it back to `Running` makes the controller create a
replacement. That is the entire lifecycle transition. Because storage owned by the sandbox is not
owned by the pod, a filesystem with its own lifetime survives the gap and is presented again to the
replacement pod. Processes do not: the replacement pod starts its image's entrypoint from the
beginning.

The control plane reaches the cluster directly over the Kubernetes API and writes only merge
patches, never a server-side apply, so it never takes ownership of fields the controller manages.
Its write vocabulary against a sandbox is small: create, patch the operating mode and expiry,
annotate, and delete.

### State is mirrored, never inferred

A separate status-mirror controller watches sandbox resources and sandbox pods and writes what it
observed into the product record. The API answers reads from that record. This is why a sandbox's
`state` reports evidence: a sandbox is `running` because a pod was observed ready, and out-of-band
pod death converges the record to a terminal state instead of leaving it green.

The mirror resolves contradictions explicitly rather than smoothing them. An unparseable organization
label, a missing resource identity, more than one child pod, or a runtime receipt it cannot read all
produce an `attention` state naming the anomaly, and a recorded capability contradiction produces
`quarantined`. It reconciles state; it never writes a pod.

## What a sandbox is at the pod level

One pod, with one container that matters:

* **The workload container runs your image's entrypoint unmodified.** No command, no arguments, no
  injected environment, and no security-context override are applied to it. What runs is what your
  image says runs, as the user your image declares.
* **`restartPolicy: Never`.** A workload that exits has exited; it is not silently restarted
  underneath you.
* **No Kubernetes service-account token is mounted.** A sandbox cannot talk to the Kubernetes API,
  because it holds no credential for it.
* **The pod requests the gVisor runtime class**, which also carries a fixed per-pod memory overhead
  for the sandboxing layer so the scheduler accounts for it.

Platform-owned helper containers, where a deployment uses them, are hardened the opposite way from
the workload: they run with a read-only root filesystem, every capability dropped, privilege
escalation disabled, and the default seccomp profile. The asymmetry is deliberate. Your container is
untrusted and unconstrained inside the boundary; the boundary itself is what is constrained.

Sandbox identity stays opaque across the API: a sandbox id never exposes a pod name, container id,
or node name.

## Reaching into a sandbox

`hiloop sandbox exec` is brokered through the Kubernetes API as an exec into the workload container.
The command's real exit code is recovered from the API server's terminating status frame rather than
inferred from a closed stream, so a non-zero exit is reported as a non-zero exit and not as a
transport error.

Interactive access is served by a small static server that runs inside the workload container, so a
shell sees your filesystem as your image's own user. It shares the pod's network namespace but not
its mount namespace, which is why the exec path, not the session path, is what reaches the
container's own filesystem view.

Port forwarding through the container runtime is deliberately not offered. Runtime port-forward
implementations work by entering the pod's host-side network namespace and dialing loopback, which
cannot reach a workload whose network stack lives in a user-space kernel. Forwarding a port is done
over the session instead, with the standard OpenSSH `-L` flag.

## The isolation boundary

Sandboxes run arbitrary, untrusted images, so every control sits outside the workload.

* **gVisor.** Each sandbox pod runs under the `runsc` runtime class: a user-space kernel that
  intercepts system calls instead of passing them to the host kernel.
* **Admission pins the boundary.** A cluster admission policy evaluates sandbox resources at the API
  server and refuses any whose pod template does not request the gVisor runtime class, or which asks
  to share the host's network, process, or IPC namespaces. The policy fails closed, and it binds the
  invariant to the API server rather than to the code that renders sandboxes, so neither a
  control-plane regression nor a stolen control-plane credential can place a customer workload on the
  host kernel.
* **Pod Security.** The sandbox namespace enforces the `baseline` Pod Security Standard and audits
  against `restricted`. Baseline rather than restricted is a considered choice: arbitrary customer
  images legitimately run as root and declare no seccomp or capability posture, so they cannot
  satisfy restricted. Baseline still refuses privileged containers and host namespace escapes at the
  API server as defense in depth.
* **Network policy.** Ingress to sandboxes is default-deny, with a single exception for the control
  plane's session port. Egress allows cluster DNS and the public internet while denying private
  address ranges and the link-local range, so a workload can reach the internet but not the
  deployment's own infrastructure or the cloud metadata endpoint. See
  [network access](/sandboxes/network-egress).
* **An organization-scoped identity.** Each organization's sandboxes run under their own service account, with
  token mounting disabled.
* **An object-count ceiling.** A resource quota on sandbox objects bounds the fleet transactionally
  at the API server, behind the per-organization quotas applied at admission.

Admission fails closed throughout: a request a deployment cannot enforce exactly, such as a storage
class or transport it does not back, is refused with `unsupported_capability` rather than quietly
downgraded.

## Related pages

* [Deploy on Kubernetes](/deployment/kubernetes): what to install, and what the chart needs from your cluster.
* [Architecture](/concepts/architecture): the components around the sandbox runtime.
* [Sandbox lifecycle](/sandboxes/lifecycle): the states and transitions this loop produces.
* [Security model](/deployment/security-model): the trust boundaries in full.