Architecture - hiloop

hiloop is an organization-aware control plane over a Kubernetes sandbox runtime, with telemetry as a first-class plane and proof-bound model access as an explicit target gap. The same current components ship for hosted and self-managed deployments.

Components

Component	Responsibility
API edge	Authenticate every request, resolve the credential to an organization, user, and scope, and stamp that identity for backend services. Services trust only edge-stamped identity.
Control-plane API	Validate requests, own the product records (sandboxes, snapshots, volumes, secrets, projects, runs, workloads, keys), and render admitted sandboxes into runtime objects. Errors use one flat `{code, message, request_id}` envelope with stable codes.
Sandbox runtime	A Kubernetes controller (built on the open-source agent-sandbox controller) reconciles each sandbox into a pod in a dedicated workload namespace. A status mirror converges observed runtime state back into the product record, so `state` reports evidence, not intent.
Model gateway target	External Envoy data plane plus a Hiloop authorization/control service for proof-bound model access. This sandbox path is not deployed; the existing Rust body proxy is operator-only. See model gateway.
Telemetry	The telemetry gateway ingests captured events under the caller’s organization; the store serves read-only SQL over them. See run-scoped observability.
Web console	Browse projects, runs, and telemetry in the browser.

Sandbox isolation

Sandboxes run arbitrary, untrusted images, so isolation is layered and enforced outside the workload:

Every sandbox pod runs under gVisor (the runsc runtime class), a user-space kernel between the workload and the host. A cluster admission policy pins this invariant: a sandbox that does not request gVisor, or requests host namespaces or host-path mounts, is refused at the Kubernetes API server.
The dedicated workload namespace enforces the Kubernetes baseline pod security standard as defense in depth, and sandbox pods carry no Kubernetes service-account token.
A namespace-wide egress policy allows DNS, the public internet, and the API’s dedicated capture listener. The workload has no capture proof, so that listener rejects it; all other cluster services, private address ranges, and the cloud metadata endpoint are unreachable. See network access.
Admission fails closed: a request the deployment cannot enforce exactly (an unavailable storage class, mount transport, or injection path) is rejected with unsupported_capability, never silently weakened.

Sandbox identity stays opaque: the public sandbox id never exposes a pod name, container id, or node name.

Shell access

hiloop sandbox ssh <name> opens an interactive shell in a sandbox. Your end is stock OpenSSH: a normal ssh client, with no jump host, no public port on the sandbox, and no SSH credential for you to create, install, or rotate. Inside the sandbox, the server is a single static binary that runs in your workload container, so a shell starts in your image’s filesystem as your image’s own user. It is started by a lifecycle hook and mounted read-only; your image’s entrypoint runs unmodified and your container gains nothing else. The ssh client’s ProxyCommand re-invokes the hiloop CLI, which opens one authenticated HTTPS connection to the API and upgrades it to a byte stream. The API is the only authority on the path. It authorizes the caller against the organization that owns the sandbox, confirms the sandbox is running, resolves that sandbox’s currently running instance, opens a connection to it, and splices the bytes. The check runs on every connect, and nothing the client sends chooses the target.

┌─ your machine ─────────────────────────────────────────────────────────┐
│  ssh (stock OpenSSH)  ──ProxyCommand──▶  hiloop CLI                    │
│                                                                        │
│  holds for the life of the session: the sandbox's pinned host public   │
│  key, and a one-shot client private key (mode 0600, removed at exit)   │
└────────────────────────────────────┬───────────────────────────────────┘
                                     │  one HTTPS connection, upgraded to
                                     │  a byte stream, org-scoped API key
                                     ▼
┌─ hiloop API ── the only place the access decision is made ─────────────┐
│  1. authenticate the credential                                        │
│  2. does the caller's organization own this sandbox?  no ─▶ not found  │
│  3. is the sandbox running?                           no ─▶ refused    │
│  4. resolve the running instance, connect to it, splice the bytes      │
└────────────────────────────────────┬───────────────────────────────────┘
                                     │  connection to the running instance
                                     ▼
┌─ your sandbox ─────────────────────────────────────────────────────────┐
│  the endpoint, running IN your workload container: your filesystem,    │
│  your image's user, your WORKDIR                                       │
│                                                                        │
│  inside: the host private key, and one authorized public key           │
│  never inside: the client private key, or any hiloop credential        │
└────────────────────────────────────────────────────────────────────────┘

Authorization uses the same organization-scoped API credential as every other hiloop command. There is no separate SSH credential and nothing to rotate. A caller from another organization gets exactly the same answer as for a sandbox that does not exist. hiloop mints an SSH key pair per sandbox. The host private key and a single authorized public key are placed inside the sandbox; the matching client private key never is. It is released only to a caller the API has already authorized for that sandbox, and the CLI writes it to a temporary file (mode 0600) that it removes when the session ends. The CLI pins the sandbox’s host key into a dedicated strict known_hosts entry and asserts it as authoritative ssh command-line options, so neither your own -o StrictHostKeyChecking=no nor a Host * block in ~/.ssh/config can downgrade it. Two consequences worth knowing: you never see a trust-on-first-use prompt, because the key arrives over the authenticated API rather than from the network, and a recreated sandbox presents a different host key, so a stale connection attempt fails rather than silently reaching a different machine. See interactive access for the command forms, what a session may and may not do, and the current filesystem limitation.

Sandbox sources and snapshots

Every sandbox starts from one source: an unmodified OCI image (the platform default image when the create names none) or a snapshot. Snapshots capture disk state, report truthful durability (local until remote replication is confirmed, then replicated), and record lineage so branched sandboxes stay attributable to their common ancestor. The model is covered in snapshots and branching.

Credentials

No hiloop-managed credential is injected into a sandbox. Registered secrets remain write-only, envelope-encrypted control-plane data, and a deployment without the complete proof-bound release path refuses bindings at admission (see managing secrets). The approved model gateway target uses an external Envoy data plane plus a Hiloop control service to validate workload proof and select the authorized credential outside the guest. That sandbox path is not deployed today. Direct provider access is not a supported fallback, so clean sandbox model access remains unavailable.

Hosted and self-managed

Both deployment styles install the same charts and run the same contract; what changes is ownership:

Area	Hosted	Self-managed
Kubernetes and nodes	hiloop-operated	customer-operated
Image registry	hiloop registry	customer mirror, same digests
Databases and object storage	hiloop-operated	customer-selected
Ingress, DNS, and PKI	hiloop-operated	customer trust roots

See deployment models and the Helm install guide.

​Components

​Sandbox isolation

​Shell access

​Sandbox sources and snapshots

​Credentials

​Hosted and self-managed