Skip to main content
hiloop installs into a Kubernetes cluster you own with one Helm chart. This page covers the components, the cluster contract they depend on, and the substrate that runs sandboxes. For how a sandbox is represented and isolated once it is running, see Kubernetes architecture.

​
What you install

One umbrella chart composes the platform from per-component subcharts, each behind its own toggle, plus a shared library chart of templates.

​
What the chart needs from your cluster

​
Platform

  • Kubernetes 1.31 or later, and Helm 3.18 or later. The chart’s preflight check fails explicitly on an older cluster rather than rendering something that half works.
  • A CNI that enforces NetworkPolicy. This is a trust boundary, not an optimization, and it carries two separate controls. It is what keeps sandbox workloads off your internal network and off the node metadata endpoint. It is also what fences the backend services: the API and the telemetry gateway take the caller’s organization from the identity headers your edge stamps and cannot tell a stamped header from a forged one, so the policy restricting who may reach those Services directly is the only thing standing between any pod in the cluster and any organization’s data. A CNI that accepts policy objects and ignores them leaves both controls entirely unenforced while everything still appears healthy. The install proves enforcement before it installs any component: a pre-install gate dials a port its own NetworkPolicy denies, and refuses to install if that port keeps answering. It gives a policy that is still converging time to land, so only a port that answers throughout fails the install. The one step that runs ahead of it is the migration gate, which holds the release’s lowest weight, so a cluster rejected here is left with the schema applied and nothing else. Treat a failure there as a cluster prerequisite, not a chart problem.
  • A StorageClass for the components that keep local state, including the telemetry gateway’s write-ahead log. Without one those claims stay pending indefinitely.
  • Namespaces created in advance, including the release namespace and the sandbox namespace. Parts of the chart render objects into the sandbox namespace, so installing before it exists fails.

​
Database

  • PostgreSQL 17 or later, with TLS, reachable from the cluster. The database is always external; the chart never ships one.
  • Two logical databases, one for the control plane and one for telemetry, both created by the migration gate.
  • Credentials are supplied as connection strings, with the TLS mode carried in the string. Cloud IAM database authentication is a hosted-only path.
  • Migrations run as a blocking pre-install and pre-upgrade hook at the release’s lowest weight, so schema is in place before any component starts. Schema changes are expand and contract, so a release is compatible with the previous one. Do not pass --atomic on a first install: a rollback of a failed first install tears down the very namespace and hook objects the diagnosis needs.

​
Object storage

Telemetry and sandbox workspaces are backed by object storage you name. Any S3-compatible endpoint works, including MinIO, so an install can stay entirely inside your own network. Configuration takes an explicit endpoint, region, whether plain HTTP is allowed, a credentials secret, and a trust bundle for a private certificate authority.

​
Ingress and TLS

You front the platform with your own ingress controller and your own certificates. Gateway API routing is a hosted-only convenience, disabled in the self-managed profile. Two requirements are easy to miss:
  • Naming your ingress as a network-policy peer is a required input. The install does not render until you name it, because the components’ own policies would otherwise deny your ingress controller.
  • Three paths must be reachable unauthenticated for workload identity to work: the OpenID configuration document, the JWKS document, and the workload-identity token endpoint.

​
Custom resource definitions

Helm does not manage custom resource definition lifecycle, so the sandbox resource definitions are applied separately with a server-side apply, both before the first install and before every upgrade. Apply the full set. The controller patches every definition’s conversion webhook configuration at startup and skips an absent one silently, so a partial set produces an install that looks complete and fails later.

​
Secrets you create

Self-managed installs have no secret-operator dependency, so database credentials, object-store credentials, certificate authority bundles, model-provider credentials, and the workload-identity signing keyset are all secrets you create before installing. Validate them by content with the chart’s preflight check, which reads the actual values rather than checking that keys exist. The first API key is generated into a secret at install time and is the only copy. Back it up.

​
The sandbox node substrate

Sandbox nodes are a distinct pool, because they carry a different kernel contract from the rest of the cluster. Nodes are labeled and tainted. Sandbox nodes carry a label marking them as sandbox capacity and a permanent taint keeping general workloads off. The gVisor runtime class carries the matching node selector and toleration, so requesting the runtime class is what places a pod on the pool. Nothing else needs to know the topology. A readiness gate proves the runtime before the node accepts work. New nodes come up with a startup taint that the gVisor runtime class deliberately does not tolerate, so no sandbox can be scheduled there yet. A privileged daemon on each node then verifies the substrate: that the expected runtime binary and its container-runtime shim are present and the correct version, that the container runtime’s configuration actually registers the runtime handler, that the local storage pool is healthy, and finally that a real probe pod pinned to that node runs under gVisor through the kubelet’s own runtime resolution. Only when every check passes does it remove the startup taint. The gate stays running. After the boot-only probe pod succeeds, bounded host checks continue periodically and re-apply the taint if the substrate regresses. That stops new admissions onto a broken node without disturbing pods already running there, and avoids consuming pod slots for recurring probes. A node stuck behind the gate is visible as a daemon that is not ready. The runtime class charges its own overhead. gVisor’s supervisor and file-system gateway consume host memory per pod, so the runtime class declares a fixed per-pod memory overhead and the scheduler accounts for it. Sandbox capacity math is therefore honest without operators modeling it by hand. Node-local fast storage. The workspace daemon keeps its bounded cache on each sandbox node’s local filesystem. The default uses the node’s provisioned disk; an operator can point the cache path at dedicated local media when the cluster provides it. Durability never depends on that cache. Per-node daemons are privileged, deliberately. The workspace daemon plumbs block devices and mounts filesystems for other pods to consume. The checkpoint daemon drives the sandbox runtime against the container runtime’s root-owned state directory. Neither job is expressible without privilege, so the charts refuse to render them unprivileged rather than shipping a configuration that fails at runtime in a confusing way. Both run at node-critical priority, mount no service account token, and are confined to the sandbox node pool.

​
The workspace storage layer

A sandbox’s durable filesystem does not live on the node and is not a per-sandbox network volume. It lives in your workspace store, and the per-node daemon presents it to the sandbox as /workspace.
  • Hydration is lazy. A fresh workspace materializes from a published base filesystem image, and data is fetched from the store on demand rather than copied up front, so starting a sandbox does not wait for a whole filesystem to be provisioned.
  • The node keeps a cache. Reads are served from a node-local cache. Cached data expires on a time-to-live, and a disk-pressure loop evicts oldest-first once the cache filesystem crosses a usage watermark, with deletion delayed briefly so data still in use is not pulled out from under a running sandbox.
  • Writes replicate back to the store continuously. Durability comes from that replication, not from the node, which is what lets a sandbox come back on a different node after an interruption with its files intact. Replication backlog is bounded: sustained writing beyond what the store can absorb applies backpressure rather than growing an unbounded local backlog.
  • Forking a workspace is a manifest copy. A workspace is described by a manifest of content-addressed chunks, so branching one means copying that description and sharing the underlying data, rather than duplicating bytes. The cost of a fork is therefore governed by the size of the description and not by the size of the filesystem, and the parent and child diverge copy-on-write from there.
Because there is no per-sandbox network volume, two limits that normally shape a sandbox platform do not apply: sandbox density on a node is not bounded by a per-node volume attachment ceiling, and a sandbox is not pinned for its lifetime to the availability zone its volume was created in.

​
Storage portability

The workspace layer speaks three URL schemes: S3, Google Cloud Storage, and a local filesystem scheme used as a deterministic test seam rather than in production. S3-compatible endpoints are addressed by naming the endpoint explicitly and requesting path-style addressing:
Scope a bucket you share with something else using a prefix query parameter rather than a path segment, since the bucket is taken from the URL’s host:

​
Scheduling and density

What actually bounds sandboxes per node, in the order the limits bite:
  1. The kubelet’s maximum pods per node, configured at 100 on the sandbox pool. One sandbox is one pod, so this is the first ceiling.
  2. Node memory, after subtracting what is reserved for the system, the local storage pool’s cache, and the per-pod gVisor overhead.
  3. A fleet-wide object-count quota on sandbox resources, enforced transactionally at the API server, behind the per-organization quotas the API applies at admission.
Sandbox capacity is spot-first: the node pools request interruptible capacity and let the provisioner choose instance types within a family and size floor. Interruption is treated as normal, which is the reason durability lives in the workspace store rather than on the node.

​
Where data and credentials live

The object stores and the database are ones you name, in infrastructure you operate. Workspace data, volume data, memory checkpoints, telemetry, and the control-plane records all live there. The approved target keeps platform-managed model-provider credentials outside sandbox environments, exec requests, and Kubernetes Secrets. An external Envoy data plane applies decisions from a Hiloop proof/policy control service at the last boundary before the provider call. That sandbox path is not deployed today; the existing Rust body proxy is operator-only. Direct provider access is not a supported fallback, so clean sandbox model access remains unavailable. Note that a sandbox’s default egress posture allows the public internet, so platform storage staying in your infrastructure is not the same as a workload being unable to send data out. That posture and its consequences are described under where your data lives, with the enforcement detail in network requirements.