GitHub Docs

Operator Quickstart & Cluster Setup

Step-by-step installation, single-node zero-port startup, and multi-node GPU cluster setup with the Central Controller.

HyperCache provides host-based caching and acceleration for distributed AI training and inference clusters. It eliminates model loading bottlenecks from S3 and NFS by caching weights across local striped NVMe/SSD pools (and optional Tier 1 Host RAM) and streaming directly into GPU memory via Direct Block DMA / GPUDirect Storage (GDS).

System Architecture

+===================================================================================+
|                                CENTRAL CONTROLLER HOST                            |
|                                                                                   |
|  [ Operator / Admin CLI ]                                                         |
|   ./hypercache-admin nodes                                                        |
|   ./hypercache-admin preload <uri> --nodes all                                    |
|   ./hypercache-admin purge --nodes all                                            |
|         │                                                                         |
|         │ (Local Loopback http://127.0.0.1:9120 - Zero Config Admin Access)       |
|         ▼                                                                         |
|  +-----------------------------------------------------------------------------+  |
|  | HyperCache Controller Daemon (Port 9120)                                    |  |
|  | - Fleet Registry: Tracks active nodes, NVMe capacities, and health          |  |
|  | - Fleet Dispatcher: Coordinates parallel preload / evict / purge            |  |
|  +-----------------------------------------------------------------------------+  |
+===================================================================================+
                                         │
                  ┌──────────────────────┴──────────────────────┐
                  │ Authenticated Directives (Bearer Token)     │ Node Heartbeats (5s)
                  ▼                                             ▼
+=============================================+ +=============================================+
|             GPU WORKER NODE 01              | |             GPU WORKER NODE 02              |
|                                             | |                                             |
|  +---------------------------------------+  | |  +---------------------------------------+  |
|  | Agent Fleet API (Port 9115)           |  | |  | Agent Fleet API (Port 9115)           |  |
|  | - Receives preload/evict directives   |  | |  | - Receives preload/evict directives   |  |
|  +---------------------------------------+  | |  +---------------------------------------+  |
|  +---------------------------------------+  | |  +---------------------------------------+  |
|  | HyperCache Worker Engine              |  | |  | HyperCache Worker Engine              |  |
|  | - Moka W-TinyLFU Metadata in RAM      |  | |  | - Moka W-TinyLFU Metadata in RAM      |  |
|  | - Striped Raw NVMe Pool (/dev/nvme*)  |  | |  | - Striped Raw NVMe Pool (/dev/nvme*)  |  |
|  | - Direct Block DMA / GDS to GPU VRAM  |  | |  | - Direct Block DMA / GDS to GPU VRAM  |  |
|  | - Prometheus Exporter (Port 9110)     |  | |  | - Prometheus Exporter (Port 9110)     |  |
|  +---------------------------------------+  | |  +---------------------------------------+  |
+=============================================+ +=============================================+

1. Installation & Build

HyperCache compiles into two native, zero-dependency Rust executables (./hypercache and ./hypercache-admin) plus the Python client package (hypercache-io).

Option A: Automated One-Step Install (install.sh)

Run on your machines (Controller and GPU worker nodes):

# One-liner bootstrap install (fetches pre-built release binaries with source fallback):
curl -fsSL https://hypercache.sh/install.sh | bash

# Or clone and run locally:
git clone https://github.com/looelabs/hypercache.git
cd hypercache
./install.sh

Option B: Pre-Built Release Binaries, PyPI & Kubernetes

If you only want the pre-compiled Linux binaries + the PyPI Python library (or are deploying on Kubernetes):

# 1. Download pre-built Linux x86_64 binaries (hypercache & hypercache-admin)
curl -fsSL https://github.com/looelabs/hypercache/releases/latest/download/hypercache-linux-x86_64.tar.gz | tar -xz

# 2. Install the Python library into your PyTorch / vLLM environment from PyPI
pip install hypercache-io

# Optional: On NVIDIA GPU nodes, install GPUDirect Storage (cuFile / KvikIO) support
pip install "hypercache-io[gpu]" --extra-index-url https://pypi.nvidia.com

# 3. Or deploy the Controller + GPU Worker DaemonSet on Kubernetes
kubectl apply -f deploy/kubernetes/hypercache.yaml

Option C: Manual Source Build (./build.sh + pip)

If you already have Rust (1.75+) and Python (3.9+) installed:

# Compile release binaries (./hypercache and ./hypercache-admin)
./build.sh

# Install Python client from local checkout
pip install -e ./client-py

2. Pre-Flight Host Diagnostics (./hypercache check)

Before starting the daemon on a worker node, run check (or doctor) to verify your configuration syntax, drive permissions, and O_DIRECT alignment support:

./hypercache -c configs/hypercache-agent.toml check

If any check fails or warns, ./hypercache check prints the exact remediation command (such as enabling storage_backing_mode = "filesystem" if your drives are already formatted and mounted).

3. Setting Up a Single-Node GPU Server

On a standalone GPU workstation or cloud VM, edit configs/hypercache-agent.toml with your S3/NFS settings and start the daemon. By default, it opens zero network TCP ports and listens exclusively on the local Unix Domain Socket (/run/hypercache/hypercache.sock):

# Start worker daemon on discovered local NVMe drives
./hypercache -c configs/hypercache-agent.toml start

# Optional: Start with Tier 1 Host RAM caching enabled (50-100+ GB/s)
./hypercache -c configs/hypercache-agent.toml start --ram

# Optional: Enable Prometheus metrics on http://127.0.0.1:9110/metrics
./hypercache -c configs/hypercache-agent.toml start --metrics

4. Setting Up a Multi-Node GPU Cluster

Step A: Start the Central Controller Node

Configure configs/hypercache-controller.toml on the Controller host:

# Bind controller service to private cluster network IP or 0.0.0.0
controller_bind_address = "0.0.0.0"
controller_port = 9120

# Cluster Security Token for authenticating remote nodes and remote API access
# Can also be set via: export HYPERCACHE_AUTH_TOKEN="sec-cluster-secret-key"
cluster_auth_token = "sec-cluster-secret-key"

# Optional: Enable 1/N S3 origin pull + peer-to-peer chunk exchange
p2p_preload_enabled = true

Start the Controller daemon:

./hypercache-admin -c configs/hypercache-controller.toml controller

Step B: Start GPU Worker Nodes

Configure configs/hypercache-agent.toml on each GPU node to point to the Controller:

node_id = "gpu-node-01"

[node_labels]
gpu = "h100"
pool = "training"
zone = "us-east-1a"

controller_url = "http://10.0.0.10:9120"
cluster_auth_token = "sec-cluster-secret-key"

agent_enabled = true
agent_bind_address = "0.0.0.0"
agent_http_port = 9115

Start the worker agent:

./hypercache -c configs/hypercache-agent.toml start --agent

Step C: Orchestrate the Fleet from the Controller

When run on the Controller host (127.0.0.1), ./hypercache-admin requires no flags, no tokens, and no IPs:

# Inspect cluster topology and aggregate NVMe pool capacity
./hypercache-admin nodes

# Pre-warm a model across all nodes
./hypercache-admin preload hypercache://s3/models/llama-3-70b/weights.safetensors --nodes all

# Pre-warm targeting specific node labels
./hypercache-admin preload hypercache://s3/models/llama-3-70b/weights.safetensors --selector "gpu=h100,pool=training"

# Partition and pre-load a sharded training dataset using Rendezvous Hashing
./hypercache-admin preload-batch --prefix "s3://datasets/tokens/shard_" --sharded --nodes all

# Evict a specific model across all nodes
./hypercache-admin evict hypercache://s3/models/llama-2-7b/weights.safetensors --nodes all

# Purge all cached objects across the fleet
./hypercache-admin purge --nodes all

5. Local Worker CLI Operations (./hypercache)

Run directly on any individual GPU worker node:

TaskCommand
Start Daemon./hypercache -c configs/hypercache-agent.toml start
List Cached Models./hypercache ls (or ./hypercache ls --json)
Pre-load Model Locally./hypercache preload hypercache://s3/models/llama-3/weights.safetensors
Batch Pre-load from File./hypercache preload --file checkpoints.txt
Evict Model Locally./hypercache rm hypercache://s3/models/llama-3/weights.safetensors
Refresh Model from Source./hypercache refresh hypercache://s3/models/llama-3/weights.safetensors
Purge Local Cache./hypercache purge
Inspect Local SSDs./hypercache inspect
Check Local Health./hypercache status
Run DMA Benchmark./hypercache bench --size-mb 5000

6. PyTorch Integration

Install the Python client (hypercache-io) on your GPU worker nodes or inside your training/inference containers:

# Install from PyPI (exposes `import hypercache` in Python)
pip install hypercache-io

# Optional: On NVIDIA GPU nodes, install GPUDirect Storage (cuFile / KvikIO) support
pip install "hypercache-io[gpu]" --extra-index-url https://pypi.nvidia.com

# Or install from the local repository
pip install -e ./client-py

Load checkpoints directly from striped NVMe into GPU memory:

import fsspec
import torch
import hypercache  # Hooks hypercache:// protocol

# Load weights directly into GPU VRAM via Direct Block DMA
checkpoint_uri = "hypercache://s3/models/llama-3-8b/weights.pt"

with fsspec.open(checkpoint_uri, "rb") as f:
    state_dict = torch.load(f, map_location="cuda:0")

model.load_state_dict(state_dict)

7. Prometheus & Grafana Monitoring

HyperCache exports real-time metrics on port 9110 (/metrics) when enabled (metrics_enabled = true or --metrics).

Prometheus Scrape Configuration (prometheus.yml)

scrape_configs:
  - job_name: 'hypercache-workers'
    scrape_interval: 1s
    static_configs:
      - targets:
          - '10.0.0.11:9110'
          - '10.0.0.12:9110'

Key Metrics Exported

  • hypercache_bandwidth_mbps: Real-time DMA disk read throughput.
  • hypercache_hit_ratio_percent: Cache hit percentage.
  • hypercache_storage_utilization_percent: Striped NVMe pool utilization.
  • hypercache_storage_used_bytes: Total cached bytes in the NVMe pool.
  • hypercache_s3_egress_bytes_saved_total: Cumulative egress data avoided.
  • hypercache_gpu_idle_time_saved_seconds: Cumulative GPU idle time saved.

8. Network Security & Production Hardening Checklist

  1. Private Subnets Only: Ensure controller_bind_address and agent_bind_address are bound to your private cluster network (e.g. 10.0.0.x), never public IPs.
  2. Bearer Token Authentication: Set cluster_auth_token in configs/hypercache-agent.toml / configs/hypercache-controller.toml or export HYPERCACHE_AUTH_TOKEN="your-cluster-secret".
  3. Firewall Rules (Security Groups):
    • Port 9120 (Controller): Open only to GPU Worker node IPs.
    • Port 9115 (Agent API): Open only to Controller IP.
    • Port 9110 (Prometheus): Open only to Prometheus scraper IP or 127.0.0.1.