Python Data Plane & Framework Integrations

Installing the Python Package (`hypercache-io`)

Install the Python package from PyPI into your training or inference environment. It exposes the hypercache module (import hypercache):

# Install from PyPI (exposes `import hypercache` in Python)
pip install hypercache-io

# Optional: On NVIDIA GPU nodes, install GPUDirect Storage (cuFile / KvikIO) support
pip install "hypercache-io[gpu]" --extra-index-url https://pypi.nvidia.com

# Or install in editable mode from a local repository checkout
pip install -e ./client-py

URI Addressing & Backend Routing

HyperCache routes requests automatically based on the URI scheme passed to Python (fsspec.open, torch.load) or the CLI (./hypercache preload):

URI Format	Target Backend	How HyperCache Resolves It
`hypercache://s3/<bucket>/<key>`	Amazon S3 / S3-Compatible	Fetches/writes `<key>` in explicit `<bucket>` using `s3_endpoint` from `hypercache-agent.toml`.
`s3://<bucket>/<key>`	Amazon S3 / S3-Compatible	Standard S3 URI alias; routes identically to `hypercache://s3/<bucket>/<key>`.
`hypercache://<key>`	Default Configured Backend	Uses `authoritative_backend` (`"s3"` + default `s3_bucket`, or `"nfs"` + `nfs_mount_path`) from `hypercache-agent.toml`.
`hypercache://file/<abs/path>`	POSIX / Shared NFS	Reads/writes directly to `/<abs/path>` on the local or NFS-mounted filesystem.
`file:///<abs/path>` or `nfs://<rel/path>`	POSIX / Shared NFS	Reads/writes to `/<abs/path>` (or `<nfs_mount_path>/<rel/path>`).

1. Drop-In PyTorch Model Loading (`torch.load`)

Importing hypercache automatically registers the hypercache:// protocol with Python's fsspec:

import fsspec
import torch
import hypercache  # Auto-registers the hypercache:// protocol

# Streams weights from local striped NVMe/RAM cache (auto-fetches from S3 on first miss)
checkpoint_uri = "hypercache://s3/my-bucket/models/llama-3-8b/weights.pt"

with fsspec.open(checkpoint_uri, "rb") as f:
    state_dict = torch.load(f, map_location="cuda:0")

model.load_state_dict(state_dict)

2. Native Safetensors & vLLM Zero-Copy Weight Loader

Stream .safetensors files directly into GPU or host tensors with optional selective tensor filtering and cache pinning (pinned=True):

from hypercache import (
    HyperCachevLLMWeightLoader,
    load_safetensors_state_dict,
    save_safetensors_state_dict,
)

# Load all (or selected) tensors from a .safetensors shard and pin in local cache
state_dict = load_safetensors_state_dict(
    "hypercache://models/llama-3-70b/model-00001-of-00008.safetensors",
    device="cuda:0",
    pinned=True,
)

# Stream multi-shard weights directly into a vLLM model runner
vllm_loader = HyperCachevLLMWeightLoader(pin_base_weights=True)
for name, tensor in vllm_loader.iter_shard_tensors(shard_uris, device="cuda:0"):
    pass

3. Training Checkpointing: Async Write-Back vs. Sync Write-Through

import fsspec
import torch
import hypercache

checkpoint_uri = "hypercache://s3/my-bucket/checkpoints/epoch_1.pt"

# Mode A: Asynchronous Write-Back (Default - Maximum Training Speed)
# Stages to local RAM/NVMe at hardware line rate and unblocks GPUs immediately;
# background worker flushes to S3/NFS asynchronously.
with fsspec.open(checkpoint_uri, "wb") as f:
    torch.save(model.state_dict(), f)

# Mode B: Synchronous Write-Through (Strict Remote Durability)
# Blocks f.close() until confirmed written to both local cache AND remote S3/NFS.
with fsspec.open(checkpoint_uri, "wb", write_through=True) as f:
    torch.save(model.state_dict(), f)

4. PyTorch Distributed Checkpoint (FSDP / DeepSpeed / DCP)

Save and restore per-rank distributed checkpoints at local hardware speed:

from hypercache import HyperCacheDCPStorageReader, HyperCacheDCPStorageWriter

# Save per-rank FSDP shard to local NVMe/RAM with background S3 sync
writer = HyperCacheDCPStorageWriter(
    "hypercache://s3/my-bucket/checkpoints/step_1000",
    rank=rank,
    world_size=world_size,
)
writer.save_rank_shard(model.state_dict())

# Restore per-rank FSDP shard via Direct Block DMA
reader = HyperCacheDCPStorageReader(
    "hypercache://s3/my-bucket/checkpoints/step_1000",
    rank=rank,
    world_size=world_size,
)
state_dict = reader.load_rank_shard()

5. Layer-by-Layer Pipelined Model Loader (Compute / I/O Overlap)

Start forward-pass compute on Layer 0 immediately while subsequent layers prefetch in the background:

from hypercache.pipeline import HyperCachePipelinedLoader

loader = HyperCachePipelinedLoader(prefetch_depth=2)

# Layer 0 arrives immediately; Layers 1..N prefetch in background threads during GPU compute
for layer_idx, layer_bytes in loader.stream_layers(layer_uris):
    run_layer_forward_pass(layer_idx, layer_bytes)

6. KV-Cache Prefix Block Store (LMCache / Dynamo KVBM Ready)

Offload and retrieve prompt prefix KV-cache blocks on striped local NVMe (or optional Host RAM):

from hypercache import HyperCacheKVStore

kv_store = HyperCacheKVStore(default_tier="nvme")
block_key = kv_store.compute_prefix_hash(
    token_ids=[128000, 9906, 1917],
    model_id="llama-3.1-70b",
    layer_idx=0,
)

# Store KV-cache block into local striped NVMe
kv_store.put_kv_block(block_key, kv_tensor, tier="nvme", pinned=True)

# Retrieve KV-cache block directly into pre-allocated GPU tensor
kv_store.get_kv_block(block_key, out_tensor=gpu_kv_buffer)

7. Multi-Node Dataset Sharding & Distributed DataLoader

Stream consistent-hashed dataset shards directly into PyTorch DataLoader workers on each node:

import fsspec
from torch.utils.data import DataLoader, Dataset
import hypercache

class ShardedTokenDataset(Dataset):
    def __init__(self, shard_uris):
        self.shard_uris = shard_uris

    def __getitem__(self, idx):
        with fsspec.open(self.shard_uris[idx], "rb") as f:
            return f.read()

dataset = ShardedTokenDataset(["hypercache://datasets/tokens/shard_000.bin"])
loader = DataLoader(dataset, batch_size=16, num_workers=4)

Installing the Python Package (hypercache-io)