GitHub Docs

Systems Architecture Whitepaper

Deep-dive into raw NVMe auto-discovery, Moka W-TinyLFU metadata, 4 MiB round-robin striping, Direct Block DMA, and async write-back.

1. Executive Summary & Problem Statement

Modern Large Language Model (LLM) and foundation model training and inference workloads are fundamentally constrained by I/O bottlenecks. While GPU compute capabilities have grown exponentially - an NVIDIA H100 delivers up to 3,958 TFLOPS of FP8 compute - the data delivery pipeline remains tethered to remote object stores (Amazon S3, Cloudflare R2, Ceph, MinIO).

The Three AI Storage Bottlenecks

  1. Network Stack Latency & TTFB: Remote S3 GET requests suffer from DNS resolution, TLS 1.3 handshake negotiation, TCP slow-start, and HTTP roundtrip overheads, resulting in Time-To-First-Byte (TTFB) latencies of 35 ms to 130 ms per request.
  2. Shared Storage Contention & GPU Starvation: Even on 200GbE and 400GbE+ cluster fabrics, central storage bandwidth is shared across all nodes and GPUs pulling checkpoints simultaneously - saturating shared storage links, contending with inter-node NCCL traffic, and stalling GPU execution units.
  3. Host CPU & Kernel Page Cache Pollution: Traditional POSIX buffered I/O (read(), fread()) requires multiple memory copies (kernel buffer -> page cache -> user buffer -> CUDA pinned host memory -> GPU VRAM), consuming significant CPU cycles and displacing valuable system memory.

The HyperCache Solution

HyperCache is an autonomous, multi-tier host acceleration engine designed to eliminate network overhead and CPU memory copies. By combining an in-memory W-TinyLFU metadata engine (Moka) with Tier 1: Host RAM / CXL-Ready Shared Memory (/dev/shm) and Tier 2: Parallel Direct Block DMA across Striped NVMe Arrays, HyperCache achieves:

  • Tier 1 (Host RAM / Shared Memory): 50.0 GB/s to 100+ GB/s at DDR5 / CXL memory bus line rate with < 10 us metadata latency.
  • Tier 2 (Striped Local NVMe Pool): Up to ~7.0 GB/s per PCIe Gen 4 NVMe and ~14.0 GB/s per PCIe Gen 5 NVMe line-rate flash bandwidth with < 1 ms resolution.
  • CPU & GPU Agnostic: Runs natively on both GPU compute clusters (with Direct Block DMA / GDS) and CPU-only inference/data-processing nodes with zero CUDA dependencies.
┌─────────────────────────────────────────────────────────────────────────────┐
│                       AI APPLICATION LAYER (PyTorch / Safetensors)          │
│                            torch.load("hypercache://models/llama-3/weights")│
└──────────────────────────────────────┬──────────────────────────────────────┘
                                       │
            1. Metadata Locate Request │ 3. Zero-Copy DMA / Memory Mapping
               (Sub-ms Unix Domain Sock)│    (Zero-Copy Kernel Direct I/O / mmap)
                                       ▼                           ▼
┌──────────────────────────────────────────────┐        ┌─────────────────────┐
│          HYPERCACHE CONTROL PLANE (Rust)     │        │ TIER 1: HOST RAM    │
│  ┌────────────────────────────────────────┐  │        │ (POSIX /dev/shm)    │
│  │ Ephemeral Moka Metadata (W-TinyLFU)    │  │        │ 50 - 100+ GB/s      │
│  └────────────────────────────────────────┘  │        └──────────▲──────────┘
│  ┌────────────────────────────────────────┐  │                   │
│  │ Auto-Discovery & OS Root Protection    │  │                   │ 2. Route & Stripe
│  └────────────────────────────────────────┘  │        ┌──────────┴──────────┐
│  ┌────────────────────────────────────────┐  │        │ TIER 2: STRIPED NVMe│
│  │ Ingestion Engine (Multi-Tier Router)   │──┼───────►│ Drive 0: /dev/nvme0 │
│  └────────────────────────────────────────┘  │        │ Drive 1: /dev/nvme1 │
│  ┌────────────────────────────────────────┐  │        │ 7 - 14+ GB/s / drive│
│  │ Prometheus Exporter (:9110/metrics)    │  │        └─────────────────────┘
│  └────────────────────────────────────────┘  │
└──────────────────────┬───────────────────────┘
                       │
                       │ Cache Miss Fallback (HTTPS/REST or POSIX NFS)
                       ▼
┌──────────────────────────────────────────────┐
│  AUTHORITATIVE BACKENDS (S3 / Ceph / NFS)    │
└──────────────────────────────────────────────┘

2. Core Design Principles

HyperCache enforces strict separation between the Control Plane (metadata resolution, drive safety, admission/eviction lifecycle, telemetry) and the Data Plane (high-throughput zero-copy DMA streaming).

  • Zero External Database Overhead: No Redis, PostgreSQL, SQLite, or ClickHouse daemons required. The entire control plane operates as an ephemeral in-memory state engine backed by an atomic NVMe manifest (< 20 ms warm-boot restore).
  • Hardware Direct Block Access: Bypasses local filesystem overhead (ext4/XFS journal locks, inode allocation, directory tree traversal) by addressing raw block devices at physical 4 MiB striped chunk boundaries - or pre-allocated O_DIRECT filesystem arena files on mounted drives.
  • Strict Sector & Memory Alignment: Leverages Linux kernel O_DIRECT and page-aligned memory buffers (mmap.mmap(-1, len)) to initiate Direct Memory Access (DMA) transactions straight from NVMe flash controllers.

3. Subsystem Architecture & Implementation

3.1 Hardware Auto-Discovery & Safety Engine (discovery.rs)

HyperCache dynamically scans the host system on startup without requiring manual drive configurations:

/sys/class/block/
 ├── nvme0n1 ── (Excluded: Root Mountpoint "/")
 ├── nvme1n1 ── (Accepted: Raw NVMe Device -> Capacity: 3.84 TiB)
 └── nvme2n1 ── (Accepted: Raw NVMe Device -> Capacity: 3.84 TiB)
  1. Sysfs Traversal: Iterates over all block devices in /sys/class/block/ matching allowed drive prefixes (nvme*, sd*).
  2. Root Device & Mount Exclusion: Parses /proc/mounts and /sys/class/block/<dev>/holders to identify the OS root mountpoint (/) and unconditionally excludes the root drive and parent disks.
  3. Filesystem Superblock Signature Detection: Probes device sector headers for magic bytes (EXT4: 0x53EF, XFS: "XFSB", Btrfs: "_BHRfS_M", ZFS). Devices containing active filesystems are safely quarantined.
  4. Capacity Calculation: Reads /sys/class/block/<dev>/size (sectors x 512 bytes) and builds the aggregate multi-drive storage pool.

3.2 In-Memory Metadata Engine (Moka W-TinyLFU)

HyperCache uses an embedded, thread-safe in-process cache built on Moka (moka::future::Cache):

  • Window TinyLFU: Tracks object access frequency using a compact Count-Min 4-bit sketch with dynamic aging.
  • Size-Aware Weigher: Evictions are computed directly on physical chunk payload bytes (EntryWeight = ObjectSizeBytes), ensuring strict adherence to the configured NVMe + RAM capacity limit.
  • Sub-Millisecond Routing: Lookups return the full physical striping layout (chunk indices, physical device paths, exact byte offsets, and chunk lengths) in < 1 ms over Unix Domain Sockets.
// Locate Request (Python -> Rust)
{ "Locate": { "uri": "hypercache://models/llama-3/weights.safetensors", "auto_ingest": true } }

// Locate Response (Rust -> Python)
{
  "status": "Hit",
  "object_size": 5242880000,
  "chunk_size": 4194304,
  "chunk_layout": [
    { "chunk_index": 0, "device": "/dev/nvme1n1", "offset": 4096, "length": 4194304 },
    { "chunk_index": 1, "device": "/dev/nvme2n1", "offset": 4096, "length": 4194304 },
    { "chunk_index": 2, "device": "/dev/nvme1n1", "offset": 4198400, "length": 4194304 },
    { "chunk_index": 3, "device": "/dev/nvme2n1", "offset": 4198400, "length": 4194304 }
  ]
}

3.3 Ingestion & Multi-Drive Striping Engine (striper.rs)

When an object is requested for the first time (Cache MISS), the Ingestion Engine coordinates background fetching and physical block striping:

  • Fixed 4 MiB Chunk Sizing: Aligns with NVMe internal erase block boundaries and flash page allocation units, minimizing write amplification.
  • Round-Robin Multi-Drive Striping: Distributes sequential chunks evenly across all discovered block devices (D_0, D_1, ..., D_{N-1}).
  • Positional Direct Writes: Writes blocks directly using preadv/pwritev or positional file handles with a 4096-byte hardware offset safety margin (device_start_offset_bytes).

3.4 Zero-Copy Data Plane & Dual-Handle Fallback (datapath.py)

The Python Data Plane implements zero-copy DMA transfers while maintaining 100% standard POSIX compatibility via a Dual-Handle File Descriptor Architecture:

  1. Direct DMA Handle (fd_direct with O_DIRECT):
    • Opens block device or filesystem arena with os.O_DIRECT.
    • Executes multi-threaded os.preadv directly into a page-aligned memory buffer (mmap.mmap(-1, length)), saturating local PCIe NVMe lanes (~7 GB/s per Gen 4 drive, ~14 GB/s per Gen 5 drive).
  2. Buffered Handle (fd_buffered with O_RDONLY):
    • Transparently handles arbitrary, unaligned byte slices (e.g., PyTorch header probing f.read(16) or Safetensors metadata JSON slices) without raising EINVAL (Errno 22).

3.5 Asynchronous Write-Back Caching & Egress Engine

  1. Stage & Direct Write: The PyTorch client queries the local daemon (StageWrite) to obtain striped NVMe chunk allocations. The DirectBlockWriter writes chunks directly to raw disk sectors using positional os.pwrite.
  2. Instant GPU Unblock (CommitWrite): Once local NVMe striping completes (and optional k=2 peer replication confirms), the client issues CommitWrite. HyperCache updates the Moka metadata manifest and immediately unblocks the PyTorch training loop.
  3. Non-Blocking Background Egress: The daemon spawns an asynchronous Tokio task that reads the chunks directly from physical disks and streams them to authoritative S3 or NFS storage without consuming GPU cycles.

4. Flash Parallelism & NVMe Queue Depth Mechanics

An enterprise NVMe SSD is an internally distributed array of 8 to 16 parallel NAND flash channels, each controlling multiple dies:

  • Queue Depth = 1 (Synchronous Read): The SSD controller waits for a single 4KB/64KB transaction to complete across 1 flash channel. The other 15 channels remain idle, capping read bandwidth at ~500-700 MB/s.
  • Queue Depth >= 16-32 (Parallel DMA): HyperCache dynamically spawns a worker thread pool sized to the host CPU cores (min(64, max(16, CPU_Cores * 4))). By dispatching parallel chunk DMA requests simultaneously across all available flash channels and all striped drives, HyperCache saturates the full PCIe bus width.

5. Hardware Throughput & Linear Scaling Profile

By shifting I/O from shared cluster fabrics (200GbE / 400GbE+) to local PCIe NVMe lanes and DDR5/CXL host memory channels, HyperCache scales linearly with attached hardware:

Storage Tier / Data PathPer-Device / Per-Link LimitNode ThroughputMetadata / TTFB Latency
Shared Storage over 200GbE / 400GbE+ (Concurrent Multi-Node Load)25 - 50+ GB/s wire limit (shared across hosts)~1.5 - 4.0 GB/s per node (under cluster load)35 - 130 ms
Tier 2: PCIe Gen 4 NVMe Pool (O_DIRECT / GDS)~7.0 GB/s per NVMe SSD~28 GB/s (4x NVMe) / ~56 GB/s (8x NVMe)< 1 ms
Tier 2: PCIe Gen 5 NVMe Pool (O_DIRECT / GDS)~14.0 GB/s per NVMe SSD~56 GB/s (4x NVMe) / ~112 GB/s (8x NVMe)< 1 ms
Tier 1: Host RAM / POSIX SHM (DDR5 / CXL)50 - 100+ GB/s per stream300 - 460+ GB/s (8-12 channel DDR5 bus)< 10 us