Configuration Reference (TOML & Environment)

All HyperCache configuration files live in configs/ and use strongly-typed TOML syntax. Passing -c <path> to ./hypercache or ./hypercache-admin loads the specified configuration file.

Which Config File Should I Use?

Configuration File	Where It Runs	Binary Used	Purpose
configs/hypercache-agent.toml	Every GPU / CPU Worker Node	./hypercache	Configures local NVMe discovery (raw block drives or filesystem arenas), authoritative S3/NFS credentials, write-back/write-through behavior, and optional registration with a Central Controller.
configs/hypercache-controller.toml	Central Controller / Head Node (Multi-node clusters only)	./hypercache-admin	Configures the cluster coordinator bind IP/port (9120), shared Bearer token authentication (cluster_auth_token), and Peer-to-Peer fleet distribution (p2p_preload_enabled). Not needed for single-node setups.

---

1. Choosing Your Storage Backing Mode

HyperCache supports two ways to back your local NVMe cache pool depending on your environment:

A. Raw Block Device Mode (storage_backing_mode = "raw_block" or "auto")

Best for bare-metal GPU servers or Cloud VMs with dedicated, unmounted local NVMe drives (e.g., /dev/nvme1n1, /dev/nvme2n1). HyperCache automatically discovers unmounted drives, excludes your OS root disk (/), and stripes 4 MiB chunks directly across raw disk sectors with zero filesystem overhead.

storage_backing_mode = "auto"
device_discovery_mode = "auto"
exclude_root_device = true

B. Filesystem Direct I/O Arena Mode (storage_backing_mode = "filesystem")

Best for Kubernetes pods (emptyDir / local-storage PVCs), shared servers where drives are already mounted with ext4/XFS, or native NVIDIA GPUDirect Storage (cuFile). HyperCache pre-allocates aligned arena files inside the specified directories and opens them with O_DIRECT - requiring zero raw block privileges.

storage_backing_mode = "filesystem"
filesystem_cache_paths = ["/mnt/nvme0/hypercache", "/mnt/nvme1/hypercache"]
filesystem_arena_size_bytes = 107374182400 # 100 GiB arena capacity per directory

---

2. Worker Node Configuration (configs/hypercache-agent.toml)

Deployed on every GPU or CPU worker node running ./hypercache -c configs/hypercache-agent.toml start:

# ==============================================================================
# 1. STORAGE POOL & TIERING
# ==============================================================================
storage_backing_mode = "auto"              # "auto", "raw_block", or "filesystem"
device_discovery_mode = "auto"             # "auto" (scan /sys/class/block) or "manual"
include_partitions = true                  # Include unmounted partitions
device_prefixes = ["nvme", "sd"]
exclude_root_device = true                 # Always protect OS root disk (/)
chunk_size_bytes = 4194304                 # 4 MiB stripe chunk size
device_start_offset_bytes = 4096           # 4 KB alignment safety margin

# If using storage_backing_mode = "filesystem", specify mounted XFS/ext4 directories:
filesystem_cache_paths = []
filesystem_arena_size_bytes = 10737418240  # 10 GiB per arena file

# Tier 1 Host RAM / CXL Shared Memory Acceleration (/dev/shm or HugePages)
ram_cache_enabled = false                  # Set true (or pass --ram) for 50-100+ GB/s RAM tier
ram_cache_capacity_bytes = 34359738368     # 32 GiB (or auto-sizes to 75% of /dev/shm if omitted)
ram_cache_shm_path = "/dev/shm/hypercache"
ram_cache_hugepages = false                # Set true to back Tier 1 with /dev/hugepages
ram_cache_hugepages_path = "/dev/hugepages/hypercache"

# Dual-Tier Metadata Persistence (< 20 ms warm-boot recovery)
metadata_persistence_enabled = true
metadata_persistence_path = "/var/lib/hypercache/metadata.json"

# ==============================================================================
# 2. AUTHORITATIVE REMOTE STORAGE (S3 / NFS)
# ==============================================================================
authoritative_backend = "s3"               # Default backend: "s3" or "nfs"
s3_endpoint = "https://s3.us-east-1.amazonaws.com"
s3_region = "us-east-1"
s3_bucket = "model-weights-bucket"
s3_access_key = "YOUR_ACCESS_KEY"          # Or set via env: export AWS_ACCESS_KEY_ID="..."
s3_secret_key = "YOUR_SECRET_KEY"          # Or set via env: export AWS_SECRET_ACCESS_KEY="..."
allow_synthetic_s3_fallback = false        # Keep false in production so S3 errors fail fast

nfs_mount_path = "/mnt/nfs"

# Checkpoint Write Semantics & Fault Tolerance
write_through_enabled = false              # false = async Write-Back, true = sync Write-Through
peer_replication_factor = 1                # Set to 2 for k=2 peer replica before CommitWrite returns

# ==============================================================================
# 3. MULTI-NODE CLUSTER REGISTRATION (Optional - Omit for Single-Node Mode)
# ==============================================================================
node_id = "gpu-node-01"
# controller_url = "http://10.0.0.10:9120"
# cluster_auth_token = "your-cluster-token"

agent_enabled = false
agent_bind_address = "0.0.0.0"
agent_http_port = 9115

[node_labels]
gpu = "h100"
pool = "training"

# ==============================================================================
# 4. PROMETHEUS TELEMETRY (Optional - Off by Default)
# ==============================================================================
metrics_enabled = false
metrics_bind_address = "127.0.0.1"
metrics_port = 9110

---

3. Central Controller Configuration (configs/hypercache-controller.toml)

Deployed on the head/coordinator node in multi-node GPU clusters (./hypercache-admin -c configs/hypercache-controller.toml controller):

# ==============================================================================
# 1. CONTROLLER NETWORK BINDING
# ==============================================================================

# IP address to bind the Central Controller service on
# Set to "0.0.0.0" to listen on all interfaces, or bind to a private VPC IP (e.g. "10.0.0.10")
controller_bind_address = "0.0.0.0"

# Port for incoming worker node registrations, heartbeats, and admin API calls
controller_port = 9120

# Optional secondary admin port
controller_admin_port = 9121

# ==============================================================================
# 2. CLUSTER SECURITY & AUTHENTICATION
# ==============================================================================

# Shared secret Bearer token for cluster security
# Worker nodes must provide this token to register, and remote CLI calls require it.
# Local connections from 127.0.0.1 on the controller host bypass authentication automatically.
# Can also be set via: export HYPERCACHE_AUTH_TOKEN="your-cluster-token"
cluster_auth_token = "your-cluster-token"

# ==============================================================================
# 3. P2P FLEET PRE-WARMING & CONSISTENT SHARDING
# ==============================================================================

# When enabled, fleet-wide preload pulls 1/N of chunks from S3 per node and distributes
# the remaining chunks peer-to-peer across worker nodes to eliminate S3 egress storms.
p2p_preload_enabled = false

---

4. Configuration Key & Environment Variable Reference

4.1 Worker Node Keys (hypercache-agent.toml)

Config Key	Env Override	Default	Description
socket_path	HYPERCACHE_SOCKET	"/run/hypercache/hypercache.sock"	Local Unix Domain Socket path (falls back to /tmp/hypercache/hypercache.sock).
storage_backing_mode	-	"auto"	"auto", "raw_block", or "filesystem" (O_DIRECT arena files).
filesystem_cache_paths	-	[]	Mounted ext4/XFS directories for O_DIRECT arena files.
filesystem_arena_size_bytes	-	10737418240	Pre-allocated size per filesystem arena file in bytes (10 GiB default).
device_discovery_mode	-	"auto"	"auto" (scan /sys/class/block) or "manual" (use block_devices list).
block_devices	-	[]	Explicit list of block devices when device_discovery_mode = "manual".
exclude_root_device	-	true	Unconditionally exclude OS root drive (/) from the storage pool.
chunk_size_bytes	-	4194304	Stripe chunk size in bytes across RAM and NVMe tiers (4 MiB).
ram_cache_enabled	RAM_CACHE_ENABLED	false	Enable Tier 1 Host RAM / CXL shared memory tier (--ram).
ram_cache_capacity_bytes	RAM_CAPACITY_BYTES	34359738368	Maximum Tier 1 RAM cache size (32 GiB or 75% of /dev/shm if omitted).
ram_cache_hugepages	-	false	Use Linux hugetlbfs (/dev/hugepages) for Tier 1 memory blocks.
authoritative_backend	-	"s3"	Default remote storage backend ("s3" or "nfs").
s3_endpoint	S3_ENDPOINT	"http://127.0.0.1:9002"	S3 or S3-compatible endpoint URL.
s3_region	AWS_DEFAULT_REGION	"us-east-1"	S3 bucket region.
s3_bucket	S3_BUCKET	"hypercache-authoritative"	Default S3 bucket name when not specified in the URI.
s3_access_key	AWS_ACCESS_KEY_ID	""	S3 access key ID.
s3_secret_key	AWS_SECRET_ACCESS_KEY	""	S3 secret access key.
nfs_mount_path	-	"/mnt/nfs"	Root mount path for NFS / POSIX shared filesystem backend.
write_through_enabled	-	false	false = async Write-Back; true = synchronous Write-Through to S3/NFS.
peer_replication_factor	-	1	Set to 2 to enable k=2 peer checkpoint replication on CommitWrite.
controller_url	HYPERCACHE_CONTROLLER_URL	None	URL of Central Controller (e.g. "http://10.0.0.10:9120").
agent_enabled	-	false	Enables Worker Agent Fleet & P2P API on agent_http_port (9115).
metrics_enabled	-	false	Enables Prometheus HTTP exporter on metrics_port (9110).

4.2 Central Controller Keys (hypercache-controller.toml)

Config Key	Env Override	Default	Description
controller_bind_address	-	"0.0.0.0"	IP address to bind the Central Controller service on.
controller_port	-	9120	TCP port for worker registrations, heartbeats, and admin directives.
controller_admin_port	-	9121	Secondary admin port.
cluster_auth_token	HYPERCACHE_AUTH_TOKEN	None	Shared secret Bearer token for Controller and Worker Agent authentication.
p2p_preload_enabled	-	false	Enables 1/N origin pull + peer-to-peer chunk distribution during fleet preload.