CLI & Fleet Admin Reference

HyperCache ships as two self-contained Rust binaries:

./hypercache - Runs on each GPU or CPU worker node to manage local NVMe/RAM pools, serve Python workloads over a local Unix socket, and run hardware diagnostics.
./hypercache-admin - Runs on the controller host (or operator workstation) to view cluster inventory and dispatch parallel preload, sharding, and eviction commands across nodes.

1. Worker Node CLI (`./hypercache`)

1.1 Diagnostics & Hardware Discovery

Run these commands on a worker node before or during operation to verify drive permissions, O_DIRECT alignment, and cache health:

# Run pre-flight host diagnostics (config syntax, raw drive safety, O_DIRECT alignment)
./hypercache -c configs/hypercache-agent.toml check

# Scan and display discovered NVMe/SSD block devices and NUMA topology
./hypercache inspect

# Query the running daemon for active drives, cache hit ratio, and health
./hypercache status

Subcommand	Aliases	Description
`check`	`doctor`, `preflight`	Verifies configuration, permissions, raw drives, `O_DIRECT` alignment, and sockets.
`inspect`	-	Lists discovered physical/virtual SSDs, excluded OS mounts, and NUMA mappings.
`status`	-	Prints active storage pool capacity, utilization, and cache hit rate.

1.2 Starting the Worker Daemon

By default, ./hypercache start runs in local-only mode with 0 open TCP ports, listening exclusively on /run/hypercache/hypercache.sock.

# Standard local startup (0 network ports open)
./hypercache -c configs/hypercache-agent.toml start

# Enable Tier 1 Host RAM cache (/dev/shm) alongside local NVMe
./hypercache -c configs/hypercache-agent.toml start --ram

# Enable Prometheus metrics exporter on 127.0.0.1:9110
./hypercache -c configs/hypercache-agent.toml start --metrics

# Join a multi-node cluster and enable the Fleet Agent API (:9115)
./hypercache -c configs/hypercache-agent.toml start --agent

Flag	Default	Description
`-c <path>`	`configs/hypercache-agent.toml`	Path to the worker TOML configuration file.
`--ram`	`false`	Enables Tier 1 Host RAM (`/dev/shm` or HugePages) caching.
`--metrics`	`false`	Enables the Prometheus HTTP exporter on `:9110/metrics`.
`--agent`	`false`	Enables the Worker Fleet & P2P HTTP listener on `:9115`.
`--devices <list>`	Auto-discovered	Comma-separated block devices (e.g. `/dev/nvme1n1,/dev/nvme2n1`) for manual mode.

1.3 Managing Local Cache Objects

Inspect, pre-warm, pin, or evict weights and checkpoints on the local node:

# List all cached objects, sizes, storage tier (RAM/NVMe), sync state, and pin status
./hypercache ls

# Pre-load a model into local NVMe and pin it so W-TinyLFU never evicts it
./hypercache preload hypercache://s3/my-bucket/models/llama-3-8b/weights.pt --tier nvme --pin

# Batch pre-load multiple URIs from a newline-delimited file
./hypercache preload --file checkpoints.txt --pin

# Evict a specific object and reclaim its 4 MiB blocks
./hypercache rm hypercache://s3/my-bucket/models/llama-3-8b/weights.pt

# Re-fetch fresh weights from S3 or NFS
./hypercache refresh hypercache://s3/my-bucket/models/llama-3-8b/weights.pt

# Clear all cached objects from local RAM and NVMe pools
./hypercache purge

Subcommand	Key Flags	Description
`ls`	`--json`	Lists all cached objects, chunk counts, tier placement, and backend sync status.
`preload <uri>`	`--tier <ram\|nvme>`, `--pin`, `--file <path>`	Streams an object from S3/NFS into local cache ahead of workload execution.
`rm <uri>`	-	Evicts a single URI from local cache and frees its 4 MiB stripes.
`refresh <uri>`	-	Invalidates the local copy and re-ingests from authoritative S3 or NFS.
`purge`	-	Resets the entire local cache catalog and block allocators.

1.4 Hardware DMA Benchmarking

Verify real-world hardware read throughput and end-to-end SHA-256 bit parity across your drives:

# Benchmark Tier 2 (Striped NVMe Pool) with a 5 GB payload
./hypercache bench --tier nvme --size-mb 5000

# Benchmark Tier 1 (Host RAM /dev/shm Pool) with a 1 GB payload
./hypercache bench --tier ram --size-mb 1000

2. Fleet Administration CLI (`./hypercache-admin`)

When executed directly on the Controller host (127.0.0.1), ./hypercache-admin requires no flags, no tokens, and no IPs.

2.1 Starting the Central Controller

# Start the Central Cluster Controller daemon on port 9120
./hypercache-admin -c configs/hypercache-controller.toml controller

2.2 Cluster Topology & Fleet Health

# View all registered worker nodes, labels, discovered drives, and cached footprints
./hypercache-admin nodes

# View aggregate cluster capacity, online node count, and pool utilization
./hypercache-admin status

2.3 Fleet-Wide Preloading & Dataset Sharding

Warm up checkpoints across all nodes - or subset nodes by label - before launching a distributed training or inference job:

# Pre-load a checkpoint across all online nodes
./hypercache-admin preload hypercache://s3/models/llama-3-70b/weights.safetensors --nodes all

# Pre-load only on nodes matching specific hardware/pool labels
./hypercache-admin preload hypercache://s3/models/llama-3-70b/weights.safetensors \
  --selector "gpu=h100,pool=training"

# Partition and pre-load a sharded dataset across nodes via Rendezvous Hashing
./hypercache-admin preload-batch --prefix "s3://datasets/tokens/shard_" \
  --sharded --nodes all

Subcommand	Target Flags	Description
`preload <uri>`	`--nodes <all\|id1,id2>`, `--selector <k=v>`	Dispatches parallel model pre-warming (or P2P distribution) to target nodes.
`preload-batch`	`--prefix <uri>`, `--sharded`, `--nodes all`	Assigns `1/N` of matching dataset shards to each node using consistent hashing.

2.4 Fleet-Wide Eviction & Purge

# Evict a deprecated model version across all nodes
./hypercache-admin evict hypercache://s3/models/llama-2-7b/weights.safetensors --nodes all

# Purge all cached objects and reset block allocators across the entire fleet
./hypercache-admin purge --nodes all

1. Worker Node CLI (./hypercache)