Offloading Mixture-of-Experts Models to NVMe Storage

Three weeks ago, I tried running a 47B Mixture-of-Experts model on a workstation with 32GB of RAM. The setup seemed simple: memory-map the parameter files using mmap() and let the Linux page cache fetch weights from a PCIe 4.0 NVMe drive when needed. Cold start prompt processing hit 40 tokens per second. Then token generation started, and throughput collapsed to 1.2 tokens per second. The disk activity LED stayed solid red while all 16 CPU cores hovered at two percent utilization.

The Random Read IOPS Trap in Expert Routing

Standard dense transformer models read every weight tensor in order during every forward pass. Sequential memory access patterns allow operating system readahead workers to prefetch upcoming weight blocks into RAM before compute threads request them. GPU memory bandwidth or PCIe bus speed stays the primary constraint.

Sparse MoE architectures break sequential prefetching completely. Each layer routes input tokens to a small subset of experts based on gating outputs. For token A, the router selects experts 2 and 6. For token B, it selects experts 1 and 7. When total parameters exceed physical RAM, every routing decision forces the operating system kernel to issue blocking page faults across separate multi-gigabyte file offsets.

Dense offloading suffers from sequential memory bandwidth constraints. Sparse MoE offloading collapses under random storage read latency.

Consumer NVMe drives list advertised speed ratings based on sequential transfer rates, often reaching 7,000 MB/s. Their single-queue 4KB random read throughput tells a different story: most drives max out between 60 MB/s and 90 MB/s at queue depth 1. When an expert router requests scattered parameters, the inference loop halts while waiting for storage controller command queues to flush.

Measuring Page Fault Rates and Tuning Kernel Readahead

Before modifying application code, measure whether page faults cause your throughput drop. Run sar and iostat while issuing generation requests:

# Monitor major page faults and NVMe queue metrics during inference
sar -B 1 10
iostat -xz 1 10

High numbers in the majflt/s output paired with 100% utilization on your NVMe block device confirm that page faults are blocking the worker threads. Default kernel readahead exacerbates this issue. When a page fault occurs, Linux fetches adjacent disk sectors under the assumption that nearby bytes will be read next. Because sparse gating skips unpredictably between expert blocks, prefetching adjacent sectors wastes bus bandwidth on parameters the current token never touches.

Disable kernel readahead on the block device storing the model files:

# Disable block device readahead on the target NVMe partition
sudo blockdev --setra 0 /dev/nvme0n1p1

To improve throughput further, keep attention projections and gating layers resident in physical memory with mlock(), leaving only feed-forward expert matrices on storage. Gating weights take up less than two percent of total model size but execute on every token. Locking them prevents secondary page faults during the initial routing calculation.

If you run MoE offloading on local hardware, evaluate storage drives by QD1 random read latency instead of peak sequential megabytes per second. Pin router layers in system memory, set disk readahead to zero, and inspect page fault counts before blaming your CPU or RAM speed.