Why MoE Models Stream From NVMe: Kernel Prefetch, Read-Ahead, and Async I/O

Frontier MoE models are too large for memory. Colibri and similar engines stream expert weights from NVMe storage. The trick isn’t clever compression—it’s understanding that disk I/O and CPU compute can overlap perfectly if the kernel cooperates. Here’s how infrastructure makes 70B-parameter models run on consumer hardware.

The Problem

A Mixture-of-Experts model like Llama 2 70B has 70 billion parameters. At float32, that’s 280GB of memory. No consumer machine has that. Even quantized to int8, it’s 70GB. GPU VRAM maxes out at 48GB on consumer hardware (RTX 6000 Ada).

Traditional approaches: split the model across multiple GPUs (requires NVLink or high-speed interconnect), compress aggressively (hurts accuracy), or run inference on a server. But what if you streamed weights from disk on demand? Not everything at once—just the experts you need for each token.

In MoE, a routing layer decides which experts process each token. For most tokens, only 1-4 of 64 experts activate. If you could load only those experts into RAM, compute on them, then evict them for the next token’s experts, you’d stay under memory constraints. The catch: disk I/O is 1000x slower than RAM. Unless compute and I/O overlap perfectly, you’re stuck waiting.

Why It Works Now

Three kernel features make this possible. First, memory-mapped I/O. The model file is mmap’d into virtual address space. The kernel handles page faults automatically. When your process touches an unmapped page, the kernel loads it from disk into a buffer, remaps it, and returns control. No explicit read() calls. No manual buffering.

Second, read-ahead. Linux prefetches pages before they’re requested. If the kernel sees sequential access (token 1 loads experts A, B, C; token 2 likely loads experts near C), it speculatively reads the next pages. For MoE routing patterns, this is remarkably accurate. You get 50-80% of the next expert’s weights while computing on the current expert.

Third, async I/O. Modern NVMe is parallel. You can issue thousands of read commands at once without blocking. A smart inference engine prefetches the next 2-3 tokens’ experts while computing on the current token. By the time compute finishes, the data is already resident.

Colibri (and similar engines) use all three. The model is mmap’d. As tokens flow through the routing layer, you know which experts fire. Before computing on expert A, you prefetch experts for token N+1 and N+2. By the time expert A finishes, expert D (from token N+2) is paged in. No stalls. No explicit scheduling.

The kernel scheduler handles the rest. When your inference thread hits a page fault, it sleeps. The disk I/O hardware fires. Meanwhile, other threads run. When the page arrives, the kernel wakes your thread and resumes. This overlap is why streaming models work—not because disk got fast, but because I/O and compute can happen simultaneously.

How to protect

If you’re deploying MoE inference, tune the kernel for this access pattern. First, increase the read-ahead window. By default, Linux reads 128KB ahead. For 32-bit floats in sequential expert weights, bump it to 2-4MB.

# Increase read-ahead for the model file's block device
# (e.g., /dev/nvme0n1 for NVMe)
sudo blockdev --setra 8192 /dev/nvme0n1

# Verify
sudo blockdev --getra /dev/nvme0n1
# Output: 8192 (sectors, so 8192 * 512B = 4MB)

# For a single file, use posix_fadvise in your inference code:
// Hint: sequential access, aggressive read-ahead
posix_fadvise(fd, 0, filesize, POSIX_FADV_SEQUENTIAL);

Second, disable page cache writeback pressure. The kernel tries to keep dirty pages in memory, which competes with model weights. For read-only model access, this is wasted memory.

# Reduce page cache pressure
# (scale 0-200, lower = more aggressive cache retention)
echo 20 | sudo tee /proc/sys/vm/vfs_cache_pressure

# Disable swap (model working set is immutable per token)
sudo swapoff -a

# Set page reclaim to favor clean pages
echo 20 | sudo tee /proc/sys/vm/swappiness

Third, use a memory allocator designed for streaming workloads. TCMalloc or jemalloc with small allocation sizes prevents fragmentation. Standard malloc can create gaps that waste prefetched pages.

Fourth, pin the routing layer and I/O thread to separate CPU cores. Let the inference compute thread go anywhere, but keep I/O prefetching consistent. Scheduler migrations between cores cause cache thrashing and break read-ahead heuristics.

Finally, monitor it. Watch page faults, I/O latency, and CPU wait time. If you’re seeing high wait (iostat %wa), read-ahead isn’t working—check block device settings and routing patterns. If page faults are high but I/O latency is low, you’re prefetching correctly but too late. Increase prefetch depth in your inference engine.

MoE models on consumer hardware work because the kernel prefetches disk pages while your GPU computes. The overlap isn’t magic—it’s thirty years of I/O scheduling evolution. Know your kernel.