Streaming MoE Experts On-Demand: Big LLMs, Tiny RAM

A 26B-parameter Gemma model runs on an 8 GB Mac. A 685B DeepSeek runs on a laptop. Not because the weights got smaller (they didn't), but because Mixture-of-Experts models only use a tiny slice of themselves per token, and a handful of open-source runtimes finally treat that fact as a hardware opportunity instead of a curiosity. Streaming MoE experts on-demand, reading expert weights from SSD into a bounded RAM cache exactly when the router picks them, is what lets these models fit on consumer machines.

This piece is the hub for our LLM inference cluster. It walks through why MoE makes streaming possible, how the mechanics actually work, and how projects like TurboFieldfare and Swiftlet put it into practice on Apple Silicon, then how research systems generalize the idea to data-center GPUs.

Why Full MoE Models No Longer Have to Fit in Memory

The old rule of local inference was simple: your model has to fit in RAM. For dense models it still does. But MoE breaks the rule at the architectural level. Only a small fraction of parameters activate per token, so the rest of the file is dead weight the moment you finish loading it. The waste is measurable. A recent MoE inference paper points out that most expert weights remain idle in GPU memory while competing with the KV cache for the space that actually drives throughput.

If most experts sit unused for most tokens, keeping them resident is a choice, not a requirement. The interesting question is how few of them you can keep in memory before quality or latency falls apart. Modern runtimes have pushed that number surprisingly low.

MoE Fundamentals: Sparse Activation and the Router

An MoE layer replaces a single feed-forward block with a bank of parallel experts and a small router that decides which ones handle each token. NVIDIA describes this as selective activation that lets parameter counts push into the extreme range without paying the full compute cost. You scale capacity by adding experts, not by making every forward pass more expensive.

How the Gating Network Router Selects Top-K Experts

Each MoE layer has N experts {E₁, …, E_N} and a gating function G(x) that, per token, selects a sparse subset of k ≪ N experts. The router step is a tiny matmul: it produces logits over experts, keeps the top-k, and normalizes their weights. Every other expert contributes nothing to that token.

That is the entire opening for streaming MoE experts on-demand. If the router picks a handful of experts out of many, the rest of the expert weight tensors did not need to be in RAM this step. They only needed to be somewhere fast enough to fetch before the next MoE layer runs.

Routed vs. Shared Experts and the Active-Parameter Trick

Modern MoE models split the FFN into two pieces: shared experts that run on every token, and routed experts that only run when picked. HybriMoE's paper shows this pattern clearly in an MoE architecture with shared and routed experts. Shared experts carry general behavior, routed experts carry specialization.

That is where naming conventions like Qwen3-Next 80B-A3B or Gemma 4 26B-A4B come from: 80B total parameters, ~3B active per token; 26B total, ~4B active. The "A" number is what the GPU actually multiplies against. On-demand expert loading systems care about the total, because that's what has to live somewhere; latency budgets care about the active number, because that's what has to move through cache each step.

The Memory Bottleneck: Expert Weights vs. the KV Cache

Serving throughput on a GPU is bounded by KV cache GPU memory capacity, because longer contexts and more concurrent requests need more of it. The FluxMoE paper puts the tradeoff bluntly: idle expert weights compete with performance-critical runtime state such as the key-value cache, and since KV cache size determines throughput, resident experts cost real tokens per second.

On a Mac or an iPhone the constraint is even simpler. You don't have GPU memory, you have unified memory, and every gigabyte spent on cold experts is a gigabyte the OS can't give you back. Unified memory MoE inference turns idle expert weights into a storage problem instead of a memory problem, and storage is the cheap axis.

What Expert Streaming Is and How It Cuts Memory Requirements

Expert streaming keeps shared weights, attention, embeddings, and the router permanently resident, and treats routed experts as pageable objects on SSD. When the router picks its top-k, the runtime reads exactly those experts from disk into a small pool of pre-registered buffers, runs the MoE math, and reuses the buffers for the next layer. That is the shape of streaming MoE experts on-demand in one paragraph.

Expert Paging vs. Standard Weight Offloading

Standard offloading (CPU RAM ↔ GPU VRAM) moves whole layers back and forth on a schedule. Expert paging is finer-grained and content-dependent. It moves only the specific expert tensors the router asked for, on the exact step it asked for them. FluxMoE names this pattern directly, calling it an expert paging abstraction that treats expert weights as transient, streamed resources materialized on demand, backed by a bandwidth-balanced storage hierarchy and a residency planner that decides what's worth pinning.

So total resident footprint scales with active parameters plus a cache, not with total parameters.

The LFU Expert Cache and the Bounded Slot Pool

Streaming naïvely from disk every step would be catastrophic, because routers repeatedly favor the same experts across nearby tokens. So every serious streaming runtime puts a small in-memory LFU expert cache in front of the SSD, keyed by (layer, expert_id), with an eviction policy that combines frequency and recency. Swiftlet's writeup describes the policy plainly: evict with LFU plus recency, and pack experts at fixed stride so one fetch is one read.

Research systems have shown this cache design matters a lot. DALI's authors argue that prior work ignored the influence of dynamic workloads when designing cache replacement strategies for the expert cache, which is why hit rates were low. Workload-aware replacement, biasing toward experts that this prompt has been hitting rather than lifetime averages, is where a lot of the wins are.

The other half is the buffer pool. Instead of allocating a fresh buffer each fetch, runtimes pre-allocate a fixed number of aligned slots and rotate through them. TurboFieldfare's design doc spells this out: routed-expert files open lazily, and each opened layer owns one file descriptor and a fixed group of slot buffers, each registered with Metal once and reused forever.

Stream Experts from SSD: pread, Blobs, and Quantization Formats

Three implementation choices dominate.

• Read with pread, not mmap. mmap looks convenient but hands the OS control of paging, which fights the LFU policy the runtime is trying to enforce. pread gives explicit control over what's in RAM and when.
• Fixed-stride packing. If every expert of a given layer is the same size and aligned to the same stride, "load expert 47" is one seek and one read, not a scatter/gather. Swiftlet explicitly packs experts at fixed stride so one fetch is one read.
• Aggressive quantization. Runtimes stream experts from SSD in 4-bit form to multiply effective SSD bandwidth. TurboFieldfare ships 4-bit MLX affine embedding, attention, shared-expert, and routed-expert weights with an 8-bit router, plus custom Metal kernels for quantized GEMV.

TurboFieldfare: Gemma 4 26B-A4B Inference in ~2 GB of RAM

TurboFieldfare is the cleanest reference for Apple Silicon MoE inference. It targets Gemma 4 26B-A4B inference in about 2 GB of RAM, on any Apple Silicon Mac including the 8 GB models, with a custom Swift + Metal runtime.

The system design document lays out the arithmetic. The text-only Gemma 4 26B-A4B install is roughly 14.3 GB, but the target machine has 8 GB total. So the runtime keeps common weights and working state available to Metal while storing routed experts in per-layer files and reading only the experts chosen for the current token or prefill chunk. Model_weights.bin is mmapped read-only, its aligned regions wrapped in MTLBuffer without copy, and routed-expert files open lazily per layer.

Prefill uses the same discipline. TurboFieldfare handles up to 128 prompt tokens at a time and stays layer-major, moving each bounded group of rows through the transformer one layer at a time without holding expert activations for the full prompt. That keeps the working set for a long prompt roughly the same size as the working set for a single token.

The .gturbo Container and Bounded Repack

The .gturbo format is what makes fixed-stride streaming possible. It's a bounded repack of the source Gemma weights into a container designed for one-fetch-one-read: shared weights and attention in a mapped blob, routed experts in per-layer files packed at uniform stride, quantized to 4-bit MLX with an 8-bit router. Because the repack is deterministic and bounded, the runtime knows the exact byte offset of every (layer, expert) pair before inference starts, and never allocates during the token loop.

Swiftlet: Running 35B and 80B Qwen Models on Ordinary Apple Devices

Swiftlet takes TurboFieldfare's playbook and targets Qwen instead of Gemma, across both macOS and iOS. The project credits TurboFieldfare directly for proving the expert-streaming thesis on Macs and adopts its lessons: pread into a bounded slot pool instead of mmap, LFU-plus-recency eviction, and fixed-stride packing so one fetch is one read.

Qwen3-Next 80B-A3B is a particularly good target for streaming MoE experts on-demand. 80B total but only ~3B active per token, so the compute-to-memory ratio per step is exactly the shape streaming exploits. Total on-disk footprint is large, per-step working set is small. The 35B sibling is more striking still. The Qwen3.6-35B-A3B .qpack container runs a 35-billion-parameter model on an iPhone or any Apple Silicon Mac in about 2.5 GB of RAM, byte-identical to the 4-bit MLX weights, with no re-quantization.

The .qpack Streaming Container and the Swift/Metal Runtime

Swiftlet's .qpack container is the Qwen-side analog to .gturbo. Same core idea (fixed-stride expert packing, quantized weights, per-layer expert files), but tuned for Qwen3's router shape and expert count. The runtime is Swift + Metal, so it runs on iPhones and iPads whose GPUs share memory with the CPU. On an iPhone, "GPU memory" and "RAM" are the same pool, which means expert streaming isn't fighting a PCIe bus; the cache and the executor look at the same bytes.

Can You Run a 70B+ MoE Model on an 8 GB Mac or an iPhone?

Yes for MoE, with caveats. No for dense.

On an 8 GB Mac, TurboFieldfare's ~2 GB resident footprint for Gemma 4 26B-A4B fits comfortably alongside the OS and other apps, and Swiftlet's 2.5 GB target for Qwen3.6-35B-A3B fits the same way, because the routed experts live on SSD. It is not enough for a 70B dense model, where every parameter is active every token and there's nothing to stream. MoE inference on consumer hardware works precisely because MoE gives you something to page.

Latency is the caveat. Every routed-expert cache miss is an SSD read on the critical path. Fast NVMe (5+ GB/s sequential) plus 4-bit quantization plus a warm LFU cache brings this into interactive range, but a cold prompt still pays a warmup tax while the cache fills. The bigger the k in top-k, and the more experts per layer, the more the working set stresses the cache, and the more your SSD's random-read latency (not sequential bandwidth) becomes the bottleneck.

For iPhones, the same math holds but with tighter thermal and I/O ceilings. Small MoE models (a few billion active parameters, 4-bit) are feasible; 80B-total models are demo-feasible but not something you'd want a user waiting on.

What Determines Whether a Given Model Streams Well

• Total ÷ active ratio. Higher is better, because more of the model is pageable.
• Experts per layer. More experts means a smaller per-expert weight and more room for the cache to find hot ones.
• Quantization format. 4-bit MLX or 4-bit GGUF makes each SSD read cheaper.
• SSD random-read latency. Not sequential bandwidth. Miss penalty is what you feel.
• Prompt locality. Prompts that reuse a narrow expert set warm the LFU cache fast.

How Research Systems Push the Same Idea Further

Consumer runtimes optimize for "fits on my Mac." Research systems optimize for "cheapest possible datacenter serving," but the primitives for streaming MoE experts on-demand are the same: cache, prefetch, page.

FluxMoE, MoE-Lightning, DALI, and HybriMoE

FluxMoE, running on an NVIDIA L40 testbed, decouples expert parameters from GPU residency using PagedTensor and a budget-aware residency planner to reclaim the VRAM for KV cache. DALI adds workload-aware cache replacement, noticing that a given prompt's expert-usage distribution is very different from the model's average, and biasing eviction accordingly.

HybriMoE targets the messy middle: some experts on GPU, some on CPU, dispatched dynamically. Its scheduler balances workloads across GPUs and CPUs through prioritized task execution and data transfer management, plus an impact-driven prefetcher that ranks upcoming experts by how much latency their absence would cost.

colibri, ExpertFlow, and the CPU-GPU Hybrid Scheduling Frontier

On the open-source side, colibri packages the streaming approach as a family of sibling engines: one C file per architecture, a shared front end. Its author notes the engine runs GLM-5.2 as reference plus three more families, with expert paging as the common substrate. ExpertFlow takes the same idea to DeepSeek-scale weights. Its README claims you can run 685B parameter models on a laptop with dynamic MoE expert streaming for Apple Silicon, keeping hot experts in unified memory and streaming cold experts from NVMe on demand, with GPU + ANE + SSD orchestration in Rust. The CLI exposes exactly the knobs the theory papers describe:

expertflow run \
  --model ./DeepSeek-V3-Q4.gguf \
  --ram-budget 100 \
  --prefetch-depth 2 \
  --pin-threshold 0.7

--ram-budget is the cache size, --prefetch-depth is the lookahead across layers, and --pin-threshold is the temperature above which experts stay pinned. The residency planner idea, exposed as flags. GOBA's moe-stream generalizes further with three inference modes, GPU Resident, GPU Hybrid, and SSD Streaming, chosen by whether the GGUF fits in RAM.

Independent projects across Swift, Rust, and C, targeting Mac laptops through L40 servers, all build the same three components: a bounded slot pool, an LFU-with-recency cache, and a fixed-stride on-disk container.

Where This Leaves the Inference Stack

MoE was invented to make training cheaper per parameter. Streaming turns the same sparsity into an inference story: a 26B model that lives on SSD and touches 2 GB of RAM at a time, a 35B model that fits in 2.5 GB on an iPhone, a 685B model that runs on a laptop. The pattern is settled (pread from a packed container, cache with LFU-plus-recency, prefetch by router lookahead, quantize aggressively) and the remaining work is tuning, not invention.

For anyone building AI apps, the inference layer under your product is about to look very different from the "one model per GPU" world. Local models will get bigger, and the economics of open-weight MoE on commodity hardware will keep improving. On Powabase, our agent runtime and retrieval are co-located inside your isolated project stack, and we address models through LiteLLM IDs. So as streaming-capable local models mature, you point an agent at a local endpoint the same way you point it at a hosted one, and our ReAct loop, tool calls, and SSE streaming behave identically whether the model is a hosted API or a laptop streaming experts from SSD.