Run a MoE LLM Locally on Your GPU & Connect It to a BaaS

Running a Mixture-of-Experts model on your own GPU used to sound absurd. DeepSeek-V3 is 671B parameters. Kimi K2 crosses a trillion. Nothing about those numbers says "RTX 3090 in a home office." But MoE architectures activate only a small slice of their weights per token, and llama.cpp's --n-cpu-moe flag turns that sparsity into a very practical trick: keep attention and the shared expert on your GPU, park the routed experts in system RAM, and stream them over PCIe as the router calls them. A Qwen 35B-A3B model fits on a 12GB card this way, and the same PR that added the flag was validated on an RTX 2070 with only 7.78GB of VRAM running Gemma MoE at Q4_0.

This guide walks the whole path: picking a model that suits your hardware, choosing an inference engine, configuring GPU + CPU offload correctly, exposing an OpenAI-compatible API for a local LLM, and wiring that endpoint into a backend so a real application (with auth, access-controlled data, and retrieval) can use it. If you want the deeper theory of expert streaming, our pillar on streaming MoE experts on-demand covers the memory model in detail. Here we stay operational.

Why MoE Models Are a Sweet Spot for Local Inference

Dense LLMs punish you twice: every parameter has to sit in memory, and every parameter runs on every token. MoE breaks that link. A router picks a small handful of experts per token, so a 122B-A10B model behaves, computationally, like a 10B one. Large-scale MoE architectures have become a central design point for state-of-the-art language models, pushing parameter counts toward the trillion scale while keeping training and inference costs practical through sparse expert activation.

Total parameters vs. active parameters

The number that governs quality is total parameters. The number that governs speed is active parameters. Qwen's MoE lineup illustrates the split clearly: the 122B-A10B and 397B-A17B tiers activate only 10B and 17B per token respectively, even though you still store every expert weight because the router might route to any of them. Llama 4 is built on the same idea. You have to fit every expert in memory, but only a fraction activates per token, so it runs faster than a dense model of the same total size. Total vs. active parameters is the whole basis for planning VRAM requirements for MoE.

How sparse activation lets you offload experts to CPU RAM

Because only a few experts fire per token, most experts sit idle at any given moment. If they're idle anyway, they don't need to be on the GPU. That's the offload trick: keep the always-hot pieces (attention, shared expert, KV cache) in VRAM, put the routed experts in system RAM, and let the CPU handle the FFN math when the router calls them. With llama.cpp's --n-cpu-moe, you keep attention and shared weights on the GPU and stream the routed experts from system RAM, so a big MoE runs on far less VRAM than its total size implies. MoE CPU offloading is what makes any of this fit on prosumer hardware.

The cost is PCIe traffic and CPU FFN compute, which is exactly where the hardware conversation starts.

How Much Hardware You Actually Need

There's a real floor here, and it isn't your GPU. It's your RAM capacity and how fast that RAM talks to your CPU.

VRAM, system RAM, and RAM speed for expert offloading

For expert offloading, what matters ranks like this: system RAM capacity, then memory bandwidth, then VRAM, then CPU cores. Total model size (at your chosen quant) has to fit in RAM + VRAM combined. The KTransformers team's reference build for DeepSeek-V4-Flash targets an RTX 5090, an AVX2 x86 CPU, and at least 200GB of system RAM. That's the shape of a serious local-MoE rig.

Memory bandwidth is the silent throttle. Dual-channel DDR4-3200 will bottleneck a large offloaded MoE badly. DDR5-6000, or an 8/12-channel EPYC/Threadripper platform (what the r/LocalLLaMA community calls "CPUmaxx" builds), is where multi-token/sec decode on 200B+ models starts to feel usable.

Quantization formats: GGUF (Q4_K_M, IQ4_XS), FP8, MXFP4 and AWQ/GPTQ

Quantization is what makes any of this fit. For llama.cpp and ik_llama.cpp, GGUF quants like Q4_K_M and IQ4_XS are the workhorses: Q4_K_M for the best quality/size balance, IQ4_XS when you need to shave another gigabyte. For KTransformers on DeepSeek-V4-Flash, MXFP4 is the native expert format; MoE-Infinity likewise offloads FP4-quantized experts for DeepSeek-V4-Flash to fit memory-constrained GPUs. FP8 is what vLLM and SGLang prefer on Hopper/Blackwell hardware. AWQ and GPTQ show up in vLLM stacks for dense-ish paths.

Consumer GPU reality check (RTX 3060 12GB to RTX 5090 32GB)

Here's what actually runs where:

The extreme end is real but slow: the localMoE project by GitHub user andyzpb documents running a 1.6T MoE at 0.076 tok/s on a gaming laptop with a 12GB RTX. That's a proof of concept, not a workflow. DeepSeek-V3 on a consumer GPU only becomes reasonable once you pair a 5090 with 200GB+ of fast system RAM.

Choosing a Local MoE Inference Framework

Your engine choice determines how gracefully the CPU/GPU split behaves.

llama.cpp and ik_llama.cpp

llama.cpp is the default. It's the reference implementation for GGUF, supports every quant format you care about, and (since PR #15077) has proper --n-cpu-moe llama.cpp support baked in. ik_llama.cpp is a fork designed around improved CPU/CUDA hybrid performance and newer SOTA GGUF quant types. For CPUmaxx builds pushing large MoEs, ik_llama.cpp is often the faster path.

KTransformers vs. llama.cpp for hybrid CPU-GPU MoE

KTransformers is purpose-built for hybrid CPU-GPU MoE with fine-grained control the llama.cpp flag can't match. The reference DeepSeek-V4-Flash launch on a single RTX 5090 pins 10 GPU experts and 60 CPU inference threads via --kt-num-gpu-experts 10 and --kt-cpuinfer 60, using MXFP4 as the expert weight format and SGLang for serving. For DeepSeek-scale models on a single 5090, KTransformers is usually the ceiling; llama.cpp is the floor of usability but wins on portability and quant variety.

Ollama, vLLM, SGLang and MoE-Infinity

Ollama wraps llama.cpp and now passes num_cpu_moe straight through to llama-server --n-cpu-moe, so you get offload with a friendly API. vLLM and SGLang shine when the whole model fits in VRAM; PagedAttention and continuous batching aren't designed for the "experts live in RAM" world. MoE-Infinity takes a third path: expert offloading to host memory and SSD with activation-aware caching, prefetching, tracing, and fused CUDA kernels so the hot path stays lean. If your system RAM is the bottleneck rather than your GPU, MoE-Infinity's SSD tier is worth a serious look; its sample code targets DeepSeek-V2-Lite-Chat straight from a HuggingFace checkpoint.

Setting Up GPU + CPU Offloading

The mechanics are simple once you know which flag to reach for.

--n-cpu-moe vs. --override-tensor (-ot exps=CPU)

Two flags, same idea, different ergonomics. --override-tensor (or -ot) is the older, more surgical option: you write a regex matching tensor names (-ot 'exps=CPU' sends every expert tensor to CPU). --n-cpu-moe N is the newer, cleaner flag. For MoE models, it forces the expert/FFN tensors of the first N layers to stay resident in system RAM and compute on the CPU, while the rest of each layer offloads to the GPU. Start with --n-cpu-moe. Fall back to -ot only when you need per-tensor precision.

A working llama.cpp offload command, step by step

For a Qwen 122B-A10B MoE at Q4_K_M on a 24GB card with 128GB of RAM:

./llama-server \
  -m Qwen-122B-A10B-Q4_K_M.gguf \
  --n-gpu-layers 999 \
  --n-cpu-moe 94 \
  -c 8192 \
  -b 2048 -ub 512 \
  --host 0.0.0.0 --port 8080

The pattern: send all layers to the GPU with -ngl 999, then claw back the expert tensors with --n-cpu-moe. Tune N downward if VRAM has room; tune it upward if you OOM. Batch size matters more than people expect, because CPU+GPU inference is very sensitive to prompt processing batch size: the physical batch size sets how much data moves across PCIe.

Does expert offloading hurt output quality or speed?

Quality: no. The same weights run, just on a different processor. Bit-for-bit, offloaded inference matches fully-in-VRAM inference at the same quant. Speed: yes. Decode throughput drops from the "everything in VRAM" ceiling because the CPU FFN and PCIe transfers dominate. The academic reality check is that local MoE inference commonly falls short of 20 tok/s decode and 30-second TTFT for long prefills that cloud users take for granted. Set expectations accordingly. This is a workflow for building and iterating, not for serving 10,000 users.

Exposing Your Model as an OpenAI-Compatible API

The nice thing about the current ecosystem: every serious way to run MoE LLM locally speaks OpenAI's chat completions dialect. Your app code doesn't need to know it's talking to a GPU under your desk.

Starting llama-server / ollama serve on localhost

Both llama-server and ollama serve stand up an OpenAI-compatible API for a local LLM out of the box. Common defaults:

All three implement the OpenAI chat completions format, so any OpenAI SDK works by pointing base_url at them.

Testing /v1/chat/completions and pointing a client at base_url

A one-line curl confirms the server is alive:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"ping"}]}'

From Python, point the OpenAI SDK at the local base_url and set any string as the API key; most local servers ignore it:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

That's the whole client-side change. Your existing agent code, RAG pipeline, or chat UI works unchanged.

Connecting Your Local LLM to a Backend-as-a-Service

A model that only your laptop can call isn't an application. To connect a local LLM to a BaaS, the endpoint has to be reachable from your backend, and the backend has to add the pieces the model doesn't: identity, access-controlled data, retrieval, and a record of what happened.

Making the endpoint reachable (reverse proxy, ngrok tunnel)

Two paths. For quick tests, ngrok gives you a public URL like https://abc123.ngrok.io that forwards to your local server. Perfect for demos, wrong for anything production-shaped. For real use, put the model server behind a reverse proxy (Caddy or nginx) with TLS, a subdomain, and a bearer token in front of /v1/*. Bind llama-server to 127.0.0.1 and let the proxy be the only public listener.

Wiring it into Powabase

One constraint first. Powabase agents choose their reasoning model through LiteLLM, and bring-your-own-key covers four providers today: OpenAI, Anthropic, Google, and OpenRouter. There's no slot for a custom base URL, so an agent's own reasoning loop can't run on your GPU yet. What you can do is make your local model something the platform calls, in one of three ways:

• As a custom tool on an agent. A custom tool is a name, a description, a JSON Schema for its inputs, and an endpoint URL. Point one at a thin wrapper around /v1/chat/completions (say, summarize_privately(text)) and a hosted agent can hand bulk or sensitive generation to your GPU while it does the planning.
• As an MCP server. Wrap the model in a small MCP server and attach its URL to an agent. Its tools are discovered at the start of every run and sit next to the builtin ones.
• As a workflow step. A workflow's external HTTP block calls your endpoint as one step in a pipeline, which suits scheduled batch jobs like overnight classification or re-summarizing a ticket backlog.

Two limits shape the design. Custom tool and MCP calls time out after 30 seconds, and a long prefill on an offloaded MoE can eat most of that, so send focused prompts rather than whole documents. Custom tools are also SSRF-checked, so the platform won't call localhost or a private address. Register the public, TLS-fronted URL from the previous step.

Adding auth, access-controlled data, and retrieval

Raw llama-server has no concept of a user. It'll happily answer whoever hits the port. Your bearer token keeps strangers off the endpoint, but an application needs more than that, and the backend is where the rest lives:

• Auth. Users sign in through Powabase Auth and your app carries their JWT, so every request has an identity before it gets near your GPU.
• Access-controlled data. Row Level Security scopes each user's prompts, outputs, and documents at the database layer, per the RLS model.
• Retrieval for your own model. Context handlers run the platform's RAG retrieval without an agent: send a query, get ranked chunks back, and pass them to your local model. You get managed vector and hybrid search, with generation on your hardware.
• A usage log you own. Powabase meters its own agent runs, but it doesn't count tokens generated on your GPU. Write one row per local call to a Postgres table (user, tokens, latency), and quotas or chargeback become ordinary SQL.

Powabase's unified API covers agents, RAG, orchestration, database, auth, and storage from one REST API with one auth model. Your GPU generates the tokens; the backend handles users, data, and retrieval. With framework-first stacks like LangChain/LangGraph, you'd write and operate each of those pieces yourself.

A Repeatable Local-MoE-to-BaaS Workflow

The repeatable loop:

1. Pick a model whose active parameters fit comfortably in VRAM and whose total weights fit in RAM at Q4_K_M or MXFP4.
2. Start with llama.cpp + --n-cpu-moe; graduate to KTransformers or MoE-Infinity when you need more throughput on DeepSeek-scale models.
3. Bind llama-server to localhost, put a TLS-terminating reverse proxy in front with a bearer token, and confirm /v1/chat/completions responds.
4. Wire it into Powabase as a custom tool, an MCP server, or a workflow step, keep each call inside the 30-second tool timeout, and let the backend handle users, access-controlled data, and retrieval.

The first token you generate on your own hardware and route through a real backend to a real user is the moment local MoE stops being a hobby and starts being infrastructure.