Local inference has an awkward middle: models too large for one consumer card and too small to justify a datacentre GPU. The usual answer is to quantise until the checkpoint fits, trading quality for memory a bit at a time. Capy Flash takes a different route for one specific family of models. Instead of shrinking the weights until they fit in VRAM, it keeps the parts that do not fit in system RAM and computes them on the CPU.
What it is
Capy Flash is an inference server, not a framework and not a chat app. It runs two checkpoints: Qwen3.8-Flash-Next, a 125B mixture-of-experts model, and Qwen3.8-27B, a dense model.
The interface is the part that matters. It exposes an OpenAI- and Anthropic-compatible HTTP API, so agent tools such as Claude Code, OpenAI Codex and OpenCode can point at localhost and treat it as a hosted model. That is the difference between a research script and something you can actually put behind your editor.
Where the experts live
The split follows from the architecture. Flash-Next is a mixture of experts: every token activates a subset of the experts, so the checkpoint is far larger than the working set for any single token. Capy Flash keeps the experts that do not fit in VRAM in host RAM and computes them on the CPU; everything else runs on the GPU, split across two cards with tensor parallelism when there are two.
That one decision explains almost every requirement in the README. Because the routed experts sit in a page-locked host allocation, the machine needs RAM in the 64 to 96 GB range, and the CPU needs AVX-512 (F, BW, VL, VNNI) or AVX2 with FMA and F16C. The server checks this at startup and refuses with the actual numbers when it does not fit, which is better than failing three minutes into a load. NINFER_RAM_GUARD=0 overrides the check if you want to find out the hard way.
Qwen3.8-27B is dense, so it runs entirely on the GPUs and needs none of that.
The config table is the product
Four combinations, each with a stated cost. This is the most useful part of the repository, because it turns "can my machine run this" into a lookup rather than an experiment.
Qwen3.8-27B at 3-bit runs on a single 16 GB card with ordinary RAM. The cost is quality: GSM8K stays level with the NVFP4 27B, but MMLU-Pro drops to about 0.72 against 0.78.
Qwen3.8-27B at NVFP4 wants two 16 GB cards and nothing special in RAM beyond 8 GiB of context cache.
Flash-Next, 64 GB edition, runs on one or two cards with a 64 GB machine, around 56 GiB free. It uses ISTA-DASLab's GSQ-RCO experts, 48 GiB in RAM instead of 65, and the README reports quality on par with the NVFP4 build on GSM8K and MMLU-Pro. This is the configuration worth knowing about, because it is the only way to run the 125B model on a single card.
Flash-Next at NVFP4 is the default and needs two cards plus 96 GB of RAM, around 75 GiB free.
Disk follows the same pattern: 135 GB for Flash-Next NVFP4, 110 GB for the 64 GB edition, 20 GB and 15 GB for the 27B builds.
The hardware list is narrow and specific
GPUs need to be sm_120 with 16 GB of VRAM. RTX 5060 Ti 16 GB, 5070 Ti, 5080 and 5090 are reported working. A mixed pair runs at the pace of the slower card, and peer-to-peer is not needed, which matters if you are pairing cards across two generations.
The host needs Linux x86_64, with Ubuntu as what was tested, and an NVIDIA driver with CUDA 13.1 support, R580 or newer. The release does not need the CUDA toolkit; building does. Users report the 27B running on WSL2 and on Windows through Docker, and a native Windows build exists as a community port that is not in the official releases.
One detail deserves attention on Windows or WSL2: both limit page-locked memory to about half of system RAM, so the layers that do not fit stay in pageable RAM. Same tokens, slower.
Built for agents, not for benchmarks
Three features exist because this is meant to sit behind a coding agent rather than answer one question at a time. There is a prefix cache across turns, so the repeated context of a long session is not recomputed. Idle conversations move to RAM, optionally to disk, instead of being replayed. And the expert cache follows the conversation, so the experts a session keeps touching stay warm.
Anyone who has run a local model behind an editor knows the failure mode these address: the first turn is fine and the twentieth is not, because nothing is cached and the context is rebuilt every time.
Lineage, stated plainly
Capy Flash is ninfer-tp2, renamed at v0.5.0. It is the two-GPU tensor-parallel fork of Neroued/ninfer, which covered v0.1 through v0.4.9 under the old name. The NOTICE file lists every piece of prior work it builds on, and the README says plainly that it is not affiliated with Qwen or NVIDIA.
The current release is v0.7.7, built and measured on one machine, with other setups reported by users. That honesty about measurement scope is worth noting: the throughput numbers you will find are one person's hardware, not a benchmark suite.
Who this is for
It is not a general local-model runner. If you want to try fifty checkpoints, Ollama or LM Studio are the right tools and neither will ask you about AVX-512.
Capy Flash is for a narrower case. You have two 16 GB cards and 64 or 96 GB of RAM, you want a 125B model rather than a 27B one, and you are willing to accept a fixed checkpoint list, a C++ build and a 135 GB download to get it. The trade is explicit: the README does not pretend the model fits, it explains where the rest of it goes.
