The story of local inference in 2026 is less about raw speed and more about fitting. Fitting a model into memory you do not have, onto hardware that was never designed for it, or into a budget that cannot absorb a GPU bill. The projects below each attack that fitting problem from a different angle, and every one of them is live today.
Why the fitting problem moved to the front
For two years the constraint on running models locally was straightforward: you needed a graphics card with enough memory to hold the weights. That constraint has not disappeared, but it has been joined by a more interesting set of tricks. Quantization cut the memory per weight. Mixture-of-experts architectures made models large without making every token expensive. And runtimes learned to read weights from storage instead of holding them all at once.
What that means in practice is that the question is no longer "can my machine run a model" but "which trick fits the model I actually want." Here are eight projects, each answering that question differently.
A 26B model in about 2 GB of RAM
TurboFieldfare is a Swift and Metal runtime that runs Gemma 4 26B-A4B in roughly 2 GB of RAM on any Apple Silicon Mac, including the 8 GB models. That figure matters because the base MacBook Air is the machine most people actually try this on. The project is Apache-2.0, requires macOS 26 or later, and ships both a Mac application and a local server with an OpenAI-compatible endpoint, so the same runtime can back a chat window or a script with equal ease.
It also documents its system design, its benchmarks and the optimization experiments that produced them. If you want to understand where the memory actually goes in a large model, that documentation is worth reading on its own.
Stream the experts instead of loading them
Mixture-of-experts models are large but sparse: only a fraction of the weights are used for any given token. Swiftlet exploits precisely that. It is a Swift and Metal runtime that runs large Qwen MoE models on Apple devices by streaming expert weights from storage as they are needed, which is what makes 35B and 80B models usable on a Mac and even on an iPhone. The trade-off is that storage speed becomes part of your throughput.
Janas pursues the same goal in pure C, on CPU or GPU. The authors report 37 tokens per second from Qwen3-Next-80B, a 48 GB model, on a 32 GB laptop, against 2.8 tokens per second for llama.cpp, and faster results on all thirteen models they compared at Q4_K_M. Those are the project's own numbers, so treat them as a claim worth reproducing rather than an established fact. The direction, however, is consistent with what Swiftlet reports: if you can stream weights fast enough, RAM stops being a hard ceiling.
A games console as a model server
PS5LM is the most literal version of using what you already own. It ports llama.cpp to a jailbroken PlayStation 5 so the console GPU runs GGUF models and serves them to every device on your network, exposing OpenAI and Anthropic compatible APIs plus an interface designed for a television. It needs no PC while it runs, and it has since added support for Jev-style decision models.
The caveats are real: you need a jailbroken console, and console memory still caps how large a model you can load. But as a private household model server, it is difficult to beat on cost per token.
Knowing when the small model is wrong
Fitting a model is only half the problem. The other half is knowing when its answer is not good enough. Cactus Hybrid post-trains small on-device models to carry a confidence score with every answer, returned as structured data rather than parsed out of the answer text.
That turns a vague worry into an if statement: answer on-device when confidence is high, hand the query to a bigger model when it is low. The project publishes handoff rates across benchmarks including ChartQA, MMBench, LibriSpeech and MMLU-Pro, starting with a Gemma 4 E2B Hybrid build.
Compressing below one bit
LittleBit is the reference implementation of sub-1-bit LLM compression by latent factorization, published at NeurIPS 2025, together with a follow-up, LittleBit-2, at ICML 2026. It is research code rather than a packaged product, and the repository license is not a standard open source one. But if you want to understand how far weight compression can actually go, this is where the published numbers come from.
Letting an agent tune your kernels
Most local inference stacks are hand-tuned by people reading profiler output. Agentic CUDA Kernel Optimizer inverts that arrangement. It is a LangGraph workflow wrapped around a C++ CUDA harness, so an agent proposes kernel variants and the harness measures them.
The measurement loop is the entire point. A model suggesting CUDA changes in prose is producing a guess. A model suggesting changes that a compiler and a benchmark then judge is doing something closer to engineering.
Seeing what the model is doing
Once several models are running locally you need eyes on them. LLMxRay is an observability interface for models served through Ollama: one command opens a dashboard with token-by-token streaming, cache diagnostics, model comparison, benchmarks and cost tracking, alongside chat, reasoning, RAG and introspection views. It is also exposed as an MCP server, so an agent can query the same telemetry your dashboard shows.
What to pick first
If you own an Apple Silicon Mac and want a model running today, start with TurboFieldfare and check whether the model you want is in its supported set. If your model is bigger than your RAM, Swiftlet and Janas are the two projects attacking that directly, on different stacks. If you have hardware sitting idle, PS5LM is the most interesting reuse of it. And if you are putting any of this in front of real users, Cactus Hybrid is the one that tells you when to stop answering locally and pay for a bigger model instead.
