Run big models on spare GPUs

Mesh LLM v1.0 turns a set of machines into one local inference endpoint. n0's iroh team released the open-source project on July 11, 2026, with a simple product promise: pool GPUs and memory across devices, then expose them through an OpenAI-compatible API at http://localhost:9337/v1. 1 2

For PMs, the interesting part is not peer-to-peer networking by itself. The interesting part is that the application integration layer can stay almost unchanged. If a product already talks to an OpenAI-style API, Mesh LLM makes self-hosted or team-hosted inference look like a routing change, not a full rewrite. 2

The right mental model is a private inference pool for workloads that are expensive, repetitive, sensitive, or tolerant of slower responses. It is not a drop-in replacement for a managed frontier API in user-facing chat.

Mesh LLM has three serving modes. Local execution runs a model when one machine can fit it. Peer routing sends the request to another node that already has the model loaded. Skippy split divides a model by layer ranges across multiple machines when no single node can host the full model. 3 2

That last mode is the technical bet. Instead of requiring one box with enough VRAM, Skippy turns several smaller machines into a pipeline. One node runs early layers, the next node runs later layers, and activations move between stages over the network. The project documentation describes splits by layer ranges and says nodes download only the layer files assigned to them. 4

The networking layer comes from iroh, the peer-to-peer library underneath Mesh LLM. iroh uses QUIC over UDP by default, supports TLS 1.3 encryption, handles NAT traversal, and identifies endpoints by public keys rather than changing IP addresses. 5 6

The constraint is latency. Mesh LLM's own Skippy notes say the current chain sends activations downstream and waits for replies to walk back upstream, which makes network round trips a bottleneck during decoding. The project has already implemented speculative decode and direct return to reduce that cost, but those optimizations do not erase the basic physics of multi-node inference. 7

The published numbers are good enough to make Mesh LLM a pilot candidate, not good enough to make it a default serving layer. The project lists Qwen3-235B-A22B, a 235B-parameter mixture-of-experts model with 22B active parameters, running at 16 tokens per second across two nodes. 3 On Hacker News, i386 said he contributed to Mesh LLM and authored the Skippy engine; in a later comment, he reported GLM 5.2 at roughly 10 tokens per second on a Mac Studio M3 Ultra plus M1 Ultra setup over 1 Gbit Ethernet with about 5 ms simulated latency. 8 9

That points to a clear split:

Start by keeping the vendor abstraction. The product should call an OpenAI-compatible client through a configurable base_url, with cloud APIs and Mesh LLM sitting behind the same internal interface. Mesh LLM's local endpoint is localhost:9337/v1, and the project says tools such as Goose, VS Code, OpenCode, Pi.dev, Claude Code, and curl can use the API surface. 2 3

Then make the pilot private. A first test should use machines controlled by the same team or organization, not a public mesh. iroh gives the transport layer encryption and endpoint identity, but transport security is not the same as application-layer trust. The product risk is what each participating node can inspect or alter inside a distributed inference graph. 5 10

Choose one workload with a measurable fallback. A good candidate is a batch agent that enriches internal tickets, summarizes documents, or runs eval prompts overnight. Route a fixed percentage of that workload to Mesh LLM, keep the cloud model as fallback, and track latency, completion quality, failure rate, cost per completed task, and operator time spent keeping the mesh healthy.

Do not start with Skippy just because it is the novel part. The README states a single-machine-fit-first principle: if one node can host the full model, it serves the model locally without stage traffic. 2 A PM pilot should test local execution and peer routing first, then add split inference only if the target model is too large for any one available machine.

The roadmap is useful, but it should not be treated as committed product surface. Mesh LLM's roadmap says planned work includes an iOS-first mobile chat app, multimodal extensions for image generation and editing, TTS, video generation, desktop packaging, and SDK packaging. The same roadmap labels those items as ideas, not promises. 11

The PM takeaway is simple: Mesh LLM is an infrastructure option to test now and a serving dependency to trust later. Put it behind a router, bind it to private hardware, start with noninteractive workloads, and keep the cloud fallback live until reliability and trust evidence catches up with the architecture.