Mesh LLM: Distributed AI Computing Pools Local GPUs into an OpenAI-Compatible API
Tags AI · Infrastructure · OSS · Enterprise
Iroh's Mesh LLM lets users pool GPUs across multiple machines into a single OpenAI-compatible endpoint running at localhost:9337/v1. The system uses iroh's QUIC-based P2P networking with NAT traversal to route inference requests locally, to peers with loaded models, or split large models across machines via layer-pipeline parallelism (dubbed "Skippy"). The 18 MB binary includes 40+ models from 0.5B to 235B parameters, and a mobile app using iroh's Swift SDK is planned with ACP agent protocol support.
Technical significance
This enables running large models without centralized APIs or massive single GPUs, using commodity hardware pools. The QUIC-based mesh with ALPN multiplexing means developers get a local OpenAI-compatible endpoint while compute routes dynamically across available GPUs. If adopted, it could reduce inference costs for teams with existing GPU fleets and weaken lock-in to closed model APIs.