Colibri Runs GLM-5.2 (744B MoE) on Consumer Hardware: Pure C Engine Streams Experts from Disk at int4
Tags AI · Infrastructure · OSS · Startup
Colibri is a single-file C engine (~2,400 lines) that runs the 744B-parameter GLM-5.2 Mixture-of-Experts model on a consumer machine with 25 GB RAM by streaming experts from disk. The dense portion (~17B params, 9.9 GB at int4) stays resident; 21,504 routed experts (~370 GB at int4) live on NVMe and load on demand with per-layer LRU caching, OS page cache as L2, and learned hot-expert pinning. Features include MLA attention with 57× compressed KV-cache, native MTP speculative decoding (2.2–2.8 tok/forward at int8), grammar-forced speculative drafts, DSA sparse attention, KV-cache persistence across restarts, and an OpenAI-compatible API server. Built and tested on a 12-core laptop with 25 GB RAM — no GPU required.
Technical significance
This one-person project demonstrates that frontier MoE models (744B params) can run on consumer hardware without GPUs, using disk streaming and aggressive KV compression. The auto-scaling expert cache and learned hot-expert pinning mean performance improves with use. For the AI ecosystem, it proves model size is no longer a hard barrier to local inference — disk bandwidth and RAM are the tunable constraints. If the engine matures, it could democratize access to frontier models for developers and researchers without cloud GPU budgets.