Open source · MIT · v4.8.0
Make AI run onevery machine.
Text, voice, vision, images, video, 3D and agents, from one open runtime. It works on the laptop you already own, and scales to a room full of GPUs when you have one. When the engine we need is too heavy, too closed, or does not exist, we write it.
Engines we build
19 engines, written from scratch.
Those backends the core pulls in on demand are mostly ours. Most projects wrap somebody else's engine. We wrote ours, because the thing we needed came as a 9 GB Python install, or was closed, or did not exist. Each one is a binary and a GGUF file, checked against the reference implementation in CI.
parakeet.cpp
Twenty-seven times faster than whisper.cpp, on a CPU.
NVIDIA NeMo Parakeet, ported to C++ and ggml. Ten checkpoints, all of them verified at WER 0 against NeMo, which means the transcript comes out byte for byte identical while finishing first. Cache-aware streaming with end-of-utterance detection handles live audio, and the multilingual streaming model covers 40 or more locales.
depth-anything.cpp
Beats PyTorch on CPU, in half the memory.
Depth Anything 3 as a 99 MB file. It gives you metric depth, per-pixel confidence, camera intrinsics and extrinsics, and a back-projected point cloud you can export to glb or COLMAP. Output correlates 1.0 with the reference implementation, component by component, and there is no Python or CUDA toolkit anywhere at inference.
And the rest of them, running.
Every clip comes out of the benchmark suite that runs in CI on that engine. Where you see two panes, ours is racing the reference implementation on the same machine, on the same input.
In development, not announced yet
vllm.cpp
vLLM ported to C++20, with paged attention, continuous batching and prefix caching, on CPU, CUDA, Metal and Vulkan. It installs as 66 MB instead of a 9.1 GB virtualenv, and it stays ahead of vLLM at every concurrency level we have measured so far. Still being finished, so treat the numbers as provisional.
APEX quantization
The model you could not fit, on the card you already own.
The engine decides how fast a model runs. The weights decide whether it runs at all, so we build those too. APEX assigns a different precision to every tensor and every layer: a 35B mixture-of-experts model goes from 64.6 GB, out of reach of any consumer GPU, to 12.2 GB at 74 tokens a second. That is more than twice the speed of the original, and quality barely moves. The file is an ordinary GGUF, so stock llama.cpp opens it with no patches, and 201 of them are already sitting in the LocalAI gallery.
| Build | Size | Perplexity | HellaSwag | MMLU | tg128 t/s |
|---|---|---|---|---|---|
| F16 | 64.6 GB | 6.537 | 82.5% | 41.5% | 30.4 |
| Q8_0 | 34.4 GB | 6.533 | 83.0% | 41.2% | 52.5 |
| Unsloth UD-Q8_K_XL | 45.3 GB | 6.536 | 82.5% | 41.3% | 36.4 |
| APEX Quality | 21.3 GB | 6.527 | 83.0% | 41.2% | 62.3 |
| APEX I-Quality | 21.3 GB | 6.552 | 83.5% | 41.4% | 63.1 |
| APEX Compact | 16.1 GB | 6.783 | 82.5% | 40.9% | 69.8 |
| APEX Mini | 12.2 GB | 7.088 | 81.0% | 41.3% | 74.4 |
Qwen3.5-35B-A3B on an NVIDIA DGX Spark (GB10). Perplexity on wikitext-2-raw at context 2048. Full methodology in the technical report.
the agent
The agent is already in the binary.
Run local-ai chat and you are talking to an agent that already knows where your models are. It runs shell commands behind an approval gate you control, delegates to sub-agents, and loads MCP servers, plugins and skills. Claude Code plugins load as they are.
It is also nib, a single ~20 MB Go binary with no runtime and no daemon, so you can drop the same agent on any box you SSH into and press Ctrl+Space.
Since March 2023
Forty-eight thousand stars, and still shipping every week.
LocalAI has been trending on GitHub repeatedly since it launched, and it is one of the most starred self-hosted AI projects there is. 225 people have contributed code, 3,187 are in the Discord, and the README is kept translated into eight languages because the users are everywhere.
MAR 2023
First commit
An OpenAI-compatible API in front of llama.cpp, so a laptop could answer the same calls as the cloud.
JUL 2025
The core gets small
Every backend moves out of the binary. You install only the engines your models need.
MAR 2026
Agents and a new interface
Native agentic orchestration, a full React rewrite with canvas mode, WebRTC realtime audio.
APR 2026
It becomes a cluster
Distributed mode with VRAM-aware routing, autoscaling, multi-user auth and per-user quotas.
MAY 2026
It sees and hears
Voice recognition, face recognition with liveness, diarization, video generation, drop-in Ollama API.
JUL 2026
Nineteen engines of our own
The native C and C++ ports take over the heavy Python backends, one modality at a time.
Where it runs
What other people say
“Some cool ggml-based work by @mudler_it recently, make sure to check it out.”
@ggerganovGeorgi Gerganov, author of llama.cpp and ggml
On X, 2026 ↗“What a wonderful project: parakeet.cpp. A ggml based parakeet inference pipeline that is 2x faster than my ONNX parakeet pipeline on Apple Silicon.”
@badlogicgamesMario Zechner, author of pi.agent
On X, 2026 ↗“Build something good enough that the community chooses to carry it beyond your reach. This week, Ettore Di Giacinto brought NVIDIA Parakeet to the CPU.”
Adi MargolinOn LinkedIn
Read the post ↗Who shows up
225 people have put code in this repository.
Some of them do it from a desk at Microsoft, Spectro Cloud, VictoriaMetrics, Acronis, BearingPoint, Zendesk, Nextcloud, Scaleway, Kubermatic, Avanade, IBM, Ableton, Atlassian, TikTok, CockroachDB, MediaTek, Samsung SDS, Ant Group, Deloitte, solo.io and Qonto. Others from labs at MIT, University of Edinburgh, RMIT, Rochester Institute of Technology, University of Miami and USC.
That is where they work, not who sent them. Check out the full contributor list, 200+ and growing ↗
And many projects integrate it
Click any name to see where they say so.
Using LocalAI at work? Add your organisation to ADOPTERS.md.
Written about elsewhere
Built on, integrated with, written about
@sozercan builds on
AIKit now offers an extensible solution for finetuning LLMs! Thanks to @UnslothAI, you can finetune fast and efficiently. Then, deploy seamlessly with AIKit using @LocalAI_API for an end-to-end solution!
On X, 2024 ↗@ivanfioravanti benchmarks
An M5 Max with 40 GPU cores just beat an M3 Ultra with 80 on parakeet.cpp. Every model. ~1.7x faster on average, up to ~2x. Half the cores.
On X, 2026 ↗@PulumiCorp integration
Explore how to build and deploy a LLM app using @FlowiseAI and @LocalAI_API with AWS EKS, Pulumi, and TypeScript! Run your models locally or on-prem.
On X, 2024 ↗@enricoros ecosystem
Congrats to @LocalAI_API for launching LocalAGI (Agents), and LocalRecall (Memory). The Local stack is well designed and expanding.
On X, 2025 ↗@UniverseAdam in print
My hardcopies of the official @Raspberry_Pi magazine @TheMagPi have arrived! And they have my Automatic Speech Recognition project based around @NordVPN's Meshnet and a self-hosted @LocalAI_API language model inside.
On X, 2024 ↗@ivanfioravanti community
LocalAI is becoming stronger and better release, after release! Keep pushing @mudler_it and @LocalAI_API
On X, 2026 ↗@alepiad early days
I'm really excited about the prospect of open-source LLMs. In that respect, take a look at @LocalAI_API, a drop-in replacement for OpenAI's API but serving GGML models right on your own infrastructure.
On X, 2023 ↗@mattapperson builds on
Oh, high there @LocalAI_API, nice to see a terminal based UI for ya! (it's a WIP, but just wanted something cleaner then CURL calls)
On X, 2023 ↗From the team
We publish the numbers, including the ones that cost us.
APEX takes a 35B model from 64.6 GB to 12.2 GB, and from 30.4 to 74.4 tokens a second. Perplexity goes from 6.537 to 7.088. That trade is in the post too, with the command that produced it.
1 August 2026
What landed in LocalAI 4.8
A new inference engine, 3D generation, one backend that serves six audio endpoints, and a web interface 3.48x lighter. 321 pull requests in eighteen days.
Read the post →29 July 2026
LocalAI, from March 2023 to now
Three years, 133 releases and 224 contributors later. The four changes that mattered most were making the core small, adding agents, making it a cluster, …
Read the post →24 July 2026
Why we write our own C and C++ engines
A 66 MiB binary instead of a 9.1 GiB virtualenv, depth estimation that beats PyTorch on CPU in half the memory, and biometrics that match insightface bit …
Read the post →