Skip to main content
localai.io

Architecture

Note
The install script, the macOS app and the latest container image give version 4.12.1. The add-ons, Office, Sandbox, Pages and Team in these docs need a newer build.

LocalAI is an API written in Go. It follows the OpenAI API, so software that uses an OpenAI SDK can use LocalAI instead. It runs on consumer hardware. The inference runs in separate backends, many of them in C++. Backends are gRPC servers. You can build your own gRPC server, or point LocalAI at an external one that it then manages.

LocalAI uses a mixture of backends written in various languages (C++, Golang, Python, …). You can check the model compatibility table to learn about all the components of LocalAI.

How LocalAI works: clients call one API on a small core with chats, agents, an add-on host and an MCP client and server, which talks over gRPC to a separate backend process for each model, running on your own hardware.How LocalAI works: clients call one API on a small core with chats, agents, an add-on host and an MCP client and server, which talks over gRPC to a separate backend process for each model, running on your own hardware. How a request flows: the client calls the API, the model config picks a backend, the backend process runs the model over gRPC on your hardware, and tokens stream back.How a request flows: the client calls the API, the model config picks a backend, the backend process runs the model over gRPC on your hardware, and tokens stream back.

For the history of the project, read LocalAI, from March 2023 to now.