Quickstart

LocalAI is a free, open-source alternative to OpenAI (Anthropic, etc.), functioning as a drop-in replacement REST API for local inferencing. It allows you to run LLMs, generate images, and produce audio, all locally or on-premises with consumer-grade hardware, supporting multiple model families and architectures.

LocalAI comes with a built-in web interface for chatting with models, managing installations, configuring AI agents, and more — no extra tools needed.

Quickstart

This guide assumes you have already installed LocalAI. If you haven’t installed it yet, see the Installation guide first.

Starting LocalAI

Once installed, start LocalAI. For Docker installations:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest-cpu

For GPU acceleration, choose the image that matches your hardware:

For NVIDIA GPUs, add --gpus all. For AMD/Intel/Vulkan, add the appropriate --device flags. See Container images for the full reference.

Using the Web Interface

Open http://localhost:8080 in your browser. The web interface lets you:

• Chat with any installed model
• Install models from the built-in gallery (Models page)
• Generate images, audio, and more
• Create and manage AI agents with MCP tool support
• Monitor system resources and loaded models
• Configure settings including GPU acceleration

To get started, navigate to the Models page, browse the gallery, and install a model. Once installed, head to the Chat page to start a conversation.

Downloading models from the CLI

When starting LocalAI (either via Docker or via CLI) you can specify as argument a list of models to install automatically before starting the API, for example:

local-ai run llama-3.2-1b-instruct:q4_k_m
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
local-ai run ollama://gemma:2b
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest

You can also manage models with the CLI:

local-ai models list          # List available models in the gallery
local-ai models install <name> # Install a model

For a full list of options, you can run LocalAI with --help or refer to the Linux Installation guide for installer configuration options.

Using the API

LocalAI exposes an OpenAI-compatible API. You can use it with any OpenAI SDK or client by pointing it to http://localhost:8080. For example:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.2-1b-instruct:q4_k_m",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

LocalAI also supports the Anthropic Messages API, the Open Responses API, and more. See Try it out for examples of all supported endpoints.

Built-in AI Agents

LocalAI includes a built-in AI agent platform with support for the Model Context Protocol (MCP). You can create agents that use tools, browse the web, execute code, and interact with external services — all from the web interface.

To get started with agents:

1. Install a model that supports tool calling (most modern LLMs do)
2. Navigate to the Agents page in the web interface
3. Create a new agent, configure its tools and system prompt
4. Start chatting — the agent will use tools autonomously

No separate installation required — agents are part of LocalAI.

Scaling with Distributed Mode

For production deployments or when you need more compute, LocalAI supports distributed mode with horizontal scaling:

• Distributed nodes: Add GPU worker nodes that self-register with a frontend coordinator
• P2P federation: Connect multiple LocalAI instances for load-balanced inference
• Model sharding: Split large models across multiple machines

See the Nodes page in the web interface or the Distribution docs for setup instructions.

What’s Next?

There is much more to explore! LocalAI supports video generation, voice cloning, embeddings, image understanding, and more. Check out:

• Container images reference
• Try the API endpoints
• All features
• Model gallery
• Run models manually
• Build from source
• Examples