Install LocalAI

LocalAI can be installed in multiple ways depending on your platform and preferences.

Video Walkthrough

Installation Methods

Choose the installation method that best suits your needs:

1. Containers ⭐ Recommended - Works on all platforms, supports Docker and Podman
2. macOS - Download and install the DMG application
3. Linux - Install on Linux using binaries
4. Kubernetes - Deploy LocalAI on Kubernetes clusters
5. Build from Source - Build LocalAI from source code

Quickstart

LocalAI is a free, open-source alternative to OpenAI (Anthropic, etc.), functioning as a drop-in replacement REST API for local inferencing. It allows you to run LLMs, generate images, and produce audio, all locally or on-premises with consumer-grade hardware, supporting multiple model families and architectures.

LocalAI comes with a built-in web interface for chatting with models, managing installations, configuring AI agents, and more, with no extra tools needed.

Quickstart

This guide assumes you have already installed LocalAI. If you haven’t installed it yet, see the Installation guide first.

Starting LocalAI

Once installed, start LocalAI. For Docker installations:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest

For GPU acceleration, choose the image that matches your hardware:

For NVIDIA GPUs, add --gpus all. For AMD/Intel/Vulkan, add the appropriate --device flags. See Container images for the full reference.

Using the Web Interface

Open http://localhost:8080 in your browser. The web interface lets you:

• Chat with any installed model
• Explore, install, and manage models from the Models page
• Generate images, audio, and more
• Create and manage AI agents with MCP tool support
• Monitor system resources and loaded models
• Configure settings including GPU acceleration

To get your first chat working:

1. Open Models → Explore and search for qwen3-4b. Click Install on the qwen3-4b entry and wait for the download to finish. (qwen3-4b is a small, CPU-friendly Qwen3 model that also supports tool calling, so you can reuse it later in the Build your first agent walkthrough.)
2. Open the Chat page, select qwen3-4b from the model dropdown, type a message, and send it. You should get a reply within a few seconds.

To correct an earlier prompt or response without running the model again, hover over the saved message and select Edit. Save updates that conversation’s local history; Cancel discards the draft.

Downloading models from the CLI

When starting LocalAI (either via Docker or via CLI) you can specify as argument a list of models to install automatically before starting the API, for example:

local-ai run qwen3-4b
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
local-ai run ollama://gemma:2b
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest

You can also manage models with the CLI:

local-ai models list          # List available models in the gallery
local-ai models install <name> # Install a model

For a full list of options, run LocalAI with --help, or see the Linux Installation guide.

Using the API

LocalAI exposes an OpenAI-compatible API. You can use it with any OpenAI SDK or client by pointing it to http://localhost:8080. For example:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

LocalAI also supports the Anthropic Messages API, the Open Responses API, and more. See Try it out for examples of all supported endpoints.

Built-in AI Agents

LocalAI includes a built-in AI agent platform with support for the Model Context Protocol (MCP). You can create agents that use tools, browse the web, execute code, and interact with external services, all from the web interface.

To get started with agents:

1. Install a model that supports tool calling (most modern LLMs do)
2. Navigate to the Agents page in the web interface
3. Create a new agent, configure its tools and system prompt
4. Start chatting; the agent will use tools autonomously

No separate installation required: agents are part of LocalAI. For a full step-by-step walkthrough, see Build your first agent.

Scaling with Distributed Mode

For production deployments or when you need more compute, LocalAI supports distributed mode with horizontal scaling:

• Distributed nodes: Add GPU worker nodes that self-register with a frontend coordinator
• P2P federation: Connect multiple LocalAI instances for load-balanced inference
• Model sharding: Split large models across multiple machines

See the Nodes page in the web interface or the Distributed inference docs for setup instructions.

What’s Next?

There is much more to explore! LocalAI supports video generation, voice cloning, embeddings, image understanding, and more. Check out:

• Container images reference
• Try the API endpoints
• All features
• Model gallery
• Run models manually
• Build from source
• Examples

Setting Up Models

This section covers everything you need to know about installing and configuring models in LocalAI. You’ll learn multiple methods to get models running.

Prerequisites

• LocalAI installed and running (see Quickstart if you haven’t set it up yet)
• Basic understanding of command line usage

Method 1: Using the Model Gallery (Easiest)

The Model Gallery is the simplest way to install models. It provides pre-configured models ready to use.

GPU recommendations require a memory estimate within 95% of the detected model memory budget at a 4096-token context. If none of the sampled candidates fit, the recommendation section is hidden. You can still browse the gallery and check individual models at your intended context size. On the Models page the recommendations appear as a “Best for this machine” list in the pane beside the table while no model is selected. Once you have installed a model, the list shows only the best fit and offers the others behind “more that fit”. The Home page also omits static GPU suggestions when no fitting recommendation is available.

Via WebUI

1. Open the LocalAI WebUI at http://localhost:8080
2. Navigate to Models → Explore
3. Browse or search the available models
4. Click “Install” on any model you want
5. Wait for installation to complete. Progress appears in the strip at the top of the app, and Operate → Activity shows every install in flight, plus what failed and what finished (see Activity)

For more details, refer to the Gallery Documentation.

The same Models page owns the complete lifecycle. Switch to Installed to search local configurations, filter them by running, idle, disabled, pinned, or distributed state, and open a model’s runtime controls. Load and stop are on each row; edit, pin, disable, backend logs, and remove are in the row menu and in the details of the selected model. The current view, search, filter, and selection are stored in the URL so links and browser history preserve your place. The disk strip in the page header shows the free space on the models disk and opens a review of what can be removed to free more (see Model gallery).

Via CLI

# List available models
local-ai models list

# Install a specific model
local-ai models install llama-3.2-1b-instruct:q4_k_m

# Start LocalAI with a model from the gallery
local-ai run llama-3.2-1b-instruct:q4_k_m

To run models available in the LocalAI gallery, you can use the model name as the URI. For example, to run LocalAI with the Hermes model, execute:

local-ai run hermes-2-theta-llama-3-8b

To install only the model, use:

local-ai models install hermes-2-theta-llama-3-8b

Note: The galleries available in LocalAI can be customized to point to a different URL or a local directory. For more information on how to setup your own gallery, see the Gallery Documentation.

Browse Online

Visit models.localai.io to browse all available models in your browser.

On a phone, the model page wraps build descriptions in the Variants table to keep the fit status and install buttons visible.

Method 1.5: Import Models via WebUI

The WebUI import page (Build → Import) takes either a source to resolve or a configuration to write. Both live on the same page, behind the two tabs in its header. From a source, it is a short guided flow: source, review, import, done.

From a source

1. Open the LocalAI WebUI at http://localhost:8080
2. Open Build, then Import
3. Paste the source into the Source field (e.g. https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-GGUF), or start from one of the examples
4. Review what the page says, then press Enter or click Import

While you type, the page reads the spelling of the source (Hugging Face repository, direct URL, configuration file, OCI image, Ollama model, or a path on the host) and shows the request the form will send. It also runs the checks that can be made before the import starts: whether the name is already in use, whether the chosen backend is installed, and how much disk and memory are free. The server returns no preview of the model configuration, the size or the memory a model needs before the import starts, so the page does not show them. They appear once the import starts, next to the free memory and disk.

The What you can paste panel beside the field lists every accepted scheme: huggingface://, hf://, a full Hugging Face URL, any direct https:// URL, file:// and absolute paths on the host, oci://, ocifile://, and ollama://.

Expand Advanced options directly below the request preview to override settings before importing. The controls include backend, name, description, quantizations, MMProj quantizations, model type, embeddings support, the diffusers-specific fields, and arbitrary custom key-value preferences. The backend list can be narrowed by modality first. Fields that the selected backend cannot use are hidden, and anything you typed into them is kept in case you switch back.

The source format check does not contact the source or validate it as a model. An HTTPS URL can point to a web page rather than model weights.

Leaving the backend on auto-detect lets LocalAI choose from the source. Selecting a backend sends an explicit backend preference in the previewed request. If more than one installed backend can serve the detected modality, the page says so and offers the candidates inline — picking one resubmits the import.

Repositories under mlx-community are imported with the native MLX backend. LocalAI uses Hugging Face’s pipeline metadata to select mlx-vlm for vision-language models and mlx-audio for text-to-speech models; other MLX repositories use mlx. An explicit backend selection in the import form always overrides this automatic routing.

Once the import starts, the page reports the download size and the memory the model needs against what is free, then the current phase, the bytes transferred and a progress bar. When the model is ready, the page names it and links to a chat with it and to Models.

Writing YAML

For full control over model configuration, switch to the Write YAML tab and edit the configuration directly, then click Create. The editor provides syntax highlighting and a copy button, and accepts the same configuration keys documented under Advanced.

This is especially useful for:

• Custom model configurations
• Fine-tuning model parameters
• Setting up complex model setups
• Editing existing model configurations

When the configuration references remote assets, Create does not download them inside the request: the import is queued like a gallery installation, and the page reports the job’s progress until the model is ready. These configurations are queued:

• a URL in parameters.model or mmproj, or download_files entries;
• an hf:// reference or a bare owner/repo in parameters.model, on a backend that consumes a model directory;
• an explicit artifacts: block. The configuration file is written once every download has finished, so a failed download never leaves a config pointing at files that do not exist. A configuration that only references files already on disk is written immediately, as before.

A queued import runs through the same installer as a gallery installation, so it differs from an immediate write in two ways:

• Recommended sampling parameters for the model family (for example temperature and top_p) are added under parameters: when the configuration does not set them. The immediate write stores your configuration exactly as posted.
• When automatic backend installation is enabled, the backend named in the configuration is installed as part of the job.

Set the parameters you care about explicitly if you need the same stored file in both cases.

The same split applies to the API behind the page: POST /models/import answers a queued import with a job id and a status URL to poll (/models/jobs/{id} — the same shape gallery installs use), and a download-free import with a plain success response.

If you cancel a queued import, the import page and model editor report the cancellation and stay on the current page. They report success only after the job succeeds.

Method 2: Installing from Hugging Face

LocalAI can directly install models from Hugging Face:

# Install and run a model from Hugging Face
local-ai run huggingface://TheBloke/phi-2-GGUF

The format is: huggingface://<repository>/<model-file> ( is optional)

Examples

local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf

Method 3: Installing from OCI Registries

Ollama Registry

local-ai run ollama://gemma:2b

Standard OCI Registry

local-ai run oci://localai/phi-2:latest

Run Models via URI

To run models via URI, specify a URI to a model file or a configuration file when starting LocalAI. Valid syntax includes:

• file://path/to/model (absolute path to a file within your models directory)
• huggingface://repository_id/model_file (e.g., huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf)
• From OCIs: oci://container_image:tag, ollama://model_id:tag
• From configuration files: https://gist.githubusercontent.com/.../phi-2.yaml

Configuration files can be used to customize the model defaults and settings. For advanced configurations, refer to the Customize Models section.

Examples

local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
local-ai run ollama://gemma:2b
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest

Method 4: Manual Installation

For full control, you can manually download and configure models.

Step 1: Download a Model

Download a GGUF model file. Popular sources:

• Hugging Face

Example:

mkdir -p models

wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q4_K_M.gguf \
  -O models/phi-2.Q4_K_M.gguf

Step 2: Create a Configuration File (Optional)

Create a YAML file to configure the model:

# models/phi-2.yaml
name: phi-2
parameters:
  model: phi-2.Q4_K_M.gguf
  temperature: 0.7
context_size: 2048
threads: 4
backend: llama-cpp

Customize model defaults and settings with a configuration file. For advanced configurations, refer to the Advanced Documentation.

Step 3: Run LocalAI

Choose one of the following methods to run LocalAI:

mkdir models

cp your-model.gguf models/

docker run -p 8080:8080 -v $PWD/models:/models -ti --rm localai/localai:latest --models-path /models --context-size 700 --threads 4

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "your-model.gguf",
     "prompt": "A long time ago in a galaxy far, far away",
     "temperature": 0.7
   }'

mkdir models

wget https://huggingface.co/TheBloke/Luna-AI-Llama2-Uncensored-GGUF/resolve/main/luna-ai-llama2-uncensored.Q4_0.gguf -O models/luna-ai-llama2

cp -rf prompt-templates/getting_started.tmpl models/luna-ai-llama2.tmpl

docker run -p 8080:8080 -v $PWD/models:/models -ti --rm localai/localai:latest --models-path /models --context-size 700 --threads 4

curl http://localhost:8080/v1/models

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "luna-ai-llama2",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9
   }'

git clone https://github.com/go-skynet/LocalAI

cd LocalAI

cp your-model.gguf models/

docker compose up -d --pull always

curl http://localhost:8080/v1/models

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "your-model.gguf",
     "prompt": "A long time ago in a galaxy far, far away",
     "temperature": 0.7
   }'

# With binary
local-ai --models-path ./models

GPU Acceleration

For instructions on GPU acceleration, visit the GPU Acceleration page.

For more model configurations, visit the Examples Section.

Understanding Model Files

File Formats

• GGUF: Modern format, recommended for most use cases
• GGML: Older format, still supported but deprecated

Quantization Levels

Models come in different quantization levels (quality vs. size trade-off):

Choosing the Right Model

Consider:

• RAM available: Larger models need more RAM
• Use case: Different models excel at different tasks
• Speed: Smaller quantizations are faster
• Quality: Higher quantizations produce better output

Model Configuration

Basic Configuration

Create a YAML file in your models directory:

name: my-model
parameters:
  model: model.gguf
  temperature: 0.7
  top_p: 0.9
context_size: 2048
threads: 4
backend: llama-cpp

Advanced Configuration

See the Model Configuration guide for all available options.

Managing Models

List Installed Models

Ollama clients can list configured models with GET /api/tags and loaded models with GET /api/ps. Each entry includes size in bytes when LocalAI can resolve a non-empty primary weights file on disk. This is the size of that file, not the total size of a multi-file model or its memory use. Unknown sizes are omitted; /api/ps also omits size_vram because per-model VRAM use is not available.

# Via API
curl http://localhost:8080/v1/models

# Via CLI
local-ai models list

Disk Usage

In the WebUI, the Installed tab of Models shows how much disk each installed configuration uses. The Size column holds the size on disk, and for a model that shares files with others it adds the shared part (“1.2 GB shared”). A file referenced by more than one configuration is counted once in the total under the table, which also gives the shared total. The inspector beside the table names the models a model shares files with, and a model page lists its files under Usage and history, with each file’s size, the other models that use it, and whether it is missing. Removing a model frees only its exclusive files; the files it shares stay while another configuration uses them. A reference whose file is not on disk (a download that never finished, or a file removed by hand) is marked on the model and in the cleanup review.

Only an admin can read the report. For anyone else, or if the read fails, the Size column shows the size of the files the gallery lists, and “size unknown” for a model the gallery does not list.

The report behind these views is available directly (admin only):

curl http://localhost:8080/api/models/storage

It returns per-model sizes with the shared portion split out, the file-to-models relation, and missing references. Files in the models directory that no configuration references are not included.

Remove Models

Simply delete the model file and configuration from your models directory:

rm models/model-name.gguf
rm models/model-name.yaml  # if exists

Troubleshooting

Model Not Loading

1. Check backend: Ensure the required backend is installed

   local-ai backends list
   local-ai backends install llama-cpp  # if needed

2. Check logs: Enable debug mode

   DEBUG=true local-ai

3. Verify file: Ensure the model file is not corrupted

Out of Memory

• Use a smaller quantization (Q4_K_S or Q2_K)
• Reduce context_size in configuration
• Close other applications to free RAM

Wrong Backend

Check the Compatibility Table to ensure you’re using the correct backend for your model.

Best Practices

1. Start small: Begin with smaller models to test your setup
2. Use quantized models: Q4_K_M is a good balance for most use cases
3. Organize models: Keep your models directory organized
4. Backup configurations: Save your YAML configurations
5. Monitor resources: Watch RAM and disk usage

Try it out

Once LocalAI is installed, you can start it (either by using docker, or the cli, or the systemd service).

By default the LocalAI WebUI should be accessible from http://localhost:8080. You can also use 3rd party projects to interact with LocalAI as you would use OpenAI (see also Integrations ).

After installation, install new models by navigating the model gallery, or by using the local-ai CLI.

You can test chat models from the CLI without keeping a separate curl command around:

# Terminal 1
local-ai run

# Terminal 2
local-ai chat --model qwen3-4b

local-ai chat connects to a running LocalAI server and opens the built-in terminal agent. It is more than a chat prompt: it can read your files and run commands on your machine, so the first time it wants to do something that changes state it stops and asks you to approve the call. Reads and searches run without asking. Press Esc or Ctrl+C to end the session. Esc is a key of the full-screen interface: in the plain --cli mode, leave with Ctrl+C, Ctrl+D, or by typing exit.

Use /models to list installed models, /model <name> to switch models while keeping the conversation, and /compact to summarize the history so far when the context fills up. The full picture is on the Terminal agent page.

If the server exposes exactly one model, LocalAI uses that model automatically:

# Terminal 1
local-ai run qwen3-4b

# Terminal 2
local-ai chat

When more than one model is configured, the agent asks you to pick one and remembers your answer, or you can pass --model with the installed model name. Use --endpoint to connect to a non-default server, for example local-ai chat --endpoint http://127.0.0.1:8081 --model qwen3-4b.

You can also test out the API endpoints using curl. A few examples are listed below.

Text Generation

Creates a model response for the given chat conversation. OpenAI documentation.

curl http://localhost:8080/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{ "model": "qwen3-4b", "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}] }' 

GPT Vision

Understand images.

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{ 
        "model": "gpt-4-vision-preview", 
        "messages": [
          {
            "role": "user", "content": [
              {"type":"text", "text": "What is in the image?"},
              {
                "type": "image_url", 
                "image_url": {
                  "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" 
                }
              }
            ], 
          "temperature": 0.9
          }
        ]
      }' 

Function calling

Call functions

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b",
    "messages": [
      {
        "role": "user",
        "content": "What is the weather like in Boston?"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_current_weather",
          "description": "Get the current weather in a given location",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {
                "type": "string",
                "description": "The city and state, e.g. San Francisco, CA"
              },
              "unit": {
                "type": "string",
                "enum": ["celsius", "fahrenheit"]
              }
            },
            "required": ["location"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'

Anthropic Messages API

LocalAI supports the Anthropic Messages API for Claude-compatible models. Anthropic documentation.

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "gpt-4",
    "max_tokens": 1024,
    "messages": [
      {"role": "user", "content": "How are you doing?"}
    ],
    "temperature": 0.7
  }'

Open Responses API

LocalAI supports the Open Responses API specification with support for background processing, streaming, and advanced features. Open Responses documentation.

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "input": "Say this is a test!",
    "max_output_tokens": 1024,
    "temperature": 0.7
  }'

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4",
    "input": "Generate a long story",
    "max_output_tokens": 4096,
    "background": true
  }'

curl http://localhost:8080/v1/responses/<response_id>

Image Generation

Creates an image given a prompt. OpenAI documentation.

curl http://localhost:8080/v1/images/generations \
      -H "Content-Type: application/json" -d '{
          "prompt": "A cute baby sea otter",
          "size": "256x256"
        }'

Text to speech

Generates audio from the input text. OpenAI documentation.

curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "The quick brown fox jumped over the lazy dog.",
    "voice": "alloy"
  }' \
  --output speech.mp3

Audio Transcription

Transcribes audio into the input language. OpenAI Documentation.

wget --quiet --show-progress -O gb1.ogg https://upload.wikimedia.org/wikipedia/commons/1/1f/George_W_Bush_Columbia_FINAL.ogg 

curl http://localhost:8080/v1/audio/transcriptions \
    -H "Content-Type: multipart/form-data" \
    -F file="@$PWD/gb1.ogg" -F model="whisper-1"

Embeddings Generation

Get a vector representation of a given input that can be easily consumed by machine learning models and algorithms. OpenAI Embeddings.

curl http://localhost:8080/embeddings \
    -X POST -H "Content-Type: application/json" \
    -d '{ 
        "input": "Your text string goes here", 
        "model": "text-embedding-ada-002"
      }'

Customizing the Model

To customize the prompt template or the default settings of the model, a configuration file is utilized. This file must adhere to the LocalAI YAML configuration standards. For comprehensive syntax details, refer to the advanced documentation. The configuration file can be located either remotely (such as in a Github Gist) or within the local filesystem or a remote URL.

LocalAI can be initiated using either its container image or binary, with a command that includes URLs of model config files or utilizes a shorthand format (like huggingface:// or github://), which is then expanded into complete URLs.

The configuration can also be set via an environment variable. For instance:

local-ai github://owner/repo/file.yaml@branch

MODELS="github://owner/repo/file.yaml@branch,github://owner/repo/file.yaml@branch" local-ai

Here’s an example to initiate the phi-2 model:

docker run -p 8080:8080 localai/localai:v4.11.0 https://gist.githubusercontent.com/mudler/ad601a0488b497b69ec549150d9edd18/raw/a8a8869ef1bb7e3830bf5c0bae29a0cce991ff8d/phi-2.yaml

You can also check all the embedded models configurations here.

Example: Customizing the Prompt Template

To modify the prompt template, create a Github gist or a Pastebin file, and copy the content from https://github.com/mudler/LocalAI-examples/blob/main/configurations/phi-2.yaml. Alter the fields as needed:

name: phi-2
context_size: 2048
f16: true
threads: 11
gpu_layers: 90
mmap: true
parameters:
  # Reference any HF model or a local file here
  model: huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
  temperature: 0.2
  top_k: 40
  top_p: 0.95
template:
  
  chat: &template |
    Instruct: {{.Input}}
    Output:
  # Modify the prompt template here ^^^ as per your requirements
  completion: *template

Then, launch LocalAI using your gist’s URL:

## Important! Substitute with your gist's URL!
docker run -p 8080:8080 localai/localai:v4.11.0 https://gist.githubusercontent.com/xxxx/phi-2.yaml

Next Steps

• Visit the advanced section for more insights on prompt templates and configuration files.
• To learn about fine-tuning an LLM model, check out the fine-tuning section.

Troubleshooting

This guide covers common issues you may encounter when using LocalAI, organized by category. For each issue, diagnostic steps and solutions are provided.

Quick Diagnostics

Before diving into specific issues, run these commands to gather diagnostic information:

# Check LocalAI is running and responsive
curl http://localhost:8080/readyz

# List loaded models
curl http://localhost:8080/v1/models

# Check LocalAI version
local-ai --version

# Enable debug logging for detailed output
DEBUG=true local-ai run
# or
local-ai run --log-level=debug

For Docker deployments:

# View container logs
docker logs local-ai

# Check container status
docker ps -a | grep local-ai

# Test GPU access (NVIDIA)
docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi

Request latency diagnostics

Debug logging alone does not enable profiling or request phase timing. Use Profiling and request diagnostics for an opt-in, bounded investigation on one frontend. The guide distinguishes CPU execution, lock waits, discovery filesystem work, and the model router callback. It includes direct pod port-forward instructions and warnings about unauthenticated, sensitive profiles. Enable diagnostics only through a separately approved rollout; do not expose profiling through the public API. Report non-reproduction if matched requests do not reproduce the delay.

Installation Issues

Binary Won’t Execute on Linux

Symptoms: Permission denied or “cannot execute binary file” errors.

Solution:

chmod +x local-ai-*
./local-ai-Linux-x86_64 run

If you see “cannot execute binary file: Exec format error”, you downloaded the wrong architecture. Verify with:

uname -m
# x86_64 → download the x86_64 binary
# aarch64 → download the arm64 binary

macOS: Application Is Quarantined

Symptoms: macOS blocks LocalAI from running because the DMG is not signed by Apple.

Solution: See GitHub issue #6268 for quarantine bypass instructions. This is tracked for resolution in issue #6244.

Model Loading Problems

Model Not Found

Symptoms: API returns 404 or "model not found" error.

Diagnostic steps:

1. Check the model exists in your models directory:

   ls -la /path/to/models/

2. Verify your models path is correct:

   # Check what path LocalAI is using
   local-ai run --models-path /path/to/models --log-level=debug

3. Confirm the model name matches your request:

   # List available models
   curl http://localhost:8080/v1/models | jq '.data[].id'

Model Fails to Load (Backend Error)

Symptoms: Model is found but fails to load, with backend errors in the logs.

Common causes and fixes:

• Wrong backend: Ensure the backend in your model YAML matches the model format. GGUF models use llama-cpp, diffusion models use diffusers, etc. See the compatibility table for details.
• Backend not installed: Check installed backends:

  local-ai backends list
  # Install a missing backend:
  local-ai backends install llama-cpp

• Corrupt model file: Re-download the model. Partial downloads or disk errors can corrupt files.
• Wrong model format: LocalAI uses GGUF format for llama.cpp models. Older GGML format is deprecated.

Model Configuration Issues

Symptoms: Model loads but produces unexpected results or errors during inference.

Check your model YAML configuration:

# Example model config
name: my-model
backend: llama-cpp
parameters:
  model: my-model.gguf  # Relative to models directory
context_size: 2048
threads: 4  # Should match physical CPU cores

Common mistakes:

• model path must be relative to the models directory, not an absolute path
• threads set higher than physical CPU cores causes contention
• context_size too large for available RAM causes OOM errors

GPU and Memory Issues

GPU Not Detected

NVIDIA (CUDA):

# Verify CUDA is available
nvidia-smi

# For Docker, verify GPU passthrough
docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi

When working correctly, LocalAI logs should show: ggml_init_cublas: found X CUDA devices.

Ensure you are using a CUDA-enabled container image (tags containing cuda11, cuda12, or cuda13). CPU-only images cannot use NVIDIA GPUs.

AMD (ROCm):

# Verify ROCm installation
rocminfo

# Docker requires device passthrough
docker run --device=/dev/kfd --device=/dev/dri --group-add=video ...

If your GPU is not in the default target list, open up an Issue. Supported targets include: gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101, gfx1102, gfx1200, gfx1201.

Intel (SYCL):

# Docker requires device passthrough
docker run --device /dev/dri ...

Use container images with gpu-intel in the tag. Known issue: SYCL hangs when mmap: true is set - disable it in your model config:

mmap: false

If the Intel diffusers backend exits while importing PyTorch with libsycl.so.9: undefined symbol: urDeviceWaitExp, version LIBUR_LOADER_0.12, the SYCL library has loaded an incompatible Unified Runtime loader. This happens before the gRPC server starts; later connection-refused errors are a consequence of that import failure.

Update the separately installed Intel diffusers backend as well as the gpu-intel container image. Updating the parent image alone does not replace a backend retained in your backends directory. On Linux, the Python backend launcher prefers venv/lib when it contains libsycl.so*, ahead of the backend’s packaged lib directory and inherited LD_LIBRARY_PATH. The venv still needs a compatible SYCL library, Unified Runtime loader and adapters; changing search order cannot repair an incomplete runtime installation.

If the failure persists, include the container image digest, installed backend name and OCI digest, full import traceback, launch command, and any library mounts in your issue. From inside the affected container, record the inherited LD_LIBRARY_PATH, the installed backend path, and the libsycl.so*, libur_loader.so* and libur_adapter_* files in both venv/lib and lib. Use readelf --dyn-syms --wide and readelf --version-info on the SYCL library and loaders to compare their symbols and versions. Running the installed backend’s run.sh with LD_DEBUG=libs,versions captures which loader is actually selected under its startup environment. An import torch check using the venv Python and the same library search path can isolate this failure without generating an image.

Overriding backend auto-detection:

If LocalAI picks the wrong GPU backend, override it:

LOCALAI_FORCE_META_BACKEND_CAPABILITY=nvidia local-ai run
# Options: default, nvidia, amd, intel

Out of Memory (OOM)

Symptoms: Model loading fails or the process is killed by the OS.

Solutions:

1. Use smaller quantizations: Q4_K_S or Q2_K use significantly less memory than Q8_0 or Q6_K
2. Reduce context size: Lower context_size in your model YAML
3. Enable low VRAM mode: Add low_vram: true to your model config
4. Limit active models: Only keep one model loaded at a time:

   local-ai run --max-active-backends=1

5. Enable idle watchdog: Automatically unload unused models:

   local-ai run --enable-watchdog-idle --watchdog-idle-timeout=10m

6. Manually unload a model:

   curl -X POST http://localhost:8080/backend/shutdown \
     -H "Content-Type: application/json" \
     -d '{"model": "model-name"}'

Models Stay Loaded and Consume Memory

By default, models remain loaded in memory after first use. This can exhaust VRAM when switching between models.

Configure LRU eviction:

# Keep at most 2 models loaded; evict least recently used
local-ai run --max-active-backends=2

Configure watchdog auto-unload:

local-ai run \
  --enable-watchdog-idle --watchdog-idle-timeout=15m \
  --enable-watchdog-busy --watchdog-busy-timeout=5m

These can also be set via environment variables (LOCALAI_WATCHDOG_IDLE=true, LOCALAI_WATCHDOG_IDLE_TIMEOUT=15m) or in the Web UI under Settings → Watchdog Settings.

See the VRAM Management guide for more details.

API Connection Problems

Connection Refused

Symptoms: curl: (7) Failed to connect to localhost port 8080: Connection refused

Diagnostic steps:

1. Verify LocalAI is running:

   # Direct install
   ps aux | grep local-ai
   
   # Docker
   docker ps | grep local-ai

2. Check the bind address and port:

   # Default is :8080. Override with:
   local-ai run --address=0.0.0.0:8080
   # or
   LOCALAI_ADDRESS=":8080" local-ai run

3. Check for port conflicts:

   ss -tlnp | grep 8080

Authentication Errors (401)

Symptoms: 401 Unauthorized response.

If API key authentication is enabled (LOCALAI_API_KEY or --api-keys), include the key in your requests:

curl http://localhost:8080/v1/models \
  -H "Authorization: Bearer YOUR_API_KEY"

Keys can also be passed via x-api-key or xi-api-key headers.

Request Errors (400/422)

Symptoms: 400 Bad Request or 422 Unprocessable Entity.

Common causes:

• Malformed JSON in request body
• Missing required fields (e.g., model or messages)
• Invalid parameter values (e.g., negative top_n for reranking)

Enable debug logging to see the full request/response:

DEBUG=true local-ai run

See the API Errors reference for a complete list of error codes and their meanings.

Performance Issues

Slow Inference

Diagnostic steps:

1. Enable debug mode to see inference timing:

   DEBUG=true local-ai run

2. Use streaming to measure time-to-first-token:

   curl http://localhost:8080/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model": "my-model", "messages": [{"role": "user", "content": "Hello"}], "stream": true}'

Common causes and fixes:

• Model on HDD: Move models to an SSD. If stuck with HDD, disable memory mapping (mmap: false) to load the model entirely into RAM.
• Thread overbooking: Set --threads to match your physical CPU core count (not logical/hyperthreaded count).
• Default sampling: LocalAI uses mirostat sampling by default, which produces better quality output but is slower. Disable it for benchmarking:

  # In model config
  mirostat: 0

• No GPU offloading: Ensure gpu_layers is set in your model config to offload layers to GPU:

  gpu_layers: 99  # Offload all layers

• Context size too large: Larger context sizes require more memory and slow down inference. Use the smallest context size that meets your needs.

High Memory Usage

• Use quantized models (Q4_K_M is a good balance of quality and size)
• Reduce context_size
• Enable low_vram: true in model config
• Disable mmlock (memory locking) if it’s enabled
• Set --max-active-backends=1 to keep only one model in memory

Docker-Specific Problems

Container Fails to Start

Diagnostic steps:

# Check container logs
docker logs local-ai

# Check if port is already in use
ss -tlnp | grep 8080

# Verify the image exists
docker images | grep localai

GPU Not Available Inside Container

NVIDIA:

# Ensure nvidia-container-toolkit is installed, then:
docker run --gpus all ...

AMD:

docker run --device=/dev/kfd --device=/dev/dri --group-add=video ...

Intel:

docker run --device /dev/dri ...

Health Checks Failing

Add a health check to your Docker Compose configuration:

services:
  local-ai:
    image: localai/localai:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      interval: 30s
      timeout: 10s
      retries: 3

Models or Settings Not Persisted Between Upgrades

Container-local files are discarded when an upgrade recreates the container. Mount all of LocalAI’s stateful paths:

services:
  local-ai:
    volumes:
      - ./models:/models
      - ./backends:/backends
      - ./configuration:/configuration
      - ./data:/data

The paths on the left can be any persistent host directories or named volumes. The container paths on the right must match exactly. See Persistent Storage for Docker, Podman, and UnRAID guidance.

Network and P2P Issues

P2P Workers Not Discovered

Symptoms: Distributed inference setup but workers are not found.

Key requirements:

• Use --net host or network_mode: host in Docker
• Share the same P2P token across all nodes

Debug P2P connectivity:

LOCALAI_P2P_LOGLEVEL=debug \
LOCALAI_P2P_LIB_LOGLEVEL=debug \
LOCALAI_P2P_ENABLE_LIMITS=true \
LOCALAI_P2P_TOKEN="<TOKEN>" \
local-ai run

If DHT is causing issues, try disabling it to use local mDNS discovery instead:

LOCALAI_P2P_DISABLE_DHT=true local-ai run

P2P Limitations

• Only a single model is currently supported for distributed inference
• Workers must be detected before inference starts - you cannot add workers mid-inference
• Workers mode supports llama-cpp compatible models only

See the Distributed Inferencing guide for full setup instructions.

Still Having Issues?

If your issue isn’t covered here:

1. Search existing issues: Check the GitHub Issues for similar problems
2. Enable debug logging: Run with DEBUG=true or --log-level=debug and include the logs when reporting
3. Open a new issue: Include your OS, hardware (CPU/GPU), LocalAI version, model being used, full error logs, and steps to reproduce
4. Community help: Join the LocalAI Discord for community support

Build your first agent

LocalAGI is embedded in LocalAI. There is nothing separate to install or run.

The agent platform ships inside the LocalAI binary and container image, and it is enabled by default. If you already have LocalAI running and a model installed, you have everything you need to build an agent. This page walks you from an empty Agents page to an agent that answers a message and uses one tool.

Before you start: install a tool-calling model

An agent is a loop around a chat model, so it needs a model that supports tool (function) calling. This guide uses qwen3-4b, a small CPU-friendly Qwen3 model that supports tool calling. It is the same model used in the Quickstart, so if you followed that page you already have it.

Install it either from the web interface or from the CLI:

• Web interface: open the Models page at http://localhost:8080, search for qwen3-4b, and click Install.
• CLI:

local-ai run qwen3-4b

For other ways to install models (Hugging Face, OCI, local files), see the model gallery.

Give the agent a model

An agent with no model set cannot answer. The agent has nothing to send your message to, so it will fail to respond until you assign it a model. You can set the model in two ways:

• Per agent: choose qwen3-4b in the agent’s Model field when you create or edit it (covered below). This is the usual choice.
• As a default for every new agent: start LocalAI with an environment variable so new agents are created with that model already selected:

LOCALAI_AGENT_POOL_DEFAULT_MODEL=qwen3-4b

Setting a per-agent model always overrides the default.

Create the agent

1. Open the Agents page in the web interface.
2. Click Create Agent.
3. Fill in the form:
   • Name: for example helper.
   • Model: select qwen3-4b.
   • System prompt: a short instruction that sets the agent’s behavior, for example You are a concise, helpful assistant.
   • Action: add one simple action so the agent has a tool to call. A search action is a good first choice. Some actions need credentials (for example an API key); pick one whose requirements you can satisfy, or start with an action that needs none.
4. Save the agent.

Send a message

Open the new agent from the Agents page, type a task in its task box, for example Hello, what can you do?, and press Start run. The run page shows the agent working and then its answer, within a few seconds. When the agent decides to use the action you configured, the step count and the tool in use show live, and the report lists what the tool returned.

That is a complete agent: a model, a system prompt, and one tool, all running inside your LocalAI process.

If your imported agent will not run

If you imported an agent from the Agent Hub or a JSON file and it does not respond, work through this checklist. Each symptom maps to a fix:

• The agent does not answer at all: the model it references is not installed. Open the Models page and install the model named in the agent’s configuration (or change the agent’s model to one you have, such as qwen3-4b).
• An action always fails: the action is missing its API keys or other credentials. Open the agent’s action configuration and supply the required keys.
• A tool times out or is unavailable: the MCP server that provides it is unreachable. Confirm the MCP server is running and that the agent points at the correct address.
• A skill the agent expects is not available: skills are disabled by default. Start LocalAI with LOCALAI_AGENT_POOL_ENABLE_SKILLS=true to turn the skills service on (the default is LOCALAI_AGENT_POOL_ENABLE_SKILLS=false).

Next steps

• Model gallery - install more models, including larger tool-calling models for more capable agents.
• Agent actions catalog - the full list of built-in actions an agent can use and how to configure them.
• Agent-scoped MCP - connect an agent to external Model Context Protocol servers to give it more tools.

Containers

LocalAI supports Docker, Podman, and other OCI-compatible container engines. This guide covers the common aspects of running LocalAI in containers.

Prerequisites

Before you begin, ensure you have a container engine installed:

• Install Docker (Mac, Windows, Linux)
• Install Podman (Linux, macOS, Windows WSL2)

Podman might not resolve short image names such as localai/localai:latest. The Podman and Compose examples use the docker.io/ prefix to identify the container registry explicitly.

Quick Start

The fastest way to get started is with the CPU image:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest
# Or with Podman:
podman run -p 8080:8080 --name local-ai -ti docker.io/localai/localai:latest

This will:

• Start LocalAI (you’ll need to install models separately)
• Make the API available at http://localhost:8080

Image Types

LocalAI provides several image types to suit different needs. These images work with both Docker and Podman.

Standard Images

Standard images don’t include pre-configured models. Use these if you want to configure models manually.

CPU Image

docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 docker.io/localai/localai:latest

GPU Images

Choose the image that matches your hardware and installed drivers:

• NVIDIA CUDA 12 is the compatibility choice for systems with CUDA 12 drivers. Use CUDA 13 when your NVIDIA driver and toolkit support CUDA 13.
• AMD ROCm is for supported AMD GPUs, while Intel is for Intel GPUs with the required device runtime.
• Jetson uses the L4T ARM64 image. Choose its CUDA 12 image for Jetson AGX Orin-class devices or CUDA 13 for DGX Spark.
• Vulkan works across vendors and is the fallback when no matching CUDA, ROCm, or Intel image is available.

NVIDIA CUDA 13:

docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 --device nvidia.com/gpu=all docker.io/localai/localai:latest-gpu-nvidia-cuda-13

NVIDIA CUDA 12:

docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 --device nvidia.com/gpu=all docker.io/localai/localai:latest-gpu-nvidia-cuda-12

AMD GPU (ROCm):

docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblas
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 --device rocm.com/gpu=all docker.io/localai/localai:latest-gpu-hipblas

Intel GPU:

docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-intel
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 --device gpu.intel.com/all docker.io/localai/localai:latest-gpu-intel

Vulkan:

docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 docker.io/localai/localai:latest-gpu-vulkan

NVIDIA Jetson (L4T ARM64):

CUDA 12 (for Nvidia AGX Orin and similar platforms):

docker run -ti --name local-ai -p 8080:8080 --runtime nvidia --gpus all localai/localai:latest-nvidia-l4t-arm64

CUDA 13 (for Nvidia DGX Spark):

docker run -ti --name local-ai -p 8080:8080 --runtime nvidia --gpus all localai/localai:latest-nvidia-l4t-arm64-cuda-13

Using Compose

The repository’s docker-compose.yaml installs phi-2-chat from the model gallery by default. Change its command list to select a different gallery model.

For a more manageable setup, especially with persistent volumes, use Docker Compose or Podman Compose:

Using CDI (Container Device Interface) - Recommended for NVIDIA Container Toolkit 1.14+

The CDI approach is recommended for newer versions of the NVIDIA Container Toolkit (1.14 and later). It provides better compatibility and is the future-proof method:

version: "3.9"
services:
  api:
    image: docker.io/localai/localai:latest-gpu-nvidia-cuda-12
    # For CUDA 13, use: docker.io/localai/localai:latest-gpu-nvidia-cuda-13
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      # start_period, not timeout, is the knob for a slow first boot: startup
      # preload can download tens of GB before the API binds, and failures
      # inside the start period leave the container `starting` rather than
      # marking it unhealthy. timeout is a per-probe deadline.
      start_period: 60m
      interval: 1m
      timeout: 10s
      retries: 3
    ports:
      - 8080:8080
    environment:
      - DEBUG=false
    volumes:
      - ./models:/models:cached
    # CDI driver configuration (recommended for NVIDIA Container Toolkit 1.14+)
    # This uses the nvidia.com/gpu resource API
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia.com/gpu
              count: all
              capabilities: [gpu]

Save this as compose.yaml and run:

docker compose up -d
# Or with Podman:
podman-compose up -d

Using Legacy NVIDIA Driver - For Older NVIDIA Container Toolkit

If you are using an older version of the NVIDIA Container Toolkit (before 1.14), or need backward compatibility, use the legacy approach:

version: "3.9"
services:
  api:
    image: docker.io/localai/localai:latest-gpu-nvidia-cuda-12
    # For CUDA 13, use: docker.io/localai/localai:latest-gpu-nvidia-cuda-13
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      # start_period, not timeout, is the knob for a slow first boot: startup
      # preload can download tens of GB before the API binds, and failures
      # inside the start period leave the container `starting` rather than
      # marking it unhealthy. timeout is a per-probe deadline.
      start_period: 60m
      interval: 1m
      timeout: 10s
      retries: 3
    ports:
      - 8080:8080
    environment:
      - DEBUG=false
    volumes:
      - ./models:/models:cached
    # Legacy NVIDIA driver configuration (for older NVIDIA Container Toolkit)
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Persistent Storage

The container exposes the following volumes:

The host paths can be anywhere on persistent storage, but the container paths must be exactly /models, /backends, /configuration, and /data. In UnRAID and other container-template UIs, create one path mapping for each row in the table above.

Backend OCI images contain symbolic links. When /backends is stored on a filesystem that cannot create links, such as some CIFS/SMB mounts, LocalAI materializes each link as a regular file so installation can complete. This can use more disk space than a local filesystem. Prefer a Docker or Podman named volume for /backends when possible.

To use bind mounts:

docker run -ti --name local-ai -p 8080:8080 \
  -v $PWD/models:/models \
  -v $PWD/backends:/backends \
  -v $PWD/configuration:/configuration \
  -v $PWD/data:/data \
  localai/localai:latest
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 \
  -v $PWD/models:/models \
  -v $PWD/backends:/backends \
  -v $PWD/configuration:/configuration \
  -v $PWD/data:/data \
  docker.io/localai/localai:latest

Or use named volumes:

docker volume create localai-models
docker volume create localai-backends
docker volume create localai-configuration
docker volume create localai-data
docker run -ti --name local-ai -p 8080:8080 \
  -v localai-models:/models \
  -v localai-backends:/backends \
  -v localai-configuration:/configuration \
  -v localai-data:/data \
  localai/localai:latest
# Or with Podman:
podman volume create localai-models
podman volume create localai-backends
podman volume create localai-configuration
podman volume create localai-data
podman run -ti --name local-ai -p 8080:8080 \
  -v localai-models:/models \
  -v localai-backends:/backends \
  -v localai-configuration:/configuration \
  -v localai-data:/data \
  docker.io/localai/localai:latest

Next Steps

After installation:

1. Access the WebUI at http://localhost:8080
2. Check available models: curl http://localhost:8080/v1/models
3. Install additional models
4. Try out examples

Troubleshooting

Container won’t start

• Check container engine is running: docker ps or podman ps
• Check port 8080 is available: netstat -an | grep 8080 (Linux/Mac)
• View logs: docker logs local-ai or podman logs local-ai

GPU not detected

• Ensure Docker has GPU access: docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi
• For Podman, pass the GPU with the --device flags shown in the GPU sections above (for example --device nvidia.com/gpu=all)
• For NVIDIA: Install NVIDIA Container Toolkit
• For AMD: Ensure devices are accessible: ls -la /dev/kfd /dev/dri

NVIDIA Container fails to start with “Auto-detected mode as ’legacy’” error

If you encounter this error:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: invalid expression

This indicates a Docker/NVIDIA Container Toolkit configuration issue. The container runtime’s prestart hook fails before LocalAI starts. This is not a LocalAI code bug.

Solutions:

1. Use CDI mode (recommended): Update your docker-compose.yaml to use the CDI driver configuration:

   deploy:
     resources:
       reservations:
         devices:
           - driver: nvidia.com/gpu
             count: all
             capabilities: [gpu]

2. Upgrade NVIDIA Container Toolkit: Ensure you have version 1.14 or later, which has better CDI support.

3. Check NVIDIA Container Toolkit configuration: Run nvidia-container-cli --query-gpu to verify your installation is working correctly outside of containers.

4. Verify Docker GPU access: Test with docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

Models not downloading

• Check internet connection
• Verify disk space: df -h
• Check container logs for errors: docker logs local-ai or podman logs local-ai
• If the logs show status 401 or 403, the source needs authentication. See Private Registries and Galleries

Full image reference

The quick-start examples above use the Docker Hub image names. Every image is published to both Docker Hub and Quay. The tables below map the Docker Hub tag to its Quay equivalent for each variant. Replace v4.11.0 with a released version to pin a specific build.

See Also

• Full image reference - Complete Quay and Docker Hub image matrix
• Install Models - Install and configure models
• GPU Acceleration - GPU setup and optimization
• Kubernetes Installation - Deploy on Kubernetes

Linux Installation

Desktop launcher AppImage (x86_64)

Download LocalAI-Launcher-x86_64.AppImage and its .sha256 file from GitHub Releases. In that directory:

sha256sum --check LocalAI-Launcher-x86_64.AppImage.sha256
chmod +x LocalAI-Launcher-x86_64.AppImage
./LocalAI-Launcher-x86_64.AppImage

You can also double-click the executable image in a file manager. Normal AppImage mounting requires FUSE support (on Ubuntu 22.04, libfuse2). If FUSE is unavailable, extract it into a writable directory and run the launcher:

./LocalAI-Launcher-x86_64.AppImage --appimage-extract
./squashfs-root/AppRun

This is the existing desktop/tray launcher, not the server bundled into an image. First use needs internet access to download LocalAI. Models and backends are downloaded separately. The launcher stores configuration in ~/.localai/launcher.json, runtime downloads in ~/.localai/bin, and default models/backends under ~/.localai/. Fyne preferences use the user configuration directory. These remain writable outside the read-only image. Keep your home writable; replacing or deleting the AppImage does not delete your data.

The supported release target is x86_64 Ubuntu 22.04 or newer, with glibc 2.35+, OpenGL and X11 desktop libraries (XWayland on Wayland). A tray-enabled desktop is recommended. This is not a static executable: other distributions must provide compatible graphics libraries. ARM64 AppImages are not currently published; use the server binaries below instead. Server/backend requirements are separate from launcher requirements. Replace the AppImage manually to update the launcher; there is no AppImage self-updater.

For build dependencies, pinned tools and smoke checks, see the AppImage build guide.

Manual Installation

Download Binary

You can manually download the appropriate binary for your system from the releases page:

1. Go to GitHub Releases
2. Download the binary for your architecture (amd64, arm64, etc.)
3. Make it executable:

chmod +x local-ai-*

4. Run LocalAI:

./local-ai-*

Run your first model

Starting the binary on its own gives you an empty server. To get a working chat right away, run LocalAI with a model name and it will download and serve it from the gallery:

./local-ai-* run qwen3-4b

Once it is ready, open the WebUI at http://localhost:8080 or send a request to the API:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "qwen3-4b",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

System Requirements

Hardware requirements vary based on:

• Model size
• Quantization method
• Backend used

For performance benchmarks with different backends like llama.cpp, visit this link.

Configuration

After installation, you can:

• Access the WebUI at http://localhost:8080
• Configure models in the models directory
• Customize settings via environment variables or config files

Start LocalAI on demand with systemd

LocalAI accepts a single TCP listener passed through the systemd socket activation protocol. This lets systemd listen on the public port and start LocalAI only when the first client connects.

Create /etc/systemd/system/local-ai.socket:

[Unit]
Description=LocalAI API socket

[Socket]
ListenStream=8080
NoDelay=true

[Install]
WantedBy=sockets.target

Create the matching /etc/systemd/system/local-ai.service:

[Unit]
Description=LocalAI

[Service]
Type=simple
User=localai
Group=localai
ExecStart=/usr/local/bin/local-ai run
WorkingDirectory=/var/lib/local-ai

Adjust the user, binary path, working directory, and model configuration for your installation. Then enable the socket, not the service:

sudo systemctl daemon-reload
sudo systemctl enable --now local-ai.socket

The first connection to port 8080 starts local-ai.service; systemd holds that connection until LocalAI is ready to accept it. LOCALAI_ADDRESS and --address are ignored while an inherited listener is present. LocalAI rejects activation with multiple stream listeners so it cannot silently choose the wrong endpoint.

For a Podman-managed container, configure Podman to preserve and pass the systemd socket file descriptor into the container. The LocalAI process inside the container consumes the same activation protocol.

Activation needs both LISTEN_PID and LISTEN_FDS. If only one of them is set, LocalAI ignores them and binds --address as usual. A container engine started from a socket-activated system unit can leak a bare LISTEN_PID into every container it spawns, and that is not an activation attempt.

Next Steps

• Try it out with examples
• Learn about available models
• Configure GPU acceleration
• Customize your configuration

macOS Installation

The easiest way to install LocalAI on macOS is using the DMG application.

Download

Download the latest DMG from GitHub releases:

Installation Steps

1. Download the LocalAI.dmg file from the link above
2. Open the downloaded DMG file
3. Drag the LocalAI application to your Applications folder
4. Launch LocalAI from your Applications folder

First Launch

The app you installed is a small launcher that manages the LocalAI server for you. On the first launch it offers to download and install the latest server release; once that finishes, the server starts automatically and on every following launch of the app.

The launcher lives in the menu bar (look for the LocalAI icon in the top right of your screen) and does not open a window of its own. From the menu bar icon you can start and stop the server, open the WebUI, check for updates, and change settings, including turning off the automatic server start (“Start LocalAI when the launcher opens” under Settings).

Once the server is running, the WebUI is available at http://localhost:8080.

Verification

The LocalAI.dmg (and the app inside it) and the local-ai server binary are signed with an Apple Developer ID and notarized by Apple, so they launch with no quarantine prompt or workaround. To inspect the signature yourself:

spctl --assess --type open --context context:primary-signature -v /Applications/LocalAI.app
codesign --verify --deep --strict --verbose=2 /Applications/LocalAI.app

Next Steps

After installing LocalAI, you can:

• Access the WebUI at http://localhost:8080
• Try it out with examples
• Learn about available models
• Customize your configuration

Run with Kubernetes

For installing LocalAI in Kubernetes, the deployment file from the examples can be used and customized as preferred:

kubectl apply -f https://raw.githubusercontent.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment.yaml

For Nvidia GPUs:

kubectl apply -f https://raw.githubusercontent.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment-nvidia.yaml

Alternatively, the helm chart can be used as well:

helm repo add go-skynet https://go-skynet.github.io/helm-charts/
helm repo update
helm show values go-skynet/local-ai > values.yaml


helm install local-ai go-skynet/local-ai -f values.yaml

Build LocalAI

Build

LocalAI can be built as a container image or as a single, portable binary. Note that some model architectures might require Python libraries, which are not included in the binary.

LocalAI’s extensible architecture allows you to add your own backends, which can be written in any language, and as such the container images contains also the Python dependencies to run all the available backends (for example, in order to run backends like Diffusers that allows to generate images and videos from text).

This section contains instructions on how to build LocalAI from source.

Build LocalAI locally

Requirements

In order to build LocalAI locally, you need the following requirements:

• Golang >= 1.21
• GCC
• GRPC

To install the dependencies follow the instructions below:

brew install go protobuf protoc-gen-go protoc-gen-go-grpc wget

apt install golang make protobuf-compiler-grpc

go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af

make build

Build

To build LocalAI with make:

git clone https://github.com/go-skynet/LocalAI
cd LocalAI
make build

This should produce the binary local-ai

Container image

Requirements:

• Docker or podman, or a container engine

In order to build the LocalAI container image locally you can use docker, for example:

docker build -t localai .
docker run localai

Example: Build on mac

Building on Mac (M1, M2 or M3) works, but you may need to install some prerequisites using brew.

The below has been tested by one mac user and found to work. Note that this doesn’t use Docker to run the server:

Install xcode from the Apps Store (needed for metalkit)

brew install abseil cmake go grpc protobuf wget protoc-gen-go protoc-gen-go-grpc

git clone https://github.com/go-skynet/LocalAI.git

cd LocalAI

make build

wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q2_K.gguf -O models/phi-2.Q2_K

cp -rf prompt-templates/ggml-gpt4all-j.tmpl models/phi-2.Q2_K.tmpl

./local-ai backends install llama-cpp

./local-ai --models-path=./models/ --debug=true

curl http://localhost:8080/v1/models

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "phi-2.Q2_K",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9 
   }'

Troubleshooting mac

• If you encounter errors regarding a missing utility metal, install Xcode from the App Store.

• After the installation of Xcode, if you receive a xcrun error 'xcrun: error: unable to find utility "metal", not a developer tool or in PATH'. You might have installed the Xcode command line tools before installing Xcode, the former one is pointing to an incomplete SDK.

xcode-select --print-path

sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer

• If completions are slow, ensure that gpu-layers in your model yaml matches the number of layers from the model in use (or simply use a high number such as 256).

• If you get a compile error: error: only virtual member functions can be marked 'final', reinstall all the necessary brew packages, clean the build, and try again.

brew reinstall go grpc protobuf wget

make clean

make build

Build backends

LocalAI have several backends available for installation in the backend gallery. The backends can be also built by source. As backends might vary from language and dependencies that they require, the documentation will provide generic guidance for few of the backends, which can be applied with some slight modifications also to the others.

Manually

Typically each backend include a Makefile which allow to package the backend.

In the LocalAI repository, for instance you can build a backend by doing:

git clone https://github.com/go-skynet/LocalAI.git

make -C LocalAI/backend/python/vllm

With Docker

Building with docker is simpler as abstracts away all the requirement, and focuses on building the final OCI images that are available in the gallery. This allows for instance also to build locally a backend and install it with LocalAI. You can refer to Backends for general guidance on how to install and develop backends.

In the LocalAI repository, you can build a backend by doing:

git clone https://github.com/go-skynet/LocalAI.git

make docker-build-<backend-name>

Note that make is only by convenience, in reality it just runs a simple docker command as:

docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:<backend-name> -f LocalAI/backend/Dockerfile.golang --build-arg BACKEND=<backend-name> .               

Note:

• BUILD_TYPE can be either: cublas, hipblas, sycl_f16, sycl_f32, metal.
• BASE_IMAGE is tested on ubuntu:24.04 (and defaults to it) and quay.io/go-skynet/intel-oneapi-base:latest for intel/sycl

Docker Installation

See Containers for the complete guide to running LocalAI with Docker and Podman.

Text Generation (GPT)

LocalAI supports generating text with GPT with llama.cpp and other backends (such as rwkv.cpp as ) see also the Model compatibility for an up-to-date list of the supported model families.

Note:

• You can also specify the model name as part of the OpenAI token.
• If only one model is available, the API will use it for all the requests.

API Reference

Chat completions

https://platform.openai.com/docs/api-reference/chat

For example, to generate a chat completion, you can send a POST request to the /v1/chat/completions endpoint with the instruction as the request body:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "messages": [{"role": "user", "content": "Say this is a test!"}],
  "temperature": 0.7
}'

Available additional parameters: top_p, top_k, max_tokens

Reasoning models return their thinking in the reasoning field. When a model reasons and calls a tool in the same turn, see Interleaved Thinking with Tool Calls.

When stream: true is set and the llama.cpp backend fails before the first chunk, for example because the prompt exceeds the context size, the request fails with an HTTP error. The error message is not streamed as assistant content. An error after streaming has started is reported inside the stream.

Edit completions

https://platform.openai.com/docs/api-reference/edits

To generate an edit completion you can send a POST request to the /v1/edits endpoint with the instruction as the request body:

curl http://localhost:8080/v1/edits -H "Content-Type: application/json" -d '{
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "instruction": "rephrase",
  "input": "Black cat jumped out of the window",
  "temperature": 0.7
}'

Available additional parameters: top_p, top_k, max_tokens.

Completions

https://platform.openai.com/docs/api-reference/completions

To generate a completion, you can send a POST request to the /v1/completions endpoint with the instruction as per the request body:

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "prompt": "A long time ago in a galaxy far, far away",
  "temperature": 0.7
}'

Available additional parameters: top_p, top_k, max_tokens

List models

You can list all the models available with:

curl http://localhost:8080/v1/models

Anthropic Messages API

LocalAI supports the Anthropic Messages API, which is compatible with Claude clients. This endpoint provides a structured way to send messages and receive responses, with support for tools, streaming, and multimodal content.

Endpoint: POST /v1/messages or POST /messages

Reference: https://docs.anthropic.com/claude/reference/messages_post

Basic Usage

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "max_tokens": 1024,
    "messages": [
      {"role": "user", "content": "Say this is a test!"}
    ]
  }'

Request Parameters

Message Format

Messages can contain text or structured content blocks:

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "max_tokens": 1024,
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What is in this image?"
          },
          {
            "type": "image",
            "source": {
              "type": "base64",
              "media_type": "image/jpeg",
              "data": "base64_encoded_image_data"
            }
          }
        ]
      }
    ]
  }'

Tool Calling

The Anthropic API supports function calling through tools:

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "max_tokens": 1024,
    "tools": [
      {
        "name": "get_weather",
        "description": "Get the current weather",
        "input_schema": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state"
            }
          },
          "required": ["location"]
        }
      }
    ],
    "tool_choice": "auto",
    "messages": [
      {"role": "user", "content": "What is the weather in San Francisco?"}
    ]
  }'

Streaming

Enable streaming responses by setting stream: true:

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "max_tokens": 1024,
    "stream": true,
    "messages": [
      {"role": "user", "content": "Tell me a story"}
    ]
  }'

Streaming responses use Server-Sent Events (SSE) format with event types: message_start, content_block_start, content_block_delta, content_block_stop, message_delta, and message_stop.

Response Format

{
  "id": "msg_abc123",
  "type": "message",
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "This is a test!"
    }
  ],
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "stop_reason": "end_turn",
  "usage": {
    "input_tokens": 10,
    "output_tokens": 5
  }
}

Open Responses API

LocalAI supports the Open Responses API specification, which provides a standardized interface for AI model interactions with support for background processing, streaming, tool calling, and advanced features like reasoning.

Endpoint: POST /v1/responses or POST /responses

Reference: https://www.openresponses.org/specification

Basic Usage

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": "Say this is a test!",
    "max_output_tokens": 1024
  }'

WebSocket Responses

Connect to ws://localhost:8080/v1/responses (or wss:// when TLS is enabled) and send response.create messages over the WebSocket. Only one response may be in progress on a connection. Wait for response.completed or response.failed for the active response before sending the next response.create. An error that rejects an invalid or additional response.create applies only to that rejected message; it does not terminate a response that is already in progress.

Set the WebSocket-only generate field to false to prepare a request without running inference:

{
  "type": "response.create",
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "generate": false,
  "store": false,
  "input": "Say this is a test!"
}

LocalAI emits response.created followed by response.completed with no generated output. The completed response still has an ID that can continue the prepared request:

{
  "type": "response.create",
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "store": false,
  "previous_response_id": "resp_abc123",
  "input": []
}

Response IDs created with store: false are available only on the WebSocket connection that created them and are removed when that connection closes. A previous_response_id chain can contain multiple responses; LocalAI replays the complete conversation from the oldest response through the referenced response before appending the new input. The replay includes server-side MCP tool calls and their results, which are stored as function_call and function_call_output output items. Keep store: false for every descendant of a connection-local response. LocalAI rejects a store: true response that depends on connection-local history because the resulting stored chain would be broken after the WebSocket closes.

After delivery, LocalAI discards buffered stream events for store: false responses but retains their request and final response for continuation. A connection admits at most 128 local responses and accepts a new local request only while its current serialized history plus that request is at most 64 MiB. When either admission threshold is reached, LocalAI returns connection_store_limit_reached; reconnect to start a new local history. With authentication enabled, globally stored response IDs can be continued only by the identity that created them.

Request Parameters

Input Format

Input can be a simple string or an array of structured items:

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": [
      {
        "type": "message",
        "role": "user",
        "content": "What is the weather?"
      }
    ],
    "max_output_tokens": 1024
  }'

Streaming responses

Set "stream": true to receive Server-Sent Events. Each response.output_item.added event assigns an output_index to an item. Use that index and the item ID to associate later deltas and completion events with the same item.

If a request without explicit tools produces reasoning, the stream uses separate items for reasoning and answer text. Each item keeps its original index throughout the stream. The response.completed event includes both items in the same index order, followed by any automatically parsed tool calls.

Background Processing

Run requests in the background for long-running tasks:

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": "Generate a long story",
    "max_output_tokens": 4096,
    "background": true
  }'

The response will include a response ID that can be used to poll for completion:

{
  "id": "resp_abc123",
  "object": "response",
  "status": "in_progress",
  "created_at": 1234567890
}

Retrieving Background Responses

Use the GET endpoint to retrieve background responses:

# Get response by ID
curl http://localhost:8080/v1/responses/resp_abc123

# Resume streaming with query parameters
curl "http://localhost:8080/v1/responses/resp_abc123?stream=true&starting_after=10"

Canceling Background Responses

Cancel a background response that’s still in progress:

curl -X POST http://localhost:8080/v1/responses/resp_abc123/cancel

Multiple Replicas (Distributed Mode)

In distributed mode LocalAI replicates response metadata across frontend replicas, so retrieval, previous_response_id chaining and cancellation work regardless of which replica the load balancer picks:

• GET /v1/responses/{id} returns the response from any replica.
• POST /v1/responses/{id}/cancel is delegated over NATS to the replica that is actually generating, so generation really stops. If that replica is gone, the response is reported as cancelled without blocking.
• Streaming resume (?stream=true) is served only by the replica that created the response. The event buffer lives in that process’s memory and is not replicated. A resume request that reaches another replica returns HTTP 409 naming the owning replica instead of silently returning a truncated stream. Poll the response instead, or route resume requests with session affinity.

Tool Calling

Open Responses API supports function calling with tools:

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": "What is the weather in San Francisco?",
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "description": "Get the current weather",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state"
            }
          },
          "required": ["location"]
        }
      }
    ],
    "tool_choice": "auto",
    "max_output_tokens": 1024
  }'

For streaming requests with JSON tool output, LocalAI waits for the complete JSON object before emitting a completed function_call item. Arguments can span multiple tokens. Read the arguments from the response.output_item.done event before executing the tool.

Reasoning Configuration

Configure reasoning effort and summary style:

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": "Solve this complex problem step by step",
    "reasoning": {
      "effort": "high",
      "summary": "detailed"
    },
    "max_output_tokens": 2048
  }'

Response Format

{
  "id": "resp_abc123",
  "object": "response",
  "created_at": 1234567890,
  "completed_at": 1234567895,
  "status": "completed",
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "output": [
    {
      "type": "message",
      "id": "msg_001",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "This is a test!",
          "annotations": [],
          "logprobs": []
        }
      ],
      "status": "completed"
    }
  ],
  "error": null,
  "incomplete_details": null,
  "temperature": 0.7,
  "top_p": 1.0,
  "presence_penalty": 0.0,
  "frequency_penalty": 0.0,
  "usage": {
    "input_tokens": 10,
    "output_tokens": 5,
    "total_tokens": 15,
    "input_tokens_details": {
      "cached_tokens": 0
    },
    "output_tokens_details": {
      "reasoning_tokens": 0
    }
  }
}

Backends

RWKV

RWKV support is available through llama.cpp (see below)

llama.cpp

llama.cpp is a popular port of Facebook’s LLaMA model in C/C++.

Features

The llama.cpp model supports the following features:

• 📖 Text generation (GPT)
• 🧠 Embeddings
• 🔥 OpenAI functions
• ✍️ Constrained grammars

Setup

LocalAI supports llama.cpp models out of the box. You can use the llama.cpp model in the same way as any other model.

Manual setup

It is sufficient to copy the ggml or gguf model files in the models folder. You can refer to the model in the model parameter in the API calls.

You can optionally create an associated YAML model config file to tune the model’s parameters or apply a template to the prompt.

Prompt templates are useful for models that are fine-tuned towards a specific prompt.

With the llama.cpp backend, streaming and non-streaming chat requests preserve the template’s grammar, stop strings, and message boundaries.

Automatic setup

LocalAI supports model galleries which are indexes of models. For instance, the huggingface gallery contains a large curated index of models from the huggingface model hub for ggml or gguf models.

For instance, if you have the galleries enabled and LocalAI already running, you can just start chatting with models in huggingface by running:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "TheBloke/WizardLM-13B-V1.2-GGML/wizardlm-13b-v1.2.ggmlv3.q2_K.bin",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.1
   }'

LocalAI will automatically download and configure the model in the model directory.

Models can be also preloaded or downloaded on demand. To learn about model galleries, check out the model gallery documentation.

YAML configuration

To use the llama.cpp backend, specify llama-cpp as the backend in the YAML file:

name: llama
backend: llama-cpp
parameters:
  # Relative to the models path
  model: file.gguf

Backend Options

The llama.cpp backend supports additional configuration options that can be specified in the options field of your model YAML configuration. These options allow fine-tuning of the backend behavior:

Example configuration with options:

name: llama-model
backend: llama
parameters:
  model: model.gguf
options:
  - use_jinja:true
  - context_shift:true
  - cache_ram:4096
  - parallel:2
  - devices:CUDA1,CUDA2,CUDA3
  - fit_params:true
  - fit_target:1024
  - slot_prompt_similarity:0.5

Note: The parallel option can also be set via the LLAMACPP_PARALLEL environment variable, and grpc_servers can be set via the LLAMACPP_GRPC_SERVERS environment variable. Options specified in the YAML file take precedence over environment variables.

An explicit parallel: 1 (or n_parallel: 1) in the model options takes precedence over LLAMACPP_PARALLEL, like any other value. The environment variable is only used when neither option is set; if it is missing or not a number, the backend uses one slot.

Hardware auto-tuning (and how to override it)

On a detected GPU, LocalAI fills a few performance-relevant defaults the model config leaves unset - a larger physical batch on NVIDIA Blackwell, and a VRAM-scaled parallel slot count for concurrent serving. Both are gated on per-device VRAM at the model’s context: when a large context already fills a single card (e.g. a 27B model with a 200k context across 2×16 GiB), the batch boost and the extra parallel slots are suppressed so they can’t tip the tighter GPU into CUDA out-of-memory.

Anything you set explicitly in the model YAML always wins, so to pin a value just set it (e.g. batch: 512 or options: ["parallel:1"]). The effective values are logged at INFO when a model loads (effective runtime tuning …). To turn the hardware auto-tuning off entirely and run llama.cpp’s stock behavior, set:

LOCALAI_DISABLE_HARDWARE_DEFAULTS=true

Server-side prompt cache (repeated system prompts)

Agents, coding assistants, and Anthropic/OpenAI-compatible CLIs typically resend the same large system prompt on every turn. The llama.cpp server can short-circuit prefill for the matching prefix by stashing idle slot KV states in host RAM and reloading them on a hit. Three settings interact:

All three are on by default since LocalAI v4.3, so the prompt cache works out of the box for the common single-slot setup. If you’re on an older release, or you’ve explicitly disabled one of them, add the following to recover the behaviour:

options:
  - cache_ram:4096       # or -1 for no limit
  - kv_unified:true
  - cache_idle_slots:true

Set cache_ram:0 to opt out of the prompt cache entirely (saves host RAM at the cost of re-prefilling repeated prompts).

cache_ram:-1 removes the limit. With idle-slot saving on, every distinct prompt then leaves its slot state in host RAM, so a workload with many different prompts (classification, ingestion) grows the backend by roughly the KV size of each prompt until the host runs out of memory.

Saving and restoring a prompt cache

The server-side prompt cache lives in RAM and is lost when the model unloads. To keep the prompt state of one slot on disk, set slot_save_path and use the slot API. A typical use is a voice assistant with a long system prompt: save the state once, then restore it when the wake word fires, so the first request does not wait for prefill.

options:
  - slot_save_path:/var/lib/localai/slots

Both calls need an admin API key when authentication is on. Without slot_save_path they return 400.

The server chooses the name of every saved file. A save returns an id, and a restore takes that id. A client cannot pick a file name or a path.

# Save slot 0 of the model. The reply contains the id.
curl http://localhost:8080/v1/slots/save \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model", "slot": 0}'

# Load that state back into slot 0
curl http://localhost:8080/v1/slots/restore \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model", "slot": 0, "id": "0123456789abcdef0123456789abcdef"}'

The save reply gives the id, the slot, the number of tokens and bytes, and the time taken in duration_ms. The message field lists the IDs of any older saves that were deleted to make room. The restore reply has the same fields. Neither reply contains a path on the host.

Keep these points in mind:

• A client cannot choose names. The backend creates a directory for each model under slot_save_path (mode 0700) and writes a manifest.json in it. A restore works only for an id in the manifest of that model. Any other id returns 404 with no saved state with this id for this model.
• The backend keeps at most 32 saves for each model. When a save would go over that, it deletes the oldest one and lists its id in message. A save larger than 8 GiB is deleted and fails.
• A restore also checks that the file is a plain file and has the size that was recorded at save time. A file that was replaced or cut short is refused.
• Keep slot_save_path private to the user that runs LocalAI. LocalAI creates it with mode 0700 when it is missing, and writes a warning to the log when an existing directory is open to other users. The checks above cannot stop a local user who can write to the directory.
• A saved state is only as fresh as its save. To change the cached prompt, save again and use the new id.
• A saved state holds the prompt state for the model and quantisation it came from. Do not restore it into a different model, quantisation or context size. The restore call fails if the state does not fit, but a state from another model is not always detected.
• A slot that is busy with a request is served after that request ends.
• The API is for llama-cpp only. Other backends return 501.

Reference

• llama

ik_llama.cpp

ik_llama.cpp is a hard fork of llama.cpp by Iwan Kawrakow that focuses on superior CPU and hybrid GPU/CPU performance. It ships additional quantization types (IQK quants), custom quantization mixes, Multi-head Latent Attention (MLA) for DeepSeek models, and fine-grained tensor offload controls - particularly useful for running very large models on commodity CPU hardware.

Features

The ik-llama-cpp backend supports the following features:

• 📖 Text generation (GPT)
• 🧠 Embeddings
• IQK quantization types for better CPU inference performance
• Multimodal models (via clip/llava)

Setup

The backend is distributed as a separate container image and can be installed from the LocalAI backend gallery, or specified directly in a model configuration. GGUF models loaded with this backend benefit from ik_llama.cpp’s optimized CPU kernels - especially useful for MoE models and large quantized models that would otherwise be GPU-bound.

YAML configuration

To use the ik-llama-cpp backend, specify it as the backend in the YAML file:

name: my-model
backend: ik-llama-cpp
parameters:
  # Relative to the models path
  model: file.gguf

The aliases ik-llama and ik_llama are also accepted.

Reference

• ik_llama.cpp

turboquant (llama.cpp fork with TurboQuant KV-cache)

llama-cpp-turboquant is a llama.cpp fork that adds the TurboQuant KV-cache quantization scheme. It reuses the upstream llama.cpp codebase and ships as a drop-in alternative backend inside LocalAI, sharing the same gRPC server sources as the stock llama-cpp backend - so any GGUF model that runs on llama-cpp also runs on turboquant.

You would pick turboquant when you want smaller KV-cache memory pressure (longer contexts on the same VRAM) or to experiment with the fork’s quantized KV representations on top of the standard cache_type_k / cache_type_v knobs already supported by upstream llama.cpp.

Features

• Drop-in GGUF compatibility with upstream llama.cpp.
• TurboQuant KV-cache quantization (see fork README for the current set of accepted cache_type_k / cache_type_v values).
• Same feature surface as the llama-cpp backend: text generation, embeddings, tool calls, multimodal via mmproj.
• Available on CPU (AVX/AVX2/AVX512/fallback), NVIDIA CUDA 12/13, AMD ROCm/HIP, Intel SYCL f32/f16, Vulkan, and NVIDIA L4T.

Setup

turboquant ships as a separate container image in the LocalAI backend gallery. Install it like any other backend:

local-ai backends install turboquant

Or pick a specific flavor for your hardware (example tags: cpu-turboquant, cuda12-turboquant, cuda13-turboquant, rocm-turboquant, intel-sycl-f16-turboquant, vulkan-turboquant).

YAML configuration

To run a model with turboquant, set the backend in your model YAML and optionally pick quantized KV-cache types:

name: my-model
backend: turboquant
parameters:
  # Relative to the models path
  model: file.gguf
# Use TurboQuant's own KV-cache quantization schemes. The fork accepts
# the standard llama.cpp types (f16, f32, q8_0, q4_0, q4_1, q5_0, q5_1)
# and adds three TurboQuant-specific ones: turbo2, turbo3, turbo4.
# turbo3 / turbo4 auto-enable flash_attention (required for turbo K/V)
# and offer progressively more aggressive compression.
cache_type_k: turbo3
cache_type_v: turbo3
context_size: 8192

The cache_type_k / cache_type_v fields map to llama.cpp’s -ctk / -ctv flags. The stock llama-cpp backend only accepts the standard llama.cpp types - to use turbo2 / turbo3 / turbo4 you need this turboquant backend, which is where the fork’s TurboQuant code paths actually take effect. Pick q8_0 here and you’re just running stock llama.cpp KV quantization; pick turbo* and you’re running TurboQuant.

Reference

• llama-cpp-turboquant
• Tracked branch: feature/turboquant-kv-cache

vLLM

vLLM is a fast and easy-to-use library for LLM inference.

LocalAI has a built-in integration with vLLM, and it can be used to run models. You can check out vllm performance here.

Setup

Create a YAML file for the model you want to use with vllm.

To setup a model, you need to just specify the model name in the YAML config file:

name: vllm
backend: vllm
parameters:
    model: "facebook/opt-125m"

The backend will automatically download the required files in order to run the model.

Usage

Use the completions endpoint by specifying the vllm backend:

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{   
   "model": "vllm",
   "prompt": "Hello, my name is",
   "temperature": 0.1, "top_p": 0.1
 }'

Passing arbitrary vLLM options with engine_args

A subset of AsyncEngineArgs is exposed as typed YAML fields (tensor_parallel_size, gpu_memory_utilization, quantization, max_model_len, dtype, trust_remote_code, enforce_eager, …). Anything else can be passed through the generic engine_args: map. Keys are forwarded verbatim to vLLM’s engine; unknown keys fail at load time with the closest valid name as a hint. Nested maps materialise into vLLM’s nested config dataclasses (SpeculativeConfig, KVTransferConfig, CompilationConfig, …).

Speculative decoding (DFlash, ngram, eagle, deepseek_mtp, …) is configured this way:

name: qwen3.5-4b-dflash
backend: vllm
parameters:
  model: Qwen/Qwen3.5-4B
context_size: 8192
max_model_len: 8192
trust_remote_code: true
quantization: fp8
template:
  use_tokenizer_template: true
engine_args:
  speculative_config:
    method: dflash
    model: z-lab/Qwen3.5-4B-DFlash
    num_speculative_tokens: 15

The shape of speculative_config follows vLLM’s SpeculativeConfig

• method picks the algorithm, the remaining keys are method-specific. Drafters from z-lab are paired with specific target models; pick the one that matches your target. The drafter loads in its native precision regardless of the target’s quantization: setting.

Another example - picking a non-default attention backend (e.g. on hardware where the default cutlass kernels aren’t supported):

engine_args:
  attention_backend: TRITON_ATTN

CLI-style engine flags in options

Engine arguments can also be written in options: with vLLM’s own CLI spelling, which is handy when copying a command line out of vLLM’s documentation:

options:
  - --quantization:gptq_marlin
  - --enable-prefix-caching
  - --kv-cache-dtype:fp8_e5m2
  - --reasoning-parser:qwen3

Rules:

• Only -- prefixed entries are treated as engine flags; everything else in options: (tool_parser:, reasoning_parser:, …) keeps its existing meaning. Both --flag:value and --flag=value are accepted.
• Dashes become underscores (--enable-prefix-caching → enable_prefix_caching) and the value is coerced to the target field’s type. A bare --flag sets a boolean field to true.
• Unknown or uncoercible flags are logged on the backend’s stderr and skipped rather than failing the load. engine_args: stays strict and is applied last, so it wins on conflict.

engine_args: remains the form to use for anything structured (nested configs, maps); options: is a convenience for flat flags.

Multi-node data parallelism

engine_args.data_parallel_size > 1 combined with the local-ai p2p-worker vllm follower lets a single model span multiple GPU nodes. See vLLM Multi-Node (Data-Parallel) for the head/follower configuration and a worked Kimi-K2.6 example.

SGLang

SGLang is a fast serving framework for LLMs and VLMs with a focus on prefix caching, speculative decoding, and multi-modal generation. LocalAI ships a gRPC backend that wraps SGLang’s async Engine, including its native function-call and reasoning parsers.

Setup

name: sglang
backend: sglang
parameters:
  model: "Qwen/Qwen3-4B"
template:
  use_tokenizer_template: true

The backend will pull the model from HuggingFace on first load.

Passing arbitrary SGLang options with engine_args

The same engine_args: map that the vLLM backend accepts is also honoured by the SGLang backend. Keys are validated against ServerArgs

• SGLang’s central configuration dataclass - and forwarded verbatim to Engine(**kwargs). Unknown keys fail at load time with the closest valid name as a hint. Unlike vLLM, ServerArgs is flat: speculative decoding fields are top-level (speculative_algorithm, speculative_draft_model_path, etc.) rather than nested under a speculative_config: dict.

The typed YAML fields shared with vLLM are mapped to their SGLang equivalents (gpu_memory_utilization → mem_fraction_static, enforce_eager → disable_cuda_graph, tensor_parallel_size → tp_size, max_model_len → context_length). Anything else, including all speculative-decoding flags, goes under engine_args:.

Speculative decoding: Gemma 4 with Multi-Token Prediction

Google publishes paired “assistant” drafters for every Gemma 4 size. The drafters use Multi-Token Prediction (MTP) to propose several candidate tokens per target step, which SGLang then verifies in parallel. Flags below are transcribed verbatim from the SGLang Gemma 4 cookbook.

For consumer GPUs in the 16-24 GB range, use E4B (8 B total / 4 B effective parameters):

name: gemma-4-e4b-mtp
backend: sglang
parameters:
  model: google/gemma-4-E4B-it
context_size: 4096
template:
  use_tokenizer_template: true
options:
  - tool_parser:gemma4
  - reasoning_parser:gemma4
engine_args:
  mem_fraction_static: 0.85
  speculative_algorithm: NEXTN
  speculative_draft_model_path: google/gemma-4-E4B-it-assistant
  speculative_num_steps: 5
  speculative_num_draft_tokens: 6
  speculative_eagle_topk: 1

For smaller cards (8-12 GB), drop to E2B (5 B total / 2 B effective) by swapping the model paths to google/gemma-4-E2B-it and google/gemma-4-E2B-it-assistant; the rest of the flags stay the same.

NEXTN is normalised to EAGLE inside ServerArgs.__post_init__, so either value works - the cookbook uses NEXTN. mem_fraction_static is the share of GPU memory SGLang reserves for the model + KV pool; 0.85 is the cookbook’s default and adapts to whatever single GPU the backend is running on.

The 31 B dense and 26 B-A4B MoE Gemma 4 variants exist in the same cookbook but require --tp-size 2, so they’re not in the gallery as single-GPU recipes.

SGLang version requirement. Gemma 4 support landed in SGLang via PR #21952. The LocalAI sglang backend pins a release that includes it; if you’ve overridden the pin to an older version, this recipe will fail with a “model architecture not recognised” error at load time.

Other speculative algorithms

speculative_algorithm: also accepts EAGLE/EAGLE3 (paired with an EAGLE-style draft head), DFLASH (block-diffusion drafters from z-lab for the Qwen3 family), STANDALONE (a smaller draft LLM verifying a larger target), and NGRAM (no draft model - pure prefix-history speculation). See SGLang’s speculative-decoding docs for the full algorithm matrix.

Tool calling and reasoning parsers

SGLang’s native parsers stream tool_calls and reasoning_content inside ChatDelta - the LocalAI Python backend wires them up per-request rather than via engine_args:. Pick a parser by name:

options:
  - tool_parser:hermes
  - reasoning_parser:deepseek_r1

The full list of registered parsers lives in sglang.srt.function_call and sglang.srt.parser.reasoning_parser.

Reasoning defaults and token budgets

Set SGLang reasoning options in the model’s options: list:

options:
  - reasoning_parser:qwen3
  - thinking_budget:512
  - reasoning_default:on
engine_args:
  enable_strict_thinking: true

thinking_budget sets a positive integer token budget for reasoning on each request. Invalid, zero, and negative values produce a warning and leave the budget unset. SGLang requires engine_args.enable_strict_thinking: true to enforce the budget. LocalAI warns if you configure a budget without that engine option. Keep the budget well below the max_tokens of your requests: if max_tokens is reached first, the budget never triggers and the whole reply can be spent on reasoning, leaving the answer empty.

reasoning_default:on or reasoning_default:off sets the default for LocalAI’s tokenizer chat template. Request metadata enable_thinking set to "true" or "false" overrides this default. An explicit prompt bypasses tokenizer template rendering. When no default or request override is set, the template keeps its own behavior.

LocalAI signals required reasoning when the rendered prompt ends with the configured parser’s opening reasoning token. An explicit output grammar disables this detection. Configure a reasoning parser that matches your model.

The backend reads these options when it loads the model. POST /models/reload rereads model configuration files but does not update options in an already loaded backend. Restarting only the backend does not reread configuration files. Restart LocalAI after changing these options to reload both the configuration and the backend.

vllm.cpp

vllm.cpp is the LocalAI team’s C++ port of vLLM: the same continuous-batching scheduler, paged KV cache and prefix caching, with no Python at inference time. It consumes either a HuggingFace safetensors model directory or a .gguf file, and applies the model’s chat template, tool-call parsing and reasoning split engine-side.

This section is the configuration reference. For installing the backend and the ready-made models the gallery ships for it, see the vllm.cpp backend page.

Setup

name: vllm-cpp
backend: vllm-cpp
parameters:
  model: "Qwen/Qwen3-4B"
context_size: 8192
template:
  use_tokenizer_template: true

Configuring the engine with engine_args

The same engine_args: map the vLLM and SGLang backends accept is honoured here, with keys spelled exactly as vLLM’s own CLI flags - so a speculative_config or kv_transfer_config block written for vLLM works verbatim. Unknown keys are ignored rather than fatal; the engine validates the documents it is handed and reports a precise error at load.

name: qwen35-a3b
backend: vllm-cpp
parameters:
  model: "Qwen/Qwen3.5-A3B"
context_size: 16384
template:
  use_tokenizer_template: true
engine_args:
  # KV cache sizing: num_blocks * block_size tokens of cache.
  block_size: 32
  num_blocks: 1024
  # Concurrency and the per-step chunked-prefill token budget.
  max_num_seqs: 32
  max_num_batched_tokens: 8192
  # Automatic prefix caching. Omit to keep the model's own default
  # (on for dense models, off for hybrid / attention-free ones).
  enable_prefix_caching: true
  # Scheduler admission order: fcfs (default), priority, or lpm
  # (cache-aware longest-prefix-match; needs prefix caching to have any effect).
  scheduling_policy: lpm

Raising max_num_batched_tokens lets more prefill land in a single step, at the cost of decode latency for requests queued behind it. The default deliberately does not scale with max_num_seqs, which is what keeps a large concurrent prefill from blowing up the per-step activation on the hybrid architectures.

enable_prefix_caching and enable_jump_forward are tri-state at the engine boundary: omitting the key defers to a default (the model’s own capability for prefix caching, an environment variable for jump forward), while an explicit false forces the feature off. Those are genuinely different - prefix caching defaults on for dense models - so write the key only when you mean to override.

Speculative decoding

speculative_config: takes the same JSON object as vLLM’s --speculative-config. Three methods are supported.

Architecture limit. At the current engine pin, mtp and dflash are Qwen3.5 / Qwen3.6 only. The engine builds a widened speculative KV cache directly for those families rather than through the model registry, so a speculative config on any other architecture (Llama, GLM, Gemma, Mistral, …) will not work regardless of checkpoint format. ngram needs no draft weights and is not subject to this limit.

Format support. mtp and dflash now work from a .gguf target as well as safetensors. An MTP head is read from the GGUF’s nextn.* tensors when the file declares <arch>.nextn_predict_layers; a GGUF exported WITHOUT the head (converted with --no-mtp, or predating llama.cpp’s Qwen3.5 MTP support) is refused at load naming that as the reason. A DFlash draft may itself be a dflash-arch GGUF, and the target may be a GGUF too. ngram needs no draft weights and works on any format.

MTP (Multi-Token Prediction) uses a draft head shipped inside the target checkpoint’s own mtp.* tensors, so there is no second model to download. It requires a safetensors checkpoint - the mtp.* tensors do not survive GGUF conversion, and an MTP config over a .gguf model is rejected at load.

engine_args:
  speculative_config:
    method: mtp
    # Optional; defaults to the checkpoint's own head depth, which is
    # usually the right value. Must be a multiple of that depth.
    num_speculative_tokens: 1

DFlash uses a separate block-diffusion drafter that proposes a whole block of tokens in one non-autoregressive forward pass. Unlike MTP, the draft is its own checkpoint, so model: is required:

engine_args:
  speculative_config:
    method: dflash
    model: z-lab/Qwen3.6-27B-DFlash
    num_speculative_tokens: 4

The draft shares the target’s embed_tokens and lm_head, so both must come from the same model family and the target must be safetensors.

The engine does not download the draft. model: is resolved, in order, as a path as given, then as the last path segment under LocalAI’s models directory (z-lab/Qwen3.6-27B-DFlash → <models>/Qwen3.6-27B-DFlash, which is what LocalAI’s own downloader produces), then as the whole reference under the models directory. Install the draft into LocalAI first, or give an absolute path to a directory containing config.json. If none of those resolve, the load fails immediately naming every location that was tried, rather than reporting a missing checkpoint from inside the engine.

N-gram needs no draft model at all - it proposes from the prompt’s own suffix history. num_speculative_tokens is required:

engine_args:
  speculative_config:
    method: ngram
    num_speculative_tokens: 4
    prompt_lookup_min: 5
    prompt_lookup_max: 5

Auto-configuration on import. When you import a safetensors repository with backend: vllm-cpp, LocalAI reads the checkpoint’s config.json and, if it declares an MTP head (mtp_num_hidden_layers), writes speculative_config: {method: mtp} into the generated engine_args for you. An explicit speculative_config in your own config is never overwritten. Importing a DFlash draft repository is refused with a warning: a drafter cannot serve on its own, so import the target model and point speculative_config.model at the draft.

External KV cache with LMCache

kv_transfer_config: takes vLLM’s --kv-transfer-config JSON and selects an external KV-cache connector. The lm:// LMCache client lets prefill KV be stored to and reloaded from a shared lmcache.v1.server, so a prefix computed by one replica does not have to be recomputed by the next:

engine_args:
  kv_transfer_config:
    kv_connector: LMCacheConnector
    kv_role: kv_both          # required whenever kv_connector is set
    kv_connector_extra_config:
      host: 127.0.0.1
      port: 65432

kv_role is one of kv_producer (store only), kv_consumer (load only), or kv_both. An unregistered connector name, a missing role, or a malformed document fails the load with an explicit error rather than silently running without the cache.

Legacy options: list

Earlier versions configured this backend through the flat options: list, and those configs keep working. Every key in the table above is still read from there in key:value form, and engine_args wins on any key set in both:

options:
  - max_num_seqs:32
  - enable_prefix_caching:true

New configs should prefer engine_args:, which is the only place the nested speculative_config / kv_transfer_config documents can be written naturally rather than as a single-line JSON string.

Transformers

Transformers is a State-of-the-art Machine Learning library for PyTorch, TensorFlow, and JAX.

LocalAI has a built-in integration with Transformers, and it can be used to run models.

This is an extra backend - in the container images (the extra images already contains python dependencies for Transformers) is already available and there is nothing to do for the setup.

Setup

Create a YAML file for the model you want to use with transformers.

To setup a model, you need to just specify the model name in the YAML config file:

name: transformers
backend: transformers
parameters:
    model: "facebook/opt-125m"
type: AutoModelForCausalLM
quantization: bnb_4bit # One of: bnb_8bit, bnb_4bit, xpu_4bit, xpu_8bit (optional)

The backend will automatically download the required files in order to run the model.

Parameters

Type

• OVModelForCausalLM requires OpenVINO IR Text Generation models from Hugging face
• OVModelForFeatureExtraction works with any Safetensors Transformer Feature Extraction model from Huggingface (Embedding Model)

Please note that streaming is currently not implemented in AutoModelForCausalLM for Intel GPU. AMD GPU support is not implemented. Although AMD CPU is not officially supported by OpenVINO there are reports that it works: YMMV.

Embeddings

Use embeddings: true if the model is an embedding model

Inference device selection

Transformer backend tries to automatically select the best device for inference, anyway you can override the decision manually overriding with the main_gpu parameter.

Example for CUDA: main_gpu: cuda.0

Example for OpenVINO: main_gpu: AUTO:-CPU

This parameter applies to both Text Generation and Feature Extraction (i.e. Embeddings) models.

Inference Precision

Transformer backend automatically select the fastest applicable inference precision according to the device support. CUDA backend can manually enable bfloat16 if your hardware support it with the following parameter:

f16: true

Quantization

Trust Remote Code

Some models like Microsoft Phi-3 requires external code than what is provided by the transformer library. By default it is disabled for security. It can be manually enabled with: trust_remote_code: true

Maximum Context Size

Maximum context size in bytes can be specified with the parameter: context_size. Do not use values higher than what your model support.

Usage example: context_size: 8192

Auto Prompt Template

Usually chat template is defined by the model author in the tokenizer_config.json file. To enable it use the use_tokenizer_template: true parameter in the template section.

Usage example:

template:
  use_tokenizer_template: true

Custom Stop Words

Stopwords are usually defined in tokenizer_config.json file. They can be overridden with the stopwords parameter in case of need like in llama3-Instruct model.

Usage example:

stopwords:
- "<|eot_id|>"
- "<|end_of_text|>"

Usage

Use the completions endpoint by specifying the transformers model:

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{   
   "model": "transformers",
   "prompt": "Hello, my name is",
   "temperature": 0.1, "top_p": 0.1
 }'

Examples

OpenVINO

A model configuration file for openvion and starling model:

name: starling-openvino
backend: transformers
parameters:
  model: fakezeta/Starling-LM-7B-beta-openvino-int8
context_size: 8192
threads: 6
f16: true
type: OVModelForCausalLM
stopwords:
- <|end_of_turn|>
- <|endoftext|>
prompt_cache_path: "cache"
prompt_cache_all: true
template:
  chat_message: |
    {{if eq .RoleName "system"}}{{.Content}}<|end_of_turn|>{{end}}{{if eq .RoleName "assistant"}}<|end_of_turn|>GPT4 Correct Assistant: {{.Content}}<|end_of_turn|>{{end}}{{if eq .RoleName "user"}}GPT4 Correct User: {{.Content}}{{end}}

  chat: |
    {{.Input}}<|end_of_turn|>GPT4 Correct Assistant:

  completion: |
    {{.Input}}

OpenAI Functions and Tools

LocalAI supports running the OpenAI functions and tools API across multiple backends. The OpenAI request shape is the same regardless of which backend runs your model - LocalAI is responsible for extracting structured tool calls from the model’s output before returning the response.

To learn more about OpenAI functions, see also the OpenAI API blog post.

LocalAI also supports JSON mode out of the box on llama.cpp-compatible models.

💡 Check out LocalAGI for an example on how to use LocalAI functions.

Supported backends

Reasoning content (<think>...</think> blocks from DeepSeek R1, Qwen3, Gemma 4, etc.) is returned in the OpenAI reasoning_content field on the same backends. When a model both reasons and calls a tool in the same turn, see Interleaved Thinking with Tool Calls for how the reasoning survives the tool-result round trip.

Setup

llama.cpp

No configuration required - the autoparser detects the tool call format for any ggml/gguf model that was trained with tool support.

Migrating older Ministral 3 configurations

The mistralai_ministral-3-14b-reasoning-2512-multimodal gallery entry uses the model’s embedded template and llama.cpp’s native tool parser. Older installations inherit the Mistral 0.3 prompt and JSON parser, which can return tool calls such as pick_tool{...} as text.

For an existing installation, replace the model YAML’s template and function sections with:

template:
  use_tokenizer_template: true
function:
  disable_no_action: true
  automatic_tool_parsing_fallback: true
  grammar:
    disable: true

Set use_jinja:true in the existing options list:

options:
  - use_jinja:true

Remove the inherited stopwords list so llama.cpp controls the model’s end-of-turn markers. Keep your model path, mmproj, sampling settings, and MCP configuration. Reload the model after saving the YAML. Gallery changes do not rewrite installed model configurations.

vLLM / vLLM Omni

The parser must be specified explicitly because vLLM itself doesn’t auto-detect one. Pass it via the model options:

name: qwen3-8b
backend: vllm
parameters:
  model: Qwen/Qwen3-8B
options:
  - tool_parser:hermes
  - reasoning_parser:qwen3
template:
  use_tokenizer_template: true

When you import a vLLM model through the LocalAI gallery, the importer looks up the model family and pre-fills tool_parser: and reasoning_parser: for you - you only need to override them for non-standard model names.

Available tool parsers include hermes, llama3_json, llama4_pythonic, mistral, qwen3_xml, deepseek_v3, granite4, kimi_k2, glm45, and more. Available reasoning parsers include deepseek_r1, qwen3, mistral, gemma4, granite. See the upstream vLLM documentation for the full list.

MLX / MLX-VLM

MLX backends auto-detect the right tool parser by inspecting the model’s chat template - you don’t need to set anything. Just load an MLX-quantized model that was trained with tool support:

name: qwen2.5-0.5b-mlx
backend: mlx
parameters:
  model: mlx-community/Qwen2.5-0.5B-Instruct-4bit
template:
  use_tokenizer_template: true

The gallery importer will still append tool_parser: and reasoning_parser: entries to the YAML for visibility and consistency with the other backends, but those are informational - the runtime auto-detection in the MLX backend ignores them and uses the parser matched to the chat template.

Supported parser families: hermes/json_tools, mistral, gemma4, glm47, kimi_k2, longcat, minimax_m2, pythonic, qwen3_coder, function_gemma.

Usage example

You can configure a model manually with a YAML config file in the models directory, for example:

name: gpt-3.5-turbo
parameters:
  # Model file name
  model: ggml-openllama.bin
  top_p: 80
  top_k: 0.9
  temperature: 0.1

To use the functions with the OpenAI client in python:

from openai import OpenAI

messages = [{"role": "user", "content": "What is the weather like in Beijing now?"}]
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Return the temperature of the specified region specified by the user",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "User specified region",
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "temperature unit"
                    },
                },
                "required": ["location"],
            },
        },
    }
]

client = OpenAI(
    # This is the default and can be omitted
    api_key="test",
    base_url="http://localhost:8080/v1/"
)

response =client.chat.completions.create(
    messages=messages,
    tools=tools,
    tool_choice ="auto",
    model="gpt-4",
)
#...