Shell Completion

LocalAI provides shell completion support for bash, zsh, and fish shells. Once installed, tab completion works for all CLI commands, subcommands, and flags.

Generating Completion Scripts

Use the completion subcommand to generate a completion script for your shell:

local-ai completion bash
local-ai completion zsh
local-ai completion fish

Installation

Bash

Add the following to your ~/.bashrc:

source <(local-ai completion bash)

Or install it system-wide:

local-ai completion bash > /etc/bash_completion.d/local-ai

Zsh

Add the following to your ~/.zshrc:

source <(local-ai completion zsh)

Or install it to a completions directory:

local-ai completion zsh > "${fpath[1]}/_local-ai"

If shell completions are not already enabled in your zsh environment, add the following to the beginning of your ~/.zshrc:

autoload -Uz compinit
compinit

Fish

local-ai completion fish | source

Or install it permanently:

local-ai completion fish > ~/.config/fish/completions/local-ai.fish

Usage

After installation, restart your shell or source your shell configuration file. Then type local-ai followed by a tab to see available commands:

$ local-ai <TAB>
run              backends         completion       explorer         models
federated        sound-generation transcript       tts              util

Tab completion also works for subcommands and flags:

$ local-ai models <TAB>
install  list

$ local-ai run --<TAB>
--address          --backends-path    --context-size     --debug            ...

P2P API reference

LocalAI supports peer-to-peer (P2P) networking for distributed inference. The P2P API endpoints allow you to monitor connected worker and federated nodes, retrieve the P2P network token, and get cluster statistics.

For an overview of distributed inference setup, see Distributed Inference.

Endpoints

List all P2P nodes

• Method: GET
• Endpoint: /api/p2p

Returns all worker and federated nodes in the P2P network.

Response

Each node object:

Usage

curl http://localhost:8080/api/p2p

Example response

{
  "nodes": [
    {
      "Name": "worker-1",
      "ID": "abc123",
      "TunnelAddress": "192.168.1.10:9090",
      "ServiceID": "worker",
      "LastSeen": "2025-01-15T10:30:00Z"
    }
  ],
  "federated_nodes": [
    {
      "Name": "federation-1",
      "ID": "def456",
      "TunnelAddress": "192.168.1.20:9090",
      "ServiceID": "federated",
      "LastSeen": "2025-01-15T10:30:05Z"
    }
  ]
}

Get P2P token

• Method: GET
• Endpoint: /api/p2p/token

Returns the P2P network token used for node authentication.

Usage

curl http://localhost:8080/api/p2p/token

Response

Returns the token as a plain text string.

List worker nodes

• Method: GET
• Endpoint: /api/p2p/workers

Returns worker nodes with online status.

Response

A node is considered online if it was last seen within the past 40 seconds.

Usage

curl http://localhost:8080/api/p2p/workers

List federated nodes

• Method: GET
• Endpoint: /api/p2p/federation

Returns federated nodes with online status. Same response format as /api/p2p/workers.

Usage

curl http://localhost:8080/api/p2p/federation

Get P2P statistics

• Method: GET
• Endpoint: /api/p2p/stats

Returns aggregate statistics about the P2P cluster.

Response

Usage

curl http://localhost:8080/api/p2p/stats

Example response

{
  "workers": {
    "online": 3,
    "total": 5
  },
  "federated": {
    "online": 2,
    "total": 2
  }
}

Error Responses

System Info and Version

LocalAI provides endpoints to inspect the running instance, including available backends, loaded models, and version information.

System Information

• Method: GET
• Endpoint: /system

Returns available backends and currently loaded models.

Response

Per-model VRAM

On Linux, size_vram reports resident device memory for the local backend process and its child processes. LocalAI reads drm-resident-local* and drm-resident-vram* from /proc and counts each DRM client once per GPU. Host-memory regions are excluded. The reading includes buffers attributed to the backend, without separating weights, KV cache, and other allocations. See the kernel DRM accounting specification for these counters.

Kernels without CONFIG_PROC_CHILDREN are supported. When /proc does not provide per-thread children files, LocalAI finds child processes through the parent PIDs in /proc/<pid>/stat.

The field is omitted when accounting is unavailable or incomplete. This includes external and distributed backends, macOS, proprietary NVIDIA drivers, primary DRM nodes (/dev/dri/card*), missing resident counters, and unreadable process information. A present value of 0 means the supported counters report zero bytes. Treat an absent field as unknown.

This is a snapshot of driver accounting, not a memory reservation. Shared buffers can appear in different clients’ counters, and allocations can change during collection. Do not treat the sum across models as exclusive physical GPU usage. These readings do not replace capacity checks when scheduling work.

Usage

curl http://localhost:8080/system

Example response

{
  "backends": [
    "llama-cpp",
    "huggingface",
    "diffusers",
    "whisper"
  ],
  "loaded_models": [
    {
      "id": "my-llama-model",
      "backend": "llama-cpp",
      "size_vram": 5368709120,
      "process": {
        "pid": 48213,
        "rss_bytes": 5368709120,
        "memory_percent": 7.8,
        "cpu_percent": 42.5,
        "started_at": "2026-09-21T09:12:44Z"
      }
    },
    {
      "id": "whisper-1",
      "backend": "whisper"
    }
  ]
}

cpu_percent covers the time since the previous call to this endpoint, so a dashboard polling every few seconds gets a current reading. The WebUI’s Operate → This machine page polls it every five seconds.

Version

• Method: GET
• Endpoint: /version

Returns the LocalAI version and build commit.

Response

Usage

curl http://localhost:8080/version

Example response

{
  "version": "2.26.0 (a1b2c3d4)"
}

Error Responses

Model compatibility table

Besides llama based models, LocalAI is compatible also with other architectures. The table below lists all the backends, compatible models families and the associated repository.

Text Generation & Language Models

Speech-to-Text

Text-to-Speech

Speech-to-Speech

Music & Sound Generation

Image & Video Generation

Vision, Detection & Recognition

Audio Processing

Utilities & Other

Acceleration Support Summary

GPU Acceleration

• NVIDIA CUDA: CUDA 12.0, CUDA 13.0 support across most backends
• AMD ROCm: HIP-based acceleration for AMD GPUs
• Intel oneAPI: SYCL-based acceleration for Intel GPUs (F16/F32 precision)
• Vulkan: Cross-platform GPU acceleration
• Metal: Apple Silicon GPU acceleration (M1/M2/M3+)

Specialized Hardware

• NVIDIA Jetson (L4T CUDA 12): ARM64 support for embedded AI (AGX Orin, Jetson Nano, Jetson Xavier NX, Jetson AGX Xavier)
• NVIDIA Jetson (L4T CUDA 13): ARM64 support for embedded AI (DGX Spark)
• Apple Silicon: Native Metal acceleration for Mac M1/M2/M3+
• Darwin x86: Intel Mac support

CPU Optimization

• AVX/AVX2/AVX512: Advanced vector extensions for x86
• Quantization: 4-bit, 5-bit, 8-bit integer quantization support
• Mixed Precision: F16/F32 mixed precision support

Note: any backend name listed above can be used in the backend field of the model configuration file (See the advanced section).

• * Only for CUDA and OpenVINO CPU/XPU acceleration.

Architecture

LocalAI is an API written in Go that serves as an OpenAI shim, enabling software already developed with OpenAI SDKs to seamlessly integrate with LocalAI. It can be effortlessly implemented as a substitute, even on consumer-grade hardware. This capability is achieved by employing various C++ backends, including ggml, to perform inference on LLMs using both CPU and, if desired, GPU. Internally LocalAI backends are just gRPC server, indeed you can specify and build your own gRPC server and extend LocalAI in runtime as well. It is possible to specify external gRPC server and/or binaries that LocalAI will manage internally.

LocalAI uses a mixture of backends written in various languages (C++, Golang, Python, …). You can check the model compatibility table to learn about all the components of LocalAI.

Backstory

As much as typical open source projects starts, I, mudler, was fiddling around with llama.cpp over my long nights and wanted to have a way to call it from go, as I am a Golang developer and use it extensively. So I’ve created LocalAI (or what was initially known as llama-cli) and added an API to it.

But guess what? The more I dived into this rabbit hole, the more I realized that I had stumbled upon something big. With all the fantastic C++ projects floating around the community, it dawned on me that I could piece them together to create a full-fledged OpenAI replacement. So, ta-da! LocalAI was born, and it quickly overshadowed its humble origins.

Now, why did I choose to go with C++ bindings, you ask? Well, I wanted to keep LocalAI snappy and lightweight, allowing it to run like a champ on any system and avoid any Golang penalties of the GC, and, most importantly built on shoulders of giants like llama.cpp. Go is good at backends and API and is easy to maintain. And hey, don’t forget that I’m all about sharing the love. That’s why I made LocalAI MIT licensed, so everyone can hop on board and benefit from it.

As if that wasn’t exciting enough, as the project gained traction, mkellerman and Aisuko jumped in to lend a hand. mkellerman helped set up some killer examples, while Aisuko is becoming our community maestro. The community now is growing even more with new contributors and users, and I couldn’t be happier about it!

Oh, and let’s not forget the real MVP here-llama.cpp. Without this extraordinary piece of software, LocalAI wouldn’t even exist. So, a big shoutout to the community for making this magic happen!

CLI Reference

Complete reference for all LocalAI command-line interface (CLI) parameters and environment variables.

For client-side text inference measurements, see Benchmark text models.

These options configure the LocalAI server process. To configure an individual model, see Model Configuration.

Note: All CLI flags can also be set via environment variables. Environment variables take precedence over CLI flags. See .env files for configuration file support.

Global Flags

Diagnostics flags

See Profiling and request diagnostics for sampling semantics, phase boundaries, and protected pod captures.

These options apply only to local-ai run, including distributed frontends. Diagnostics state is per-instance, not shared between replicas. Debug logging enables neither feature. The profiling listener is separate from the public API and accepts only numeric loopback addresses with explicit ports. --preload-backend-only validates these options but does not start profiling or sampling.

Sampling adds overhead. The process applies configured rates once and resets them to zero on shutdown. There is no runtime toggle, continuous capture, or public API route for diagnostics. The private listener has no authentication: other processes in the same network namespace can access profiles. Profiles can contain secrets; protect captured files. The /debug/pprof/cmdline endpoint is unavailable. Shutdown allows five seconds for cleanup, then force-closes profiling connections. Existing application initialization operations may not promptly observe cancellation. The signal callback still returns after its cleanup budget expires. If initialization fails without returning an application handle, this lifecycle cannot shut down resources created inside that failed initialization. Application shutdown has no context parameter; work that ignores cancellation can outlive the cleanup budget.

Storage Flags

| --models-config-file | | YAML file containing a list of model backend configs (alias: --config-file) | $LOCALAI_MODELS_CONFIG_FILE, $CONFIG_FILE | | --addon-host-ops | fine-tuning=finetune.,quantization=quantize.,sandbox=workspace.,sandbox=egress.,office=workspace.,web=workspace.,telegram=connector. | Add-ons that may call host operations, as a comma list of addon-id=operation-prefix. An add-on that is not listed gets no host operation channel; an empty value lists none. See Host operations | $LOCALAI_ADDON_HOST_OPS | | --addons-role | combined | How this process takes part in the add-on owner: combined (the node is its own owner), owner or frontend. See Turning the owner on or off | $LOCALAI_ADDONS_ROLE | | --addons-owner-address | | https://host:port of the add-on owner carrier, with no path. Required for the owner and frontend roles | $LOCALAI_ADDONS_OWNER_ADDRESS | | --addons-owner-listen | | owner role only: the address the carrier listens on. Empty listens on all interfaces, on the port of --addons-owner-address | $LOCALAI_ADDONS_OWNER_LISTEN | | --addons-carrier-dir | | Directory with the carrier key set from local-ai util addons-carrier-keys. Required for the owner and frontend roles | $LOCALAI_ADDONS_CARRIER_DIR | | --addons-enabled | true | Run the add-on owner. --addons-enabled=false turns add-ons off completely. See The addons feature gate | $LOCALAI_ADDONS_ENABLED | | --addons-ownership-mode | local-owner without authentication, named-user with it | named-user gives each account its own access to add-ons; local-owner makes every caller share one owner | $LOCALAI_ADDONS_OWNERSHIP_MODE | | --auto-upgrade-addons | false | Upgrade installed add-ons on their own when a newer version appears in their library | $LOCALAI_AUTO_UPGRADE_ADDONS | | --addons-official-library | true | Register the official add-on library on the first start | $LOCALAI_ADDONS_OFFICIAL_LIBRARY | | --addons-backend-library | true | Register the backend library on the first start | $LOCALAI_ADDONS_BACKEND_LIBRARY | | --addons-backend-max-bytes | 8589934592 | Largest compressed size, in bytes, of a backend image installed from a library | $LOCALAI_ADDONS_BACKEND_MAX_BYTES | | --max-concurrent-backend-operations | 1 | How many backend install, upgrade and delete operations run at once. 1 runs them one after the other. Operations on the same backend always run one at a time. In distributed mode it also bounds how many nodes one operation is sent to at the same time | $LOCALAI_MAX_CONCURRENT_BACKEND_OPERATIONS | | --artifact-download-concurrency | 1 | How many files of a model artifact to download at once. 1 downloads sequentially. Raising it helps artifacts split into many files on a fast link, at the cost of more concurrent load on the models volume. Whole files only — a single file is never split, so resume and per-file checksum verification are unaffected | $LOCALAI_ARTIFACT_DOWNLOAD_CONCURRENCY |

Backend processes receive a private scratch directory through TMPDIR, TMP, and TEMP. LocalAI removes that directory when the backend exits and removes abandoned directories left by a LocalAI crash before starting another backend. Set $LOCALAI_BACKEND_TEMP_DIR to choose their base volume. LocalAI always appends localai-UID/backend-runtime; the default base is TMPDIR.

Backend Flags

For more information on VRAM management, see VRAM and Memory Management.

Models Flags

Note: You can also pass model configuration URLs as positional arguments: local-ai run MODEL_URL1 MODEL_URL2 ...

Performance Flags

Default thread count

When the server thread setting is unset or zero, LocalAI selects the smaller of the physical core count and Go’s current GOMAXPROCS value at startup. The result is at least one thread.

By default, Go accounts for CPU affinity and Linux container CPU quotas. For example, docker run --cpus=2 limits the automatic default to two threads. Go normally applies a minimum of two for CPU quotas, so --cpus=1 can still produce two threads. Set --threads=1 if you need one inference thread.

An explicit GOMAXPROCS value overrides Go’s automatic CPU detection. A nonzero server thread setting from the CLI, environment, or saved runtime settings bypasses this automatic default.

API Flags

Collections and Assistant Flags

The collections service is the knowledge base that the chats search. The old LOCALAI_AGENT_POOL_* names still work and warn once; see Moving from the old chat and agents.

LOCALAI_DISABLE_AGENTS is now a name for LOCALAI_DISABLE_COLLECTIONS; nothing switches the chats themselves off except the chats permission of each user.

Hardening Flags

Authentication Flags

See Authentication & Authorization for full documentation.

Chats Flags

See Chats, Agent tasks and jobs and Chat workers.

Collections Flags

See Collections. The old --agent-pool-* names still work and log a deprecation warning.

Chat Flags

Use local-ai chat to run the built-in terminal agent against a LocalAI server. The agent can run shell commands, delegate to sub-agents, and use MCP tools. Read-only calls run on their own; everything else goes through an approval prompt you answer. See Terminal agent for the full feature page.

The agent’s own subcommands are reached by passing them through. LocalAI’s flags must come first, because everything after the first positional argument is forwarded verbatim:

local-ai chat plugin install https://github.com/user/plugin
local-ai chat skill list
local-ai chat mcp add my-server -- npx -y @modelcontextprotocol/server-filesystem /tmp
local-ai chat --config-dir /srv/agent plugin list   # flags first

Summon the agent from any shell prompt with Ctrl+Space:

echo 'eval "$(local-ai chat --init zsh)"' >> ~/.zshrc

Inside a session:

• /models lists the models the server offers, marking the current one.
• /model <name> switches model, keeping the conversation.
• /compact summarizes the conversation so far to free up context.
• /skill <name>, /agent <name> <task>, /attach <file>, /goal <text>.

If no server is reachable, the agent offers to start one for the session (interactive terminals only) and otherwise points you at local-ai run.

Piped use needs --cli, and exits 0 when it answers:

echo "what is 2+2" | local-ai chat --cli

Read-only tools still run in a piped session. A tool call that is not read-only has nobody to approve it, so it is denied and the session exits 3, a code chosen to be distinct from the 1 a failure reports.

P2P Flags

Library sources

local-ai library manages the sources LocalAI reads add-ons, backends and models from. The sources are stored under the data path (addons/sources.json), the file the server reads, so the commands work with the server stopped.

All local-ai library commands accept --official-libraries (LOCALAI_OFFICIAL_LIBRARIES, default true). With --official-libraries=false the list shows the shipped default sources as off. The same switch is the runtime setting official_libraries and the toggle “Use official libraries” in Settings. See Library sources.

When two enabled sources list the same entry (same kind and id), your own source wins over a shipped default. If two of your own sources list it, the entry is refused and the message names both sources: disable or remove one of them. The --galleries and --backend-galleries flags keep working: each gallery they name is added as a source of your own (kind model or backend, format gallery) at startup, unless it is already a source.

Other Commands

LocalAI supports several subcommands beyond run:

• local-ai models - Manage LocalAI models and definitions
• local-ai backends - Manage LocalAI backends and definitions
• local-ai chat - Run the built-in terminal agent against a LocalAI server
• local-ai tts - Convert text to speech
• local-ai sound-generation - Generate audio files from text or audio
• local-ai transcript - Convert audio to text
• local-ai mcp-server - Run the LocalAI admin and client tool surface as a stdio MCP server (controls a remote LocalAI instance over HTTP). --files-dir (LOCALAI_MCP_FILES_DIR) is the folder where the client tools read and write files; it has no default, and without it the client tools that read or write files are not registered. --read-only also leaves out the client tools that write files or make content (generate_*, speak, embed, voice_profile create and delete)
• local-ai worker - Start a worker for distributed mode (generic, backend-agnostic)
• local-ai p2p-worker - Run workers to distribute workload via p2p (llama.cpp-only)
• local-ai agent-worker - Start a worker for distributed mode (serves MCP tool calls via NATS)
• local-ai util - Utility commands
• local-ai explorer - Run P2P explorer
• local-ai federated - Run LocalAI in federated mode
• local-ai completion - Generate shell completion scripts for bash, zsh, or fish

Use local-ai <command> --help for more information on each command.

Add-on carrier keys

local-ai util addons-carrier-keys makes the key set that a separate add-on owner and its frontends share. See Running the owner as its own workload for how the set is used.

local-ai util addons-carrier-keys --out ./carrier-keys --owner-host addons.owner.example

The command prints which files go to the owner and which go to the frontends. It writes the files with mode 0600 in a directory with mode 0700, and it does not keep the CA key.

Examples

Basic Usage

./local-ai run

./local-ai run --models-path /path/to/models --address :9090

./local-ai run --f16

Environment Variables

export LOCALAI_MODELS_PATH=/path/to/models
export LOCALAI_ADDRESS=:9090
export LOCALAI_F16=true
./local-ai run

Advanced Configuration

./local-ai run \
  --models model1.yaml model2.yaml \
  --enable-watchdog-idle \
  --watchdog-idle-timeout=10m \
  --p2p \
  --federated

Related Documentation

• See Advanced Usage for configuration examples
• See VRAM and Memory Management for memory management options

API Error Reference

This page documents the error responses returned by the LocalAI API. LocalAI supports multiple API formats (OpenAI, Anthropic, Open Responses), each with its own error structure.

Error Response Formats

OpenAI-Compatible Format

Most endpoints return errors using the OpenAI-compatible format:

{
  "error": {
    "code": 400,
    "message": "A human-readable description of the error",
    "type": "invalid_request_error",
    "param": null
  }
}

This format is used by: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/images/generations, /v1/audio/transcriptions, /models, and other OpenAI-compatible endpoints.

Anthropic Format

The /v1/messages endpoint returns errors in Anthropic’s format:

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "A human-readable description of the error"
  }
}

Open Responses Format

The /v1/responses endpoint returns errors with this structure:

{
  "error": {
    "type": "invalid_request",
    "message": "A human-readable description of the error",
    "code": "",
    "param": "parameter_name"
  }
}

HTTP Status Codes

Global Error Handling

Authentication Errors (401)

When authentication is enabled - either via API keys (LOCALAI_API_KEY) or the user auth system (LOCALAI_AUTH=true) - API requests must include valid credentials. Credentials can be provided through:

• Authorization: Bearer <key> header (API key, user API key, or session ID)
• x-api-key: <key> header
• xi-api-key: <key> header
• session cookie (user auth sessions)
• token cookie (legacy API keys)

Authorization Errors (403)

When user authentication is enabled, admin-only endpoints (model management, system settings, traces, agents, etc.) return 403 if accessed by a non-admin user. See Authentication & Authorization for the full list of admin-only endpoints.

Example request without a key:

curl http://localhost:8080/v1/models \
  -H "Content-Type: application/json"

Error response:

{
  "error": {
    "code": 401,
    "message": "An authentication key is required",
    "type": "invalid_request_error"
  }
}

The response also includes the header WWW-Authenticate: Bearer.

Request Parsing Errors (400)

All endpoints return a 400 error if the request body cannot be parsed:

{
  "error": {
    "code": 400,
    "message": "failed parsing request body: <details>",
    "type": ""
  }
}

Not Found (404)

Requests to undefined routes return:

{
  "error": {
    "code": 404,
    "message": "Resource not found"
  }
}

Opaque Errors Mode

When LOCALAI_OPAQUE_ERRORS=true is set, all error responses return an empty body with only the HTTP status code. This is a security hardening option that prevents information leaks.

Per-Endpoint Error Scenarios

Chat Completions - POST /v1/chat/completions

# Missing model field
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "hello"}]}'

See also: Text Generation

Completions - POST /v1/completions

Embeddings - POST /v1/embeddings

See also: Embeddings

Image Generation - POST /v1/images/generations

See also: Image Generation

Image Editing (Inpainting) - POST /v1/images/edits

Audio Transcription - POST /v1/audio/transcriptions

See also: Audio to Text

Text to Speech - POST /v1/audio/speech, POST /tts

See also: Text to Audio

ElevenLabs TTS - POST /v1/text-to-speech/:voice-id

ElevenLabs Sound Generation - POST /v1/sound-generation

Reranking - POST /v1/rerank, POST /jina/v1/rerank

See also: Reranker

Anthropic Messages - POST /v1/messages

# Missing model field
curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "hello"}], "max_tokens": 100}'

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "model is required"
  }
}

Open Responses - POST /v1/responses

# Missing model field
curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{"input": "hello"}'

{
  "error": {
    "type": "invalid_request",
    "message": "model is required",
    "code": "",
    "param": ""
  }
}

Open Responses - GET /v1/responses/:id

Open Responses Events - GET /v1/responses/:id/events

Object Detection - POST /v1/detection

See also: Object Detection

Video Generation - POST /v1/video/generations

Voice Activity Detection - POST /v1/audio/vad

Tokenize - POST /v1/tokenize

Slot cache - POST /v1/slots/save, POST /v1/slots/restore

Models - GET /v1/models, GET /models

See also: Model Gallery

Handling Errors in Client Code

Python (OpenAI SDK)

from openai import OpenAI, APIError

client = OpenAI(base_url="http://localhost:8080/v1", api_key="your-key")

try:
    response = client.chat.completions.create(
        model="my-model",
        messages=[{"role": "user", "content": "hello"}],
    )
except APIError as e:
    print(f"Status: {e.status_code}, Message: {e.message}")

curl

# Check HTTP status code
response=$(curl -s -w "\n%{http_code}" http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "nonexistent", "messages": [{"role": "user", "content": "hi"}]}')

http_code=$(echo "$response" | tail -1)
body=$(echo "$response" | head -1)

if [ "$http_code" -ne 200 ]; then
  echo "Error $http_code: $body"
fi

Related Configuration

LocalAI binaries

LocalAI binaries are available for both Linux and MacOS platforms and can be executed directly from your command line. These binaries are continuously updated and hosted on our GitHub Releases page. This method also supports Windows users via the Windows Subsystem for Linux (WSL).

macOS Download

You can download the DMG and install the application:

Note: the DMGs are not signed by Apple as quarantined. See https://github.com/mudler/LocalAI/issues/6268 for a workaround, fix is tracked here: https://github.com/mudler/LocalAI/issues/6244

Otherwise, use the following one-liner command in your terminal to download and run LocalAI on Linux or MacOS:

curl -Lo local-ai "https://github.com/mudler/LocalAI/releases/download/v4.12.1/local-ai-$(uname -s)-$(uname -m)" && chmod +x local-ai && ./local-ai

Otherwise, here are the links to the binaries:

Runtime errors and troubleshooting

This page maps the runtime and backend error messages you actually see in the logs (or in an API response) to their likely cause and fix. It covers failures that happen while a model is loading or running, as opposed to API-envelope validation errors (bad request shape, unknown field, wrong content type), which are documented in /docs/reference/api-errors/.

If you only have an HTTP 500 and no message, read How to read the real error first: the useful text is almost always in the server log, not in the HTTP body.

Symptom, cause, fix

The left column is the literal string as it appears in the LocalAI server log (or, for the cooldown case, in the HTTP response). Match on the string, then read across.

How to read the real error

Most user-facing failures surface as an HTTP 500 whose body is a short, generic message. The real cause is in the LocalAI server log, where the backend gRPC error is logged in full:

1. Look at the server log, not just the HTTP response. A 500 from a chat or completion request usually wraps a backend gRPC error. The log line that matters is the one printed by LocalAI when the backend replied (or failed to start).

2. Turn on debug output. Run with DEBUG=true in the environment, or pass --log-level=debug (equivalently --debug) on the command line. This prints the backend’s stdout/stderr, the load parameters, and per-token timing.

   DEBUG=true local-ai run
   # or
   local-ai run --log-level=debug

3. Read the backend’s own output. Backend engines write their diagnostics to stderr, which LocalAI captures into its log at debug level. A load failure (could not load model, grpc service not ready) almost always has the concrete reason (out of memory, missing library, unsupported quantization, illegal instruction) in the backend lines immediately above the LocalAI error.

Performance: everything is slow

Slow inference is usually a configuration or hardware issue rather than an error:

• Do not store models on an HDD. Prefer an SSD. If you are stuck with an HDD, disable mmap in the model config so the model loads fully into memory.
• Do not overbook the CPU. Ideally --threads matches the number of physical cores. For a 4-core CPU, allocate <= 4 threads per model.
• Run with DEBUG=true to see token-inference stats and confirm where time is going.
• Confirm you are getting output at all: send a simple request with "stream": true and watch how fast tokens arrive.
• Check GPU offload. If you expect GPU acceleration, confirm the model actually loaded onto the GPU (the backend log reports offloaded layers). See /docs/features/gpu-acceleration/.

See also

• /docs/reference/api-errors/ for API-envelope error formats and status codes.
• /docs/getting-started/first-agent/ for the “if your agent will not run” checklist (agent-specific failures).
• /docs/reference/cli-reference/ for the full flag list, including the model-load failure cooldown and watchdog options.

Running on Nvidia ARM64

LocalAI can be run on Nvidia ARM64 devices, such as the Jetson Nano, Jetson Xavier NX, Jetson AGX Orin, and Nvidia DGX Spark. The following instructions will guide you through building and using the LocalAI container for Nvidia ARM64 devices.

Platform Compatibility

• CUDA 12 L4T images: Compatible with Nvidia AGX Orin and similar platforms (Jetson Nano, Jetson Xavier NX, Jetson AGX Xavier)
• CUDA 13 L4T images: Compatible with Nvidia DGX Spark

Prerequisites

• Docker engine installed (https://docs.docker.com/engine/install/ubuntu/)
• Nvidia container toolkit installed (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-ap)

Pre-built Images

Pre-built images are available on quay.io and dockerhub:

CUDA 12 (for AGX Orin and similar platforms)

docker pull quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64
# or
docker pull localai/localai:latest-nvidia-l4t-arm64

CUDA 13 (for DGX Spark)

docker pull quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64-cuda-13
# or
docker pull localai/localai:latest-nvidia-l4t-arm64-cuda-13

Build the container

If you need to build the container yourself, use the following commands:

CUDA 12 (for AGX Orin and similar platforms)

git clone https://github.com/mudler/LocalAI

cd LocalAI

docker build --build-arg SKIP_DRIVERS=true --build-arg BUILD_TYPE=cublas --build-arg BASE_IMAGE=nvcr.io/nvidia/l4t-jetpack:r36.4.0 --build-arg IMAGE_TYPE=core -t quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-core .

CUDA 13 (for DGX Spark)

git clone https://github.com/mudler/LocalAI

cd LocalAI

docker build --build-arg SKIP_DRIVERS=false --build-arg BUILD_TYPE=cublas --build-arg CUDA_MAJOR_VERSION=13 --build-arg CUDA_MINOR_VERSION=0 --build-arg BASE_IMAGE=ubuntu:24.04 --build-arg IMAGE_TYPE=core -t quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-cuda-13-core .

Usage

Run the LocalAI container on Nvidia ARM64 devices using the following commands, where /data/models is the directory containing the models:

CUDA 12 (for AGX Orin and similar platforms)

docker run -e DEBUG=true -p 8080:8080 -v /data/models:/models -ti --restart=always --name local-ai --runtime nvidia --gpus all quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64

CUDA 13 (for DGX Spark)

docker run -e DEBUG=true -p 8080:8080 -v /data/models:/models -ti --restart=always --name local-ai --runtime nvidia --gpus all quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64-cuda-13

Note: /data/models is the directory containing the models. You can replace it with the directory containing your models.

GPU reporting in distributed mode

If you run a worker on a Jetson, DGX Spark (GB10), or Thor and the Nodes page in the frontend shows the node as fully used, check two things:

1. NVIDIA_DRIVER_CAPABILITIES must include utility so nvidia-smi / NVML work inside the container. With --gpus all alone (or --runtime nvidia without extra flags) only compute is wired in on some driver versions. Add -e NVIDIA_DRIVER_CAPABILITIES=compute,utility to your docker run. For Docker Compose with driver: nvidia, use capabilities: [gpu, compute, utility] on the device reservation. Include compute for CUDA libraries such as libcuda.so.1; utility alone only provides monitoring libraries and tools.
2. Pass --init to docker run (or init: true in compose) so the container has a proper PID 1 reaper - otherwise short-lived child processes like nvidia-smi can intermittently fail with waitid: no child processes.

On unified-memory devices LocalAI auto-detects the SoC via /sys/devices/soc0/{family,soc_id} and reports system RAM as VRAM, so nvidia-smi is not strictly required for VRAM metrics. See Distributed Mode → NVIDIA GPU support for full context.

AI Coding Assistants

This document provides guidance for AI tools and developers using AI assistance when contributing to LocalAI.

LocalAI follows the same guidelines as the Linux kernel project for AI-assisted contributions. See the upstream policy here: https://docs.kernel.org/process/coding-assistants.html. The rules below mirror that policy, adapted to LocalAI’s license and project layout.

AI tools helping with LocalAI development should follow the standard project development process:

• CONTRIBUTING.md - development workflow, commit conventions, and PR guidelines
• AGENTS.md - the agent entry point with links to all detailed topic guides
• .agents/ai-coding-assistants.md - the full policy source of truth

Licensing and Legal Requirements

All contributions must comply with LocalAI’s licensing requirements:

• LocalAI is licensed under the MIT License
• New source files should use the SPDX license identifier MIT where applicable to the file type
• Contributions must be compatible with the MIT License and must not introduce code under incompatible licenses (e.g., GPL) without an explicit discussion with maintainers

Signed-off-by and Developer Certificate of Origin

AI agents MUST NOT add Signed-off-by tags. Only humans can legally certify the Developer Certificate of Origin (DCO). The human submitter is responsible for:

• Reviewing all AI-generated code
• Ensuring compliance with licensing requirements
• Adding their own Signed-off-by tag (when the project requires DCO) to certify the contribution
• Taking full responsibility for the contribution

AI agents MUST NOT add Co-Authored-By trailers for themselves either. A human reviewer owns the contribution; the AI’s involvement is recorded via Assisted-by (see below).

Exception: automation operated by a maintainer

The rule above addresses the common case, an AI assistant helping a human contributor who then signs off. It does not fit automation that a maintainer runs themselves, which opens pull requests with no human submitter to sign. Applied literally there, nothing ever signs and the DCO check blocks the pull request permanently.

A maintainer-operated bot MUST therefore add a Signed-off-by trailer naming the maintainer who operates it, not the bot and not the model:

Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

This is not the AI certifying the DCO. The maintainer is, exactly as they do for a commit they typed by hand: they configured the automation, they own its output, and they take responsibility for it when they merge it. The Assisted-by trailer still records that a model produced the code, so the provenance trail is unchanged.

The exception is narrow and does not widen the rule for anyone else:

• It applies only to automation a LocalAI maintainer operates and whose output that maintainer reviews before merge.
• The sign-off names a real person who accepts DCO responsibility.
• An AI assistant helping an outside contributor still MUST NOT sign off. That contributor adds their own trailer.
• A bot MUST NOT sign off on behalf of anyone other than its operator, and MUST NOT add a trailer for a contributor whose branch it pushes to. If automation contributes to someone else’s branch, it leaves the sign-off to that contributor.

Attribution

When AI tools contribute to LocalAI development, proper attribution helps track the evolving role of AI in the development process. Contributions should include an Assisted-by tag in the commit message trailer in the following format:

Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]

Where:

• AGENT_NAME - name of the AI tool or framework (e.g., Claude, Copilot, Cursor)
• MODEL_VERSION - specific model version used (e.g., claude-opus-4-7, gpt-5)
• [TOOL1] [TOOL2] - optional specialized analysis tools invoked by the agent (e.g., golangci-lint, staticcheck, go vet)

Basic development tools (git, go, make, editors) should not be listed.

Example

fix(llama-cpp): handle empty tool call arguments

Previously the parser panicked when the model returned a tool call with
an empty arguments object. Fall back to an empty JSON object in that
case so downstream consumers receive a valid payload.

Assisted-by: Claude:claude-opus-4-7 golangci-lint
Signed-off-by: Jane Developer <jane@example.com>

Scope and Responsibility

Using an AI assistant does not reduce the contributor’s responsibility. The human submitter must:

• Understand every line that lands in the PR
• Verify that generated code compiles, passes tests, and follows the project style
• Confirm that any referenced APIs, flags, or file paths actually exist in the current tree (AI models may hallucinate identifiers)
• Not submit AI output verbatim without review

Reviewers may ask for clarification on any change regardless of how it was produced. “An AI wrote it” is not an acceptable answer to a design question.