Advanced usage

Model Configuration with YAML Files

LocalAI uses YAML configuration files to define model parameters, templates, and behavior. You can create individual YAML files in the models directory or use a single configuration file with multiple models.

Quick Example:

name: gpt-3.5-turbo
parameters:
  model: luna-ai-llama2-uncensored.ggmlv3.q5_K_M.bin
  temperature: 0.3

context_size: 512
threads: 10
backend: llama-cpp

template:
  completion: completion
  chat: chat

For a complete reference of all available configuration options, see the Model Configuration page.

Configuration File Locations:

1. Individual files: Create .yaml files in your models directory (e.g., models/gpt-3.5-turbo.yaml)
2. Single config file: Use --models-config-file or LOCALAI_MODELS_CONFIG_FILE to specify a file containing multiple models
3. Remote URLs: Specify a URL to a YAML configuration file at startup:

   local-ai run github://mudler/LocalAI/examples/configurations/phi-2.yaml@master

See also chatbot-ui as an example on how to use config files.

Prompt templates

The API doesn’t inject a default prompt for talking to the model. You have to use a prompt similar to what’s described in the standford-alpaca docs: https://github.com/tatsu-lab/stanford_alpaca#data-release.

The below instruction describes a task. Write a response that appropriately completes the request.

### Instruction:
{{.Input}}

### Response:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{{.Instruction}}

### Input:
{{.Input}}

### Response:

Install models using the API

Instead of installing models manually, you can use the LocalAI API endpoints and a model definition to install programmatically via API models in runtime.

A curated collection of model files is in the model-gallery. The files of the model gallery are different from the model files used to configure LocalAI models. The model gallery files contains information about the model setup, and the files necessary to run the model locally.

To install for example lunademo, you can send a POST call to the /models/apply endpoint with the model definition url (url) and the name of the model should have in LocalAI (name, optional):

curl --location 'http://localhost:8080/models/apply' \
--header 'Content-Type: application/json' \
--data-raw '{
    "id": "TheBloke/Luna-AI-Llama2-Uncensored-GGML/luna-ai-llama2-uncensored.ggmlv3.q5_K_M.bin",
    "name": "lunademo"
}'

Preloading models during startup

In order to allow the API to start-up with all the needed model on the first-start, the model gallery files can be used during startup.

PRELOAD_MODELS='[{"url": "https://raw.githubusercontent.com/go-skynet/model-gallery/main/gpt4all-j.yaml","name": "gpt4all-j"}]' local-ai

PRELOAD_MODELS (or --preload-models) takes a list in JSON with the same parameter of the API calls of the /models/apply endpoint.

Similarly it can be specified a path to a YAML configuration file containing a list of models with PRELOAD_MODELS_CONFIG ( or --preload-models-config ):

- url: https://raw.githubusercontent.com/go-skynet/model-gallery/main/gpt4all-j.yaml
  name: gpt4all-j

Automatic prompt caching

LocalAI can automatically cache prompts for faster loading of the prompt. This can be useful if your model need a prompt template with prefixed text in the prompt before the input.

To enable prompt caching, you can control the settings in the model config YAML file:

prompt_cache_path: "cache"
prompt_cache_all: true

prompt_cache_path is relative to the models folder. you can enter here a name for the file that will be automatically create during the first load if prompt_cache_all is set to true.

Configuring a specific backend for the model

By default LocalAI will try to autoload the model by trying all the backends. This might work for most of models, but some of the backends are NOT configured to autoload.

The available backends are listed in the model compatibility table.

In order to specify a backend for your models, create a model config file in your models directory specifying the backend:

name: gpt-3.5-turbo

parameters:
  # Relative to the models path
  model: ...

backend: llama-cpp

Connect external backends

LocalAI backends are internally implemented using gRPC services. This also allows LocalAI to connect to external gRPC services on start and extend LocalAI functionalities via third-party binaries.

The --external-grpc-backends parameter in the CLI can be used either to specify a local backend (a file) or a remote URL. The syntax is <BACKEND_NAME>:<BACKEND_URI>. Once LocalAI is started with it, the new backend name will be available for all the API endpoints.

So for instance, to register a new backend which is a local file:

./local-ai --debug --external-grpc-backends "my-awesome-backend:/path/to/my/backend.py"

Or a remote URI:

./local-ai --debug --external-grpc-backends "my-awesome-backend:host:port"

For example, to start vllm manually after compiling LocalAI (also assuming running the command from the root of the repository):

./local-ai --external-grpc-backends "vllm:$PWD/backend/python/vllm/run.sh"

Note that first is is necessary to create the environment with:

make -C backend/python/vllm

Environment variables

When LocalAI runs in a container, there are additional environment variables available that modify the behavior of LocalAI on startup:

Here is how to configure these variables:

docker run --env REBUILD=true localai
docker run --env-file .env localai

CLI Parameters

For a complete reference of all CLI parameters, environment variables, and command-line options, see the CLI Reference page.

You can control LocalAI with command line arguments to specify a binding address, number of threads, model paths, and many other options. Any command line parameter can be specified via an environment variable.

.env files

Any settings being provided by an Environment Variable can also be provided from within .env files. There are several locations that will be checked for relevant .env files. In order of precedence they are:

• .env within the current directory
• localai.env within the current directory
• localai.env within the home directory
• .config/localai.env within the home directory
• /etc/localai.env

Environment variables within files earlier in the list will take precedence over environment variables defined in files later in the list.

An example .env file is:

LOCALAI_THREADS=10
LOCALAI_MODELS_PATH=/mnt/storage/localai/models
LOCALAI_F16=true

Request headers

You can use ‘Extra-Usage’ request header key presence (‘Extra-Usage: true’) to receive inference timings in milliseconds extending default OpenAI response model in the usage field:

...
{
  "id": "...",
  "created": ...,
  "model": "...",
  "choices": [
    {
      ...
    },
    ...
  ],
  "object": "...",
  "usage": {
    "prompt_tokens": ...,
    "completion_tokens": ...,
    "total_tokens": ...,
    // Extra-Usage header key will include these two float fields:
    "timing_prompt_processing: ...,
    "timing_token_generation": ...,
  },
}
...

Extra backends

LocalAI can be extended with extra backends. The backends are implemented as gRPC services and can be written in any language. See the backend section for more details on how to install and build new backends for LocalAI.

In runtime

When using the -core container image it is possible to prepare the python backends you are interested into by using the EXTRA_BACKENDS variable, for instance:

docker run --env EXTRA_BACKENDS="backend/python/diffusers" quay.io/go-skynet/local-ai:master

Concurrent requests

LocalAI supports parallel requests for the backends that supports it. For instance, vLLM and llama.cpp supports parallel requests, and thus LocalAI allows to run multiple requests in parallel.

In order to enable parallel requests, you have to pass --parallel-requests or set the PARALLEL_REQUEST to true as environment variable.

A list of the environment variable that tweaks parallelism is the following:

### Python backends GRPC max workers
### Default number of workers for GRPC Python backends.
### This actually controls whether a backend can process multiple requests or not.

### Define the number of parallel LLAMA.cpp workers (Defaults to 1)

### Enable to run parallel requests

Note that, for llama.cpp you need to set accordingly LLAMACPP_PARALLEL to the number of parallel processes your GPU/CPU can handle. For python-based backends (like vLLM) you can set PYTHON_GRPC_MAX_WORKERS to the number of parallel requests.

VRAM and Memory Management

For detailed information on managing VRAM when running multiple models, see the dedicated VRAM and Memory Management page.

Disable CPU flagset auto detection in llama.cpp

LocalAI will automatically discover the CPU flagset available in your host and will use the most optimized version of the backends.

If you want to disable this behavior, you can set DISABLE_AUTODETECT to true in the environment variables.

VRAM and Memory Management

When running multiple models in LocalAI, especially on systems with limited GPU memory (VRAM), you may encounter situations where loading a new model fails because there isn’t enough available VRAM. LocalAI provides several mechanisms to automatically manage model memory allocation and prevent VRAM exhaustion:

1. Max Active Backends (LRU Eviction): Limit the number of loaded models, evicting the least recently used when the limit is reached
2. Concurrency Groups: Per-model anti-affinity rules that prevent specific models from coexisting on the same node
3. Watchdog Mechanisms: Automatically unload idle or stuck models based on configurable timeouts

The Problem

By default, LocalAI keeps models loaded in memory once they’re first used. This means:

• If you load a large model that uses most of your VRAM, subsequent requests for other models may fail
• Models remain in memory even when not actively being used
• There’s no automatic mechanism to unload models to make room for new ones, unless done manually via the web interface

This is a common issue when working with GPU-accelerated models, as VRAM is typically more limited than system RAM. For more context, see issues #6068, #7269, and #5352.

Solution 1: Max Active Backends (LRU Eviction)

LocalAI supports limiting the maximum number of active backends (loaded models) using LRU (Least Recently Used) eviction. When the limit is reached and a new model needs to be loaded, the least recently used model is automatically unloaded to make room.

Configuration

Set the maximum number of active backends using CLI flags or environment variables:

# Allow up to 3 models loaded simultaneously
./local-ai --max-active-backends=3

# Using environment variables
LOCALAI_MAX_ACTIVE_BACKENDS=3 ./local-ai
MAX_ACTIVE_BACKENDS=3 ./local-ai

Setting the limit to 1 is equivalent to single active backend mode (see below). Setting to 0 disables the limit (unlimited backends).

Use cases

• Systems with limited VRAM that can handle a few models simultaneously
• Multi-model deployments where you want to keep frequently-used models loaded
• Balancing between memory usage and model reload times
• Production environments requiring predictable memory consumption

How it works

1. When a model is requested, its “last used” timestamp is updated
2. When a new model needs to be loaded and the limit is reached, LocalAI identifies the least recently used model(s)
3. The LRU model(s) are automatically unloaded to make room for the new model
4. Concurrent requests for loading different models are handled safely - the system accounts for models currently being loaded when calculating evictions

Eviction Behavior with Active Requests

By default, LocalAI will skip evicting models that have active API calls to prevent interrupting ongoing requests. This means:

• If all models are busy (have active requests), eviction will be skipped and the system will wait for models to become idle
• The loading request will retry eviction with configurable retry settings
• This ensures data integrity and prevents request failures

You can configure this behavior via WebUI or using the following settings:

Force Eviction When Busy

To allow evicting models even when they have active API calls (not recommended for production):

# Via CLI
./local-ai --force-eviction-when-busy

# Via environment variable
LOCALAI_FORCE_EVICTION_WHEN_BUSY=true ./local-ai

Warning: Enabling force eviction can interrupt active requests and cause errors. Only use this if you understand the implications.

LRU Eviction Retry Settings

When models are busy and cannot be evicted, LocalAI will retry eviction with configurable settings:

# Configure maximum retries (default: 30)
./local-ai --lru-eviction-max-retries=50

# Configure retry interval (default: 1s)
./local-ai --lru-eviction-retry-interval=2s

# Using environment variables
LOCALAI_LRU_EVICTION_MAX_RETRIES=50 \
LOCALAI_LRU_EVICTION_RETRY_INTERVAL=2s \
./local-ai

These settings control how long the system will wait for busy models to become idle before giving up. The retry mechanism allows busy models to complete their requests before being evicted, preventing request failures.

Example

# Allow 2 active backends
LOCALAI_MAX_ACTIVE_BACKENDS=2 ./local-ai

# First request - model-a is loaded (1 active)
curl http://localhost:8080/v1/chat/completions -d '{"model": "model-a", ...}'

# Second request - model-b is loaded (2 active, at limit)
curl http://localhost:8080/v1/chat/completions -d '{"model": "model-b", ...}'

# Third request - model-a is evicted (LRU), model-c is loaded
curl http://localhost:8080/v1/chat/completions -d '{"model": "model-c", ...}'

# Request for model-b updates its "last used" time
curl http://localhost:8080/v1/chat/completions -d '{"model": "model-b", ...}'

Single Active Backend Mode

The simplest approach is to ensure only one model is loaded at a time. This is now implemented as --max-active-backends=1. When a new model is requested, LocalAI will automatically unload the currently active model before loading the new one.

# These are equivalent:
./local-ai --max-active-backends=1
./local-ai --single-active-backend

# Using environment variables
LOCALAI_MAX_ACTIVE_BACKENDS=1 ./local-ai
LOCALAI_SINGLE_ACTIVE_BACKEND=true ./local-ai

Note: The --single-active-backend flag is deprecated but still supported for backward compatibility. It is recommended to use --max-active-backends=1 instead.

Single backend use cases

• Single GPU systems with very limited VRAM
• When you only need one model active at a time
• Simple deployments where model switching is acceptable

Solution 1b: Concurrency Groups (per-model anti-affinity)

--max-active-backends is a global count - three loaded models is fine, but it doesn’t know that two of them are 120B and shouldn’t share a GPU. Concurrency groups give per-model rules: any two models that share a group name are mutually exclusive on the same node. Loading one evicts the others. Models with no groups behave exactly as before.

This addresses issue #9659:

allow my zed prediction model to run alongside anything but don’t allow my two 120b models to run alongside each other

Configuration

Declare groups per model in the YAML config - no CLI flag, no env var:

# llama-120b-a.yaml
name: llama-120b-a
backend: llama-cpp
parameters:
  model: llama-120b-a.gguf
concurrency_groups: ["vram-heavy"]

# llama-120b-b.yaml
name: llama-120b-b
backend: llama-cpp
parameters:
  model: llama-120b-b.gguf
concurrency_groups: ["vram-heavy"]

# zed-predict.yaml - no groups, runs alongside anything
name: zed-predict
backend: llama-cpp
parameters:
  model: zed-predict.gguf

With this configuration:

1. Request zed-predict → loads.
2. Request llama-120b-a → loads alongside zed-predict.
3. Request llama-120b-b → llama-120b-a is evicted (shared group vram-heavy); zed-predict stays loaded.

A model can declare multiple groups; two models conflict if they share any group name. Group names are arbitrary strings - pick names that make sense for your hardware (vram-heavy, gpu-1, large-context, …).

Interaction with other knobs

• --max-active-backends: groups are checked before the LRU cap. Group evictions may already make room; LRU then enforces the global count.
• pinned: true: a pinned model is never evicted, including by a group conflict. The new request is loaded with a warning logged - pinning two models in the same group is a configuration mismatch.
• --force-eviction-when-busy: same retry semantics as LRU. A busy conflict is skipped and retried (--lru-eviction-max-retries, --lru-eviction-retry-interval); after retries exhaust, the load proceeds with a warning.

Distributed mode

concurrency_groups is enforced per node, not cluster-wide - VRAM is a node-local resource, so two heavy models on different nodes is fine. The distributed scheduler additionally uses the rule as a placement hint: when choosing where to load a new model, it prefers nodes that don’t already host a same-group model, falling back to eviction only if every candidate has a conflict.

concurrency_groups composes with NodeSelector (which decides which nodes a model is eligible for) - the two filters apply in sequence. Use NodeSelector to target hardware classes; use concurrency_groups to keep specific models from co-residing on whichever node hosts them.

Solution 2: Watchdog Mechanisms

For more flexible memory management, LocalAI provides watchdog mechanisms that automatically unload models based on their activity state. This allows multiple models to be loaded simultaneously, but automatically frees memory when models become inactive or stuck.

Note: Watchdog settings can be configured via the Runtime Settings web interface, which allows you to adjust settings without restarting the application.

Idle Watchdog

The idle watchdog monitors models that haven’t been used for a specified period and automatically unloads them to free VRAM.

Configuration

Via environment variables or CLI:

LOCALAI_WATCHDOG_IDLE=true ./local-ai

LOCALAI_WATCHDOG_IDLE=true LOCALAI_WATCHDOG_IDLE_TIMEOUT=10m ./local-ai

./local-ai --enable-watchdog-idle --watchdog-idle-timeout=10m

Via web UI: Navigate to Settings → Watchdog Settings and enable “Watchdog Idle Enabled” with your desired timeout.

Busy Watchdog

The busy watchdog monitors models that have been processing requests for an unusually long time and terminates them if they exceed a threshold. This is useful for detecting and recovering from stuck or hung backends.

For parallel backends, the timeout is measured from the start of the oldest request that is still in flight. Sustained overlapping traffic does not by itself make a backend stale: when the oldest request completes, the watchdog continues from the start time of the next-oldest request.

Configuration

Via environment variables or CLI:

LOCALAI_WATCHDOG_BUSY=true ./local-ai

LOCALAI_WATCHDOG_BUSY=true LOCALAI_WATCHDOG_BUSY_TIMEOUT=10m ./local-ai

./local-ai --enable-watchdog-busy --watchdog-busy-timeout=10m

Via web UI: Navigate to Settings → Watchdog Settings and enable “Watchdog Busy Enabled” with your desired timeout.

Backend shutdown behavior

Administrative and API-triggered model shutdown is graceful by default. It waits up to 30 seconds for in-flight requests to finish and returns a model is still busy error if the deadline expires. This bounded wait keeps a busy backend from blocking lifecycle operations for other models.

Set LOCALAI_FORCE_BACKEND_SHUTDOWN=true when starting LocalAI to escalate a timed-out graceful shutdown to forced process termination. Forced shutdown can interrupt active requests and skips the backend Free() RPC; terminating the process releases its resources instead.

Combined Configuration

You can enable both watchdogs simultaneously for comprehensive memory management:

LOCALAI_WATCHDOG_IDLE=true \
LOCALAI_WATCHDOG_IDLE_TIMEOUT=15m \
LOCALAI_WATCHDOG_BUSY=true \
LOCALAI_WATCHDOG_BUSY_TIMEOUT=5m \
./local-ai

Or using command line flags:

./local-ai \
  --enable-watchdog-idle --watchdog-idle-timeout=15m \
  --enable-watchdog-busy --watchdog-busy-timeout=5m

Use cases

• Multi-model deployments where different models may be used intermittently
• Systems where you want to keep frequently-used models loaded but free memory from unused ones
• Recovery from stuck or hung backend processes
• Production environments requiring automatic resource management

Example

LOCALAI_WATCHDOG_IDLE=true \
LOCALAI_WATCHDOG_IDLE_TIMEOUT=10m \
LOCALAI_WATCHDOG_BUSY=true \
LOCALAI_WATCHDOG_BUSY_TIMEOUT=5m \
./local-ai

curl http://localhost:8080/v1/chat/completions -d '{"model": "model-a", ...}'
curl http://localhost:8080/v1/chat/completions -d '{"model": "model-b", ...}'

Timeout Format

Timeouts can be specified using Go’s duration format:

• 15m - 15 minutes
• 1h - 1 hour
• 30s - 30 seconds
• 2h30m - 2 hours and 30 minutes

Combining LRU and Watchdog

You can combine Max Active Backends (LRU eviction) with the watchdog mechanisms for comprehensive memory management:

# Allow up to 3 active backends with idle watchdog
LOCALAI_MAX_ACTIVE_BACKENDS=3 \
LOCALAI_WATCHDOG_IDLE=true \
LOCALAI_WATCHDOG_IDLE_TIMEOUT=15m \
./local-ai

Or using command line flags:

./local-ai \
  --max-active-backends=3 \
  --enable-watchdog-idle --watchdog-idle-timeout=15m

This configuration:

• Ensures no more than 3 models are loaded at once (LRU eviction kicks in when exceeded)
• Automatically unloads any model that hasn’t been used for 15 minutes
• Provides both hard limits and time-based cleanup

Example with Retry Settings

You can also configure retry behavior when models are busy:

# Allow up to 2 active backends with custom retry settings
LOCALAI_MAX_ACTIVE_BACKENDS=2 \
LOCALAI_LRU_EVICTION_MAX_RETRIES=50 \
LOCALAI_LRU_EVICTION_RETRY_INTERVAL=2s \
./local-ai

Or using command line flags:

./local-ai \
  --max-active-backends=2 \
  --lru-eviction-max-retries=50 \
  --lru-eviction-retry-interval=2s

This configuration:

• Limits to 2 active backends
• Will retry eviction up to 50 times if models are busy
• Waits 2 seconds between retry attempts
• Ensures busy models have time to complete their requests before eviction

VRAM Budget (Allocation Ceiling)

By default LocalAI treats all detected GPU memory as available for model allocation. On a shared machine (a desktop you also game on, a box that runs other GPU workloads, or a multi-tenant server) you often want to reserve some headroom and let LocalAI use only part of the card. The VRAM budget sets a hard ceiling on how much VRAM LocalAI will consider allocatable.

Configuration

Set the budget with a CLI flag or environment variable:

# Percentage of detected VRAM
./local-ai --vram-budget=80%

# Absolute amount (GB, GiB, MB, or raw bytes)
./local-ai --vram-budget=12GB

# Using environment variables
LOCALAI_VRAM_BUDGET=80% ./local-ai
LOCALAI_VRAM_BUDGET=12GB ./local-ai

Accepted formats:

• A percentage such as 80% (or the decimal 0.8).
• An absolute amount such as 12GB, 12GiB, 12000MB, or a raw byte count.

Leaving it empty or unset (the default) uses all detected VRAM, which is the historical behavior.

Semantics

The budget is a hard ceiling, never a floor:

• Everywhere LocalAI reads VRAM to decide model allocation (hardware defaults for batch size and parallelism, context auto-fit, GGUF fit warnings, and the watchdog), it uses min(detected VRAM, budget).
• The budget can only lower usable VRAM, never raise it above what the hardware physically has. An absolute value larger than physical VRAM is clamped to physical.
• A percentage above 100% is rejected as invalid.

Editing from the UI

The budget is exposed in the Settings page under the Performance section as VRAM Budget, so you can change it without editing flags or restarting from scratch. It is live-editable and persists to the runtime settings.

Distributed mode

The same LOCALAI_VRAM_BUDGET / --vram-budget applies to distributed worker nodes (local-ai worker ...), where it caps the VRAM the scheduler will use for per-node placement. Admins can also set a per-node budget live from the node UI (or the admin API); see Distributed Mode.

Limitations and Considerations

VRAM Usage Estimation

LocalAI cannot reliably estimate VRAM usage of new models to load across different backends (llama.cpp, vLLM, diffusers, etc.) because:

• Different backends report memory usage differently
• VRAM requirements vary based on model architecture, quantization, and configuration
• Some backends may not expose memory usage information before loading the model

Manual Management

If automatic management doesn’t meet your needs, you can manually stop models using the LocalAI management API:

curl -X POST http://localhost:8080/backend/shutdown \
  -H "Content-Type: application/json" \
  -d '{"model": "model-name"}'

To stop all models, you’ll need to call the endpoint for each loaded model individually, or use the web UI to stop all models at once.

Conversely, you can pre-load a model into memory ahead of its first request with POST /backend/load (the inverse of shutdown) - see Backend Monitor.

Best Practices

1. Monitor VRAM usage: Use nvidia-smi (for NVIDIA GPUs) or similar tools to monitor actual VRAM usage
2. Set an appropriate backend limit: For single-GPU systems, --max-active-backends=1 is often the simplest solution. For systems with more VRAM, you can increase the limit to keep more models loaded
3. Combine LRU with watchdog: Use --max-active-backends to limit the number of loaded models, and enable idle watchdog to unload models that haven’t been used recently
4. Tune watchdog timeouts: Adjust timeouts based on your usage patterns - shorter timeouts free memory faster but may cause more frequent reloads
5. Consider model size: Ensure your VRAM can accommodate at least one of your largest models
6. Use quantization: Smaller quantized models use less VRAM and allow more flexibility

Related Documentation

• See Advanced Usage for other configuration options
• See GPU Acceleration for GPU setup and configuration
• See Backend Flags for all available backend configuration options

Model Configuration

LocalAI uses YAML configuration files to define model parameters, templates, and behavior. This page provides a complete reference for all available configuration options.

Overview

Model configuration files allow you to:

• Define default parameters (temperature, top_p, etc.)
• Configure prompt templates
• Specify backend settings
• Set up function calling
• Configure GPU and memory options
• And much more

Configuration File Locations

You can create model configuration files in several ways:

1. Individual YAML files in the models directory (e.g., models/gpt-3.5-turbo.yaml)
2. Single config file with multiple models using --models-config-file or LOCALAI_MODELS_CONFIG_FILE
3. Remote URLs - specify a URL to a YAML configuration file at startup

Example: Basic Configuration

name: gpt-3.5-turbo
parameters:
  model: luna-ai-llama2-uncensored.ggmlv3.q5_K_M.bin
  temperature: 0.3

context_size: 512
threads: 10
backend: llama-cpp

template:
  completion: completion
  chat: chat

Example: Multiple Models in One File

When using --models-config-file, you can define multiple models as a list:

- name: model1
  parameters:
    model: model1.bin
  context_size: 512
  backend: llama-cpp

- name: model2
  parameters:
    model: model2.bin
  context_size: 1024
  backend: llama-cpp

Core Configuration Fields

Basic Model Settings

Model File and Downloads

Example:

parameters:
  model: my-model.gguf

download_files:
  - filename: my-model.gguf
    uri: https://example.com/model.gguf
    sha256: abc123...

Model artifacts

The artifacts section makes installation of a Hugging Face model eager and repeatable. LocalAI resolves the requested revision to an immutable commit, downloads the selected repository files, and commits the complete snapshot before the model installation succeeds.

artifacts:
  - name: model
    target: model
    source:
      type: huggingface
      repo: Qwen/Qwen3-ASR-1.7B
      revision: main
      token_env: HF_TOKEN
    resolved:
      endpoint: https://huggingface.co
      revision: 0123456789abcdef0123456789abcdef01234567
      cache_key: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

parameters:
  model: Qwen/Qwen3-ASR-1.7B

Declare source when authoring a configuration. LocalAI owns the resolved block and writes it after installation; do not choose its values manually. For a public repository, omit token_env. For a private or gated repository, set it to HF_TOKEN and provide that environment variable to the LocalAI controller.

Managed installation finishes only after every selected file is committed locally. parameters.model remains the logical repository ID. Once resolved.cache_key is present, LocalAI derives .artifacts/huggingface/<cache-key>/snapshot as the runtime ModelFile. Configurations without artifacts keep the existing lazy repository-ID behavior.

The initially migrated backend families are transformers and its aliases, diffusers, qwen-asr, fish-speech, nemo, voxcpm, qwen-tts, liquid-audio, vllm, vllm-omni, and sglang. Automatic imports add artifact declarations only for this set. Compatible external backends may opt in by declaring the artifact explicitly.

Parameters Section

The parameters section contains all OpenAI-compatible request parameters and model-specific options.

OpenAI-Compatible Parameters

These settings will be used as defaults for all the API calls to the model.

Language and Translation

Custom Parameters

LLM Configuration

These settings apply to most LLM backends (llama.cpp, vLLM, etc.):

Performance Settings

Memory Management

GPU Configuration

Sampling and Generation

LoRA Configuration

Advanced Options

YARN Configuration

YARN (Yet Another RoPE extensioN) settings for context extension:

Speculative Decoding

Speculative decoding speeds up text generation by predicting multiple tokens ahead and verifying them in a single forward pass. The output is identical to normal decoding - only faster. This feature is only available with the llama-cpp backend.

There are two approaches:

Draft Model Speculative Decoding

Uses a smaller, faster model from the same model family to draft candidate tokens, which the main model then verifies. Requires a separate GGUF file for the draft model.

name: my-model
backend: llama-cpp
parameters:
  model: large-model.gguf
draft_model: small-draft-model.gguf
n_draft: 8
options:
  - spec_p_min:0.8
  - draft_gpu_layers:99

N-gram Self-Speculative Decoding

Uses patterns from the token history to predict future tokens - no extra model required. Works well for repetitive or structured output (code, JSON, lists).

name: my-model
backend: llama-cpp
parameters:
  model: my-model.gguf
options:
  - spec_type:ngram_simple
  - spec_n_max:16

Speculative Decoding Options

These are set via the options: array in the model configuration (format: key:value):

Common options

Draft-model options (apply when spec_type=draft, i.e. a draft_model is configured)

ngram_simple options (used when spec_type includes ngram_simple)

ngram_mod options (used when spec_type includes ngram_mod)

ngram_map_k options (used when spec_type includes ngram_map_k)

ngram_map_k4v options (used when spec_type includes ngram_map_k4v)

ngram_cache lookup files

Speculative Type Values

The canonical names match upstream llama.cpp (dash-separated). For backward compatibility LocalAI also accepts the underscore-separated forms and the bare draft / eagle3 aliases.

Multiple types can be chained by passing a comma-separated list to spec_type (e.g. spec_type:ngram-simple,ngram-mod). The runtime tries them in order and accepts the first proposal that meets the acceptance criteria.

Multi-Token Prediction (MTP)

draft-mtp enables Multi-Token Prediction (ggml-org/llama.cpp#22673). MTP uses a small prediction head trained into the target model: the head runs alongside the main forward pass and proposes the next few tokens, which the target then verifies in a single batched step. Upstream reports ~1.85x-2.1x token throughput at ~72-82% draft acceptance on Qwen3.6 27B / 35B A3B.

Auto-detection (default). When a GGUF declares an MTP head (the upstream <arch>.nextn_predict_layers metadata key, set by convert_hf_to_gguf.py for Qwen3.5/3.6 family models and similar), LocalAI auto-enables MTP with the following defaults:

options:
  - spec_type:draft-mtp
  - spec_n_max:6
  - spec_p_min:0.75

Detection runs both at import time (the /import-model UI / POST /models/import-uri flow range-fetches the GGUF header and writes the options into the generated YAML before you save it) and at load time (every llama-cpp model start re-checks the local header and appends the options if spec_type isn’t already set). To opt out, set an explicit spec_type: / speculative_type: in your YAML - auto-detection always preserves the user value, including spec_type:none.

Two ways to load the MTP head:

1. Embedded in the target GGUF (the recommended path for LocalAI, and what auto-detection assumes). When spec_type includes draft-mtp and draft_model is empty, the backend builds the MTP draft context directly from the target model’s weights. The GGUF must have been converted with the MTP tensors included.
2. Separate mtp-*.gguf sibling file. If you point draft_model at the separate MTP-head GGUF that ships next to the main weights on HuggingFace, the backend will load it as a draft model. Note: upstream’s -hf auto-discovery of mtp-*.gguf siblings is not wired into LocalAI’s gRPC layer - you need to download the sibling file and configure draft_model explicitly.

Manual override knobs (overlap with the auto-detect defaults above):

Minimal config (override-only, since auto-detection already covers this for MTP-capable GGUFs):

name: qwen3-mtp
backend: llama-cpp
parameters:
  model: qwen3-27b-with-mtp.gguf
options:
  - spec_type:draft-mtp
  - spec_n_max:3

With a separate MTP head file:

name: qwen3-mtp
backend: llama-cpp
parameters:
  model: qwen3-27b.gguf
  draft_model: qwen3-27b-mtp-head.gguf
options:
  - spec_type:draft-mtp
  - spec_n_max:3

Chaining MTP with n-gram fallback (experimental, from the PR’s usage notes - useful when MTP acceptance drops on highly repetitive output):

options:
  - spec_type:draft-mtp,ngram-mod
  - spec_n_max:3
  - spec_ngram_mod_n_match:24

Pre-converted GGUFs with MTP heads are published on the ggml-org HuggingFace org (initially Qwen3.6 27B and Qwen3.6 35B A3B).

Reasoning Models (DeepSeek-R1, Qwen3, etc.)

These load-time options control how the backend parses <think> reasoning blocks and how much budget the model is allowed for thinking. They are set per model via the options: array. For how reasoning is returned alongside tool calls and survives the tool-result round trip, see Interleaved Thinking with Tool Calls.

reasoning_effort as a chat-template kwarg

reasoning_effort is also forwarded to the backend as a chat_template_kwarg, so models whose jinja chat template keys on it - e.g. gpt-oss (Harmony) or LFM2.5 - honor the level, not just the on/off enable_thinking flag. This matters for models that ignore enable_thinking entirely (LFM2.5 keeps emitting <think> for enable_thinking=false, but respects reasoning_effort).

Set a per-model default in the config so every request inherits it (a per-request reasoning_effort still overrides):

name: my-model
reasoning_effort: none   # none | minimal | low | medium | high

For realtime pipelines, set it on the pipeline so it applies to the pipeline’s LLM without editing that model’s own config:

name: gpt-realtime
pipeline:
  llm: lfm2.5
  reasoning_effort: none   # overrides the LLM model's own reasoning_effort

Custom chat_template_kwargs

Some jinja chat templates expose extra variables beyond enable_thinking / reasoning_effort (for example Qwen3’s preserve_thinking). Set arbitrary key/values in the model config and they are forwarded to the backend’s chat_template_kwargs as-is, so you don’t need a dedicated server option per template variable:

name: qwen3
chat_template_kwargs:
  preserve_thinking: true

You can also override (or add) any of these per request through the OpenAI metadata field on /v1/chat/completions. Values are strings; "true" / "false" are coerced to booleans, anything else is passed through as a string:

{
  "model": "qwen3",
  "messages": [{"role": "user", "content": "hi"}],
  "metadata": { "preserve_thinking": "true", "enable_thinking": "false" }
}

Per-request metadata overrides the model config defaults and the reasoning-config levers, and (for enable_thinking / reasoning_effort) takes effect across every backend that reads them, not just llama.cpp. Typed (non-boolean) values are only supported through the model YAML chat_template_kwargs, where YAML preserves the type.

Multimodal Backend Options

Embedding & Reranking Backend Options

Other Backend Tuning Options

These llama.cpp options are passed through the options: array.

Generic option passthrough

Any options: entry whose name starts with - is forwarded verbatim to upstream llama.cpp’s own llama-server argument parser. This means any flag the bundled llama.cpp supports works without LocalAI needing a dedicated option, even ones added after your LocalAI version was built. See the upstream server flags reference.

Format mirrors the rest of the array - --flag for a boolean, or --flag:value for a flag that takes a value. Everything after the first : is the value, so embedded colons (e.g. host:port) are preserved:

options:
  - "--cpu-moe"                 # boolean flag
  - "--n-cpu-moe:4"             # flag with a value
  - "--override-tensor:exps=CPU"
  - "devices:CUDA1,CUDA2,CUDA3" # skip CUDA0, e.g. a display GPU

Notes:

• Precedence: passthrough flags are applied last, so an explicit flag overrides the LocalAI option it maps to (e.g. --ctx-size:8192 overrides context_size).
• Power-user territory: an invalid flag or value is rejected by the upstream parser exactly as it would be by llama-server, which can fail model loading. Prefer the named options above when one exists.
• Flags that would terminate the process (such as --help, --usage, --version, --license, --list-devices, --cache-list, and --completion*) are ignored.

Prompt Caching

The recommended way to enable prompt caching for the llama-cpp backend is the server-side prompt cache controlled by cache_ram / kv_unified / cache_idle_slots in the options: array (see llama.cpp backend options). It’s on by default since LocalAI v4.3 and is what gives repeated system prompts a near-zero prefill on the second call.

The fields below come from upstream llama.cpp’s CLI completion tool and are passed through to the gRPC backend for compatibility, but the gRPC server itself does not consume them: keep them empty unless you’re targeting a non-llama-cpp backend that reads them.

Text Processing

System Prompt

vLLM-Specific Configuration

These options apply when using the vllm backend:

Template Configuration

Templates use Go templates with Sprig functions.

Template Variables

Templating supports sprig functions.

Following are common variables available in templates:

• {{.Input}} - User input
• {{.Instruction}} - Instruction for edit operations
• {{.System}} - System message
• {{.Prompt}} - Full prompt
• {{.Functions}} - Function definitions (for function calling)
• {{.FunctionCall}} - Function call result

Example Template

template:
  chat: |
    {{.System}}
    {{range .Messages}}
    {{if eq .Role "user"}}User: {{.Content}}{{end}}
    {{if eq .Role "assistant"}}Assistant: {{.Content}}{{end}}
    {{end}}
    Assistant:

Function Calling Configuration

Configure how the model handles function/tool calls:

Grammar Configuration

Diffusers Configuration

For image generation models using the diffusers backend:

TTS Configuration

For text-to-speech models:

For example, a custom-named model on a known cloning backend can declare support explicitly while retaining a model-wide reference fallback:

name: private-voice-model
backend: qwen3-tts-cpp
parameters:
  model: private/qwen-talker-base.gguf
known_usecases:
  - tts
tts:
  voice_cloning: true
  audio_path: voices/default-reference.wav

tts.voice_cloning: true only overrides model-variant detection. It cannot enable cloning on a backend that does not implement LocalAI’s reference-audio contract.

Roles Configuration

Map conversation roles to specific strings:

roles:
  user: "### Instruction:"
  assistant: "### Response:"
  system: "### System Instruction:"

Feature Flags

Enable or disable experimental features:

feature_flags:
  feature_name: true
  another_feature: false

MCP Configuration

Model Context Protocol (MCP) configuration:

Agent Configuration

Agent/autonomous agent configuration:

Reasoning Configuration

Configure how reasoning tags are extracted and processed from model output. Reasoning tags are used by models like DeepSeek, Command-R, and others to include internal reasoning steps in their responses.

Reasoning Tag Formats

The reasoning extraction supports multiple tag formats used by different models:

• <thinking>...</thinking> - General thinking tag
• <think>...</think> - DeepSeek, Granite, ExaOne, GLM models
• <|START_THINKING|>...<|END_THINKING|> - Command-R models
• <|inner_prefix|>...<|inner_suffix|> - Apertus models
• <seed:think>...</seed:think> - Seed models
• <|think|>...<|end|><|begin|>assistant<|content|> - Solar Open models
• [THINK]...[/THINK] - Magistral models

Examples

Disable reasoning extraction:

reasoning:
  disable: true

Extract reasoning but don’t prepend tags:

reasoning:
  disable_reasoning_tag_prefill: true

Strip reasoning tags without storing reasoning content:

reasoning:
  strip_reasoning_only: true

Complete example with reasoning configuration:

name: deepseek-model
backend: llama-cpp
parameters:
  model: deepseek.gguf

reasoning:
  disable: false
  disable_reasoning_tag_prefill: false
  strip_reasoning_only: false

Example with custom tokens and tag pairs:

name: custom-reasoning-model
backend: llama-cpp
parameters:
  model: custom.gguf

reasoning:
  thinking_start_tokens:
    - "<custom:think>"
    - "<my:reasoning>"
  tag_pairs:
    - start: "<custom:think>"
      end: "</custom:think>"
    - start: "<my:reasoning>"
      end: "</my:reasoning>"

Note: Custom tokens and tag pairs are checked before the default ones, giving them priority. This allows you to override default behavior or add support for new reasoning tag formats.

Per-Request Override via Metadata

The reasoning.disable setting from model configuration can be overridden on a per-request basis using the metadata field in the OpenAI chat completion request. This allows you to enable or disable thinking for individual requests without changing the model configuration.

The metadata field accepts a map[string]string that is forwarded to the backend. The enable_thinking key controls thinking behavior:

# Enable thinking for a single request (overrides model config)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3",
    "messages": [{"role": "user", "content": "Explain quantum computing"}],
    "metadata": {"enable_thinking": "true"}
  }'

# Disable thinking for a single request (overrides model config)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3",
    "messages": [{"role": "user", "content": "Hello"}],
    "metadata": {"enable_thinking": "false"}
  }'

Priority order:

1. Request-level metadata.enable_thinking (highest priority)
2. Model config reasoning.disable (fallback)
3. Auto-detected from model template (default)

Pipeline Configuration

Define pipelines for audio-to-audio processing and the Realtime API:

gRPC Configuration

Backend gRPC communication settings. These control the readiness handshake between LocalAI and a freshly spawned backend process - LocalAI polls the backend’s Health gRPC method up to grpc.attempts times, sleeping grpc.attempts_sleep_time seconds between polls, before giving up and terminating the backend as unresponsive.

Total load window ≈ grpc.attempts × (grpc.attempts_sleep_time + per-call gRPC dial timeout). The default of 20 × 2 s ≈ 40 s is fine for typical backends but is too short for large models that need substantial time to become gRPC-ready after the process starts - for example NVFP4 / FP8 models whose shard loading and CUDA-graph capture can take several minutes, or slow storage backends. If the backend keeps getting killed while still legitimately loading (visible as exitCode=120 + rpc error: code = Canceled desc = context canceled in the LocalAI log, while the backend’s own stderr shows continued forward progress), raise these values.

Example configuration for a model that needs up to ~10 minutes to become gRPC-ready (large NVFP4 model, cold shard load + CUDA-graph capture):

grpc:
  attempts: 140
  attempts_sleep_time: 5

This gives a ~700 s window while keeping health-check polling frequent enough to detect real backend crashes quickly. The values only affect the initial readiness handshake - inference-request timeouts and the watchdog are unchanged.

Overrides

Override model configuration values at runtime (llama.cpp):

overrides:
  - "qwen3moe.expert_used_count=int:10"
  - "some_key=string:value"

Format: KEY=TYPE:VALUE where TYPE is int, float, string, or bool.

Known Use Cases

Specify which endpoints this model supports:

known_usecases:
  - chat
  - completion
  - embeddings

Available flags: chat, completion, edit, embeddings, rerank, image, transcript, tts, sound_generation, tokenize, vad, video, detection, llm (combination of CHAT, COMPLETION, EDIT).

token_classify marks a model as a token-classification (NER) provider for the PII filter (e.g. an openai-privacy-filter GGUF). Declare it explicitly together with embeddings: true (the classifier loads via TOKEN_CLS pooling). It runs on the dedicated privacy-filter backend (backend/cpp/privacy-filter), a standalone GGML engine for the openai-privacy-filter family - separate from llama-cpp, which no longer carries the token-classification path.

Known input and output modalities

Use known_input_modalities and known_output_modalities when a use case does not fully describe a model’s I/O. For example, both text-to-video and audio-driven avatar models use the video use case, but only the avatar model accepts audio:

known_usecases:
  - video
known_input_modalities:
  - text
  - image
  - audio
known_output_modalities:
  - video

Valid modality values are text, image, audio, and video. Explicit values are combined with modalities LocalAI can infer from the model use cases and configuration. The resulting canonical, de-duplicated lists are exposed by GET /v1/models/capabilities.

PII filtering

PII redaction is NER-based and runs on the request (input) side. It has two halves:

• Detector models are token_classify models that carry the detection policy in a top-level pii_detection: block. The policy is defined once, on the model itself:

  name: privacy-filter-multilingual
  backend: llama-cpp
  embeddings: true
  known_usecases:
    - token_classify
  pii_detection:
    min_score: 0.5            # drop detections below this confidence
    default_action: mask      # mask | block | allow - applied to any detected
                              # group with no explicit entry (empty = mask)
    entity_actions:           # which PII to block vs mask vs allow-log
      PASSWORD: block
      CREDITCARD: block
      EMAIL: mask

• Consuming models opt in and reference one or more detectors by name - no per-consumer policy:

  name: my-assistant
  pii:
    enabled: true             # default: off for local backends, on for cloud-proxy
    detectors:
      - privacy-filter-multilingual

Multiple detectors union their detections; overlapping spans resolve to the strongest action (block > mask > allow). A configured detector that can’t be loaded fails the request closed (HTTP 503) rather than silently skipping the check. Detections are audited at /api/pii/events (hash-prefix only, never the raw value).

The earlier regex pattern tier (pii.patterns, the global pattern catalogue, --pii-config, and the /api/pii/patterns admin endpoints) has been removed, along with response/streaming-side redaction. Those keys now no-op with a startup warning; migrate to pii.detectors + a detector’s pii_detection block.

Complete Example

Here’s a comprehensive example combining many options:

name: my-llm-model
description: A high-performance LLM model
backend: llama-cpp

parameters:
  model: my-model.gguf
  temperature: 0.7
  top_p: 0.9
  top_k: 40
  max_tokens: 2048

context_size: 4096
threads: 8
f16: true
gpu_layers: 35

system_prompt: "You are a helpful AI assistant."

template:
  chat: |
    {{.System}}
    {{range .Messages}}
    {{if eq .Role "user"}}User: {{.Content}}
    {{else if eq .Role "assistant"}}Assistant: {{.Content}}
    {{end}}
    {{end}}
    Assistant:

roles:
  user: "User:"
  assistant: "Assistant:"
  system: "System:"

stopwords:
  - "\n\nUser:"
  - "\n\nHuman:"

prompt_cache_path: "cache/my-model"
prompt_cache_all: true

function:
  grammar:
    parallel_calls: true
    mixed_mode: false

feature_flags:
  experimental_feature: true

Related Documentation

• See Advanced Usage for other configuration options
• See Prompt Templates for template examples
• See CLI Reference for command-line options

GPU Auto-Fit Mode

Note: By default, LocalAI sets gpu_layers to a very large value (9999999), which effectively disables llama-cpp’s auto-fit functionality. This is intentional to work with LocalAI’s VRAM-based model unloading mechanism.

To enable llama-cpp’s auto-fit mode, set gpu_layers: -1 in your model configuration. However, be aware of the following:

1. Trade-off: Enabling auto-fit conflicts with LocalAI’s built-in VRAM threshold-based unloading. Auto-fit attempts to fit all tensors into GPU memory automatically, while LocalAI’s unloading mechanism removes models when VRAM usage exceeds thresholds.

2. Known Issues: Setting gpu_layers: -1 may trigger tensor_buft_override buffer errors in some configurations, particularly when the model exceeds available GPU memory.

3. Recommendation:

   • Use the default settings for most use cases (LocalAI manages VRAM automatically)
   • Only enable gpu_layers: -1 if you understand the implications and have tested on your specific hardware
   • Monitor VRAM usage carefully when using auto-fit mode

This is a known limitation being tracked in issue #8562. A future implementation may provide a runtime toggle or custom logic to reconcile auto-fit with threshold-based unloading.

TLS Reverse Proxy Configuration

TLS Reverse Proxy Configuration

When running LocalAI behind a TLS termination reverse proxy, the Web UI may fail to load static assets (CSS, JS) correctly because the application doesn’t automatically detect that it’s being served over HTTPS. This guide explains how to properly configure your reverse proxy to work with LocalAI.

How It Works

LocalAI uses the X-Forwarded-Proto HTTP header to determine the protocol used by clients. When this header is set to https, LocalAI will generate HTTPS URLs for static assets in the Web UI.

Running behind a reverse proxy (HTTPS / subpath)

LocalAI does not terminate TLS itself, so HTTPS is provided by a reverse proxy in front of it. Self-referential links (generated image and video URLs, async job status URLs, OAuth callbacks) need the externally visible scheme, host and port.

LocalAI determines these in this order:

1. LOCALAI_BASE_URL - if set, it is authoritative for the origin. Set it to the externally visible base URL, e.g. LOCALAI_BASE_URL=https://localai.example.com or https://192.168.0.13:34567. Recommended whenever links come back with the wrong scheme or host.
2. Otherwise, the X-Forwarded-Proto and X-Forwarded-Host headers (or the RFC 7239 Forwarded header) sent by the proxy. Ensure your proxy forwards X-Forwarded-Proto: https.

A reverse-proxy subpath mount is supported via X-Forwarded-Prefix; it is appended to LOCALAI_BASE_URL when both are present.

Required Headers

Your reverse proxy must forward these headers to LocalAI:

HAProxy Configuration

frontend https-in
    bind *:443 ssl crt /path/to/cert.pem
    mode http
    
    # Set the X-Forwarded-Proto header
    http-request set-header X-Forwarded-Proto https
    
    # Pass the original host
    http-request set-header X-Forwarded-Host %[hdr(host)]
    
    # If serving under a sub-path, set the prefix
    # http-request set-header X-Forwarded-Prefix /localai
    
    default_backend localai

backend localai
    mode http
    server localai1 127.0.0.1:8080 check

Apache Configuration

<VirtualHost *:443>
    ServerName your-domain.com
    SSLEngine on
    SSLCertificateFile /path/to/cert.pem
    SSLCertificateKeyFile /path/to/key.pem
    
    # Enable proxy and headers modules
    ProxyRequests Off
    ProxyPreserveHost On
    
    <Proxy *>
        Require all granted
    </Proxy>
    
    # Set the X-Forwarded-Proto header
    RequestHeader set X-Forwarded-Proto "https"
    
    # Set the X-Forwarded-Host header (optional, usually automatic)
    RequestHeader set X-Forwarded-Host "%{HTTP_HOST}s"
    
    # If serving under a sub-path
    # RequestHeader set X-Forwarded-Prefix "/localai"
    
    ProxyPass / http://127.0.0.1:8080/
    ProxyPassReverse / http://127.0.0.1:8080/
</VirtualHost>

Nginx Configuration

server {
    listen 443 ssl;
    server_name your-domain.com;
    
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    
    # Set the X-Forwarded-Proto header
    proxy_set_header X-Forwarded-Proto $scheme;
    
    # Pass the original host
    proxy_set_header X-Forwarded-Host $host;
    
    # If serving under a sub-path
    # proxy_set_header X-Forwarded-Prefix /localai;
    
    # Other proxy settings
    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
}

Serving Under a Sub-Path

If you serve LocalAI under a sub-path (e.g., https://your-domain.com/localai), you need to:

1. Configure your reverse proxy to set the X-Forwarded-Prefix header

Example with Nginx:

proxy_set_header X-Forwarded-Prefix /localai;

Testing Your Configuration

1. Start LocalAI: localai
2. Configure your reverse proxy as shown above
3. Access the Web UI through the proxy
4. Check the browser’s developer console for any mixed content warnings or failed asset loads
5. Verify that the HTML source contains https:// URLs for static assets

Troubleshooting

Static Assets Not Loading

• Verify the X-Forwarded-Proto header is being forwarded
• Check that the header value is exactly https (lowercase)
• Inspect the network tab in your browser to see which requests are failing

Mixed Content Warnings

• Ensure LocalAI is generating HTTPS URLs (check the BaseURL middleware is working)
• Verify the X-Forwarded-Proto header is set before LocalAI processes the request

Redirect Loops

• Check that your proxy is not adding duplicate headers
• Verify X-Forwarded-Proto is not being set to both http and https

Security Note

When using reverse proxies, ensure your proxy only accepts connections from trusted sources and properly validates SSL certificates. Never expose LocalAI directly to the internet without TLS termination.