Model Configuration

LocalAI uses YAML configuration files to define model parameters, templates, and behavior. This page provides a complete reference for all available configuration options.

Overview link

Model configuration files allow you to:

• Define default parameters (temperature, top_p, etc.)
• Configure prompt templates
• Specify backend settings
• Set up function calling
• Configure GPU and memory options
• And much more

Configuration File Locations link

You can create model configuration files in several ways:

1. Individual YAML files in the models directory (e.g., models/gpt-3.5-turbo.yaml)
2. Single config file with multiple models using --models-config-file or LOCALAI_MODELS_CONFIG_FILE
3. Remote URLs - specify a URL to a YAML configuration file at startup

Example: Basic Configuration link

  name: gpt-3.5-turbo
parameters:
  model: luna-ai-llama2-uncensored.ggmlv3.q5_K_M.bin
  temperature: 0.3

context_size: 512
threads: 10
backend: llama-stable

template:
  completion: completion
  chat: chat
  

Example: Multiple Models in One File link

When using --models-config-file, you can define multiple models as a list:

  - name: model1
  parameters:
    model: model1.bin
  context_size: 512
  backend: llama-stable

- name: model2
  parameters:
    model: model2.bin
  context_size: 1024
  backend: llama-stable
  

Core Configuration Fields link

Basic Model Settings link

Model File and Downloads link

Example:

  parameters:
  model: my-model.gguf

download_files:
  - filename: my-model.gguf
    uri: https://example.com/model.gguf
    sha256: abc123...
  

Parameters Section link

The parameters section contains all OpenAI-compatible request parameters and model-specific options.

OpenAI-Compatible Parameters link

These settings will be used as defaults for all the API calls to the model.

Language and Translation link

Custom Parameters link

LLM Configuration link

These settings apply to most LLM backends (llama.cpp, vLLM, etc.):

Performance Settings link

Memory Management link

GPU Configuration link

Sampling and Generation link

LoRA Configuration link

Advanced Options link

YARN Configuration link

YARN (Yet Another RoPE extensioN) settings for context extension:

Prompt Caching link

Text Processing link

System Prompt link

vLLM-Specific Configuration link

These options apply when using the vllm backend:

Template Configuration link

Templates use Go templates with Sprig functions.

Template Variables link

Templating supports sprig functions.

Following are common variables available in templates:

• {{.Input}} - User input
• {{.Instruction}} - Instruction for edit operations
• {{.System}} - System message
• {{.Prompt}} - Full prompt
• {{.Functions}} - Function definitions (for function calling)
• {{.FunctionCall}} - Function call result

Example Template link

  template:
  chat: |
    {{.System}}
    {{range .Messages}}
    {{if eq .Role "user"}}User: {{.Content}}{{end}}
    {{if eq .Role "assistant"}}Assistant: {{.Content}}{{end}}
    {{end}}
    Assistant:
  

Function Calling Configuration link

Configure how the model handles function/tool calls:

Grammar Configuration link

Diffusers Configuration link

For image generation models using the diffusers backend:

TTS Configuration link

For text-to-speech models:

Roles Configuration link

Map conversation roles to specific strings:

  roles:
  user: "### Instruction:"
  assistant: "### Response:"
  system: "### System Instruction:"
  

Feature Flags link

Enable or disable experimental features:

  feature_flags:
  feature_name: true
  another_feature: false
  

MCP Configuration link

Model Context Protocol (MCP) configuration:

Agent Configuration link

Agent/autonomous agent configuration:

Pipeline Configuration link

Define pipelines for audio-to-audio processing:

gRPC Configuration link

Backend gRPC communication settings:

Overrides link

Override model configuration values at runtime (llama.cpp):

  overrides:
  - "qwen3moe.expert_used_count=int:10"
  - "some_key=string:value"
  

Format: KEY=TYPE:VALUE where TYPE is int, float, string, or bool.

Known Use Cases link

Specify which endpoints this model supports:

  known_usecases:
  - chat
  - completion
  - embeddings
  

Available flags: chat, completion, edit, embeddings, rerank, image, transcript, tts, sound_generation, tokenize, vad, video, detection, llm (combination of CHAT, COMPLETION, EDIT).

Complete Example link

Here’s a comprehensive example combining many options:

  name: my-llm-model
description: A high-performance LLM model
backend: llama-stable

parameters:
  model: my-model.gguf
  temperature: 0.7
  top_p: 0.9
  top_k: 40
  max_tokens: 2048

context_size: 4096
threads: 8
f16: true
gpu_layers: 35

system_prompt: "You are a helpful AI assistant."

template:
  chat: |
    {{.System}}
    {{range .Messages}}
    {{if eq .Role "user"}}User: {{.Content}}
    {{else if eq .Role "assistant"}}Assistant: {{.Content}}
    {{end}}
    {{end}}
    Assistant:

roles:
  user: "User:"
  assistant: "Assistant:"
  system: "System:"

stopwords:
  - "\n\nUser:"
  - "\n\nHuman:"

prompt_cache_path: "cache/my-model"
prompt_cache_all: true

function:
  grammar:
    parallel_calls: true
    mixed_mode: false

feature_flags:
  experimental_feature: true
  

Related Documentation link

• See Advanced Usage for other configuration options
• See Prompt Templates for template examples
• See CLI Reference for command-line options