Text Generation (GPT)

LocalAI supports generating text with GPT with llama.cpp and other backends (such as rwkv.cpp as ) see also the Model compatibility for an up-to-date list of the supported model families.

Note:

• You can also specify the model name as part of the OpenAI token.
• If only one model is available, the API will use it for all the requests.

API Reference

Chat completions

https://platform.openai.com/docs/api-reference/chat

For example, to generate a chat completion, you can send a POST request to the /v1/chat/completions endpoint with the instruction as the request body:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "messages": [{"role": "user", "content": "Say this is a test!"}],
  "temperature": 0.7
}'

Available additional parameters: top_p, top_k, max_tokens

When a request omits max_tokens, the vLLM and SGLang backends do not cap generation at 200 tokens. Generation is bounded only by the context window of the model.

Reasoning models return their thinking in the reasoning field. When a model reasons and calls a tool in the same turn, see Interleaved Thinking with Tool Calls.

When stream: true is set and the llama.cpp backend fails before the first chunk, for example because the prompt exceeds the context size, the request fails with an HTTP error. The error message is not streamed as assistant content. An error after streaming has started is reported inside the stream.

Edit completions

https://platform.openai.com/docs/api-reference/edits

To generate an edit completion you can send a POST request to the /v1/edits endpoint with the instruction as the request body:

curl http://localhost:8080/v1/edits -H "Content-Type: application/json" -d '{
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "instruction": "rephrase",
  "input": "Black cat jumped out of the window",
  "temperature": 0.7
}'

Available additional parameters: top_p, top_k, max_tokens.

Completions

https://platform.openai.com/docs/api-reference/completions

To generate a completion, you can send a POST request to the /v1/completions endpoint with the instruction as per the request body:

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "prompt": "A long time ago in a galaxy far, far away",
  "temperature": 0.7
}'

Available additional parameters: top_p, top_k, max_tokens

List models

You can list all the models available with:

curl http://localhost:8080/v1/models

Anthropic Messages API

LocalAI supports the Anthropic Messages API, which is compatible with Claude clients. This endpoint provides a structured way to send messages and receive responses, with support for tools, streaming, and multimodal content.

Endpoint: POST /v1/messages or POST /messages

Reference: https://docs.anthropic.com/claude/reference/messages_post

Basic Usage

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "max_tokens": 1024,
    "messages": [
      {"role": "user", "content": "Say this is a test!"}
    ]
  }'

Request Parameters

Message Format

Messages can contain text or structured content blocks:

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "max_tokens": 1024,
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What is in this image?"
          },
          {
            "type": "image",
            "source": {
              "type": "base64",
              "media_type": "image/jpeg",
              "data": "base64_encoded_image_data"
            }
          }
        ]
      }
    ]
  }'

Tool Calling

The Anthropic API supports function calling through tools:

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "max_tokens": 1024,
    "tools": [
      {
        "name": "get_weather",
        "description": "Get the current weather",
        "input_schema": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state"
            }
          },
          "required": ["location"]
        }
      }
    ],
    "tool_choice": {"type": "auto"},
    "messages": [
      {"role": "user", "content": "What is the weather in San Francisco?"}
    ]
  }'

Streaming

Enable streaming responses by setting stream: true:

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "max_tokens": 1024,
    "stream": true,
    "messages": [
      {"role": "user", "content": "Tell me a story"}
    ]
  }'

Streaming responses use Server-Sent Events (SSE) format with event types: message_start, content_block_start, content_block_delta, content_block_stop, message_delta, and message_stop.

Response Format

{
  "id": "msg_abc123",
  "type": "message",
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "This is a test!"
    }
  ],
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "stop_reason": "end_turn",
  "usage": {
    "input_tokens": 10,
    "output_tokens": 5
  }
}

stop_reason is end_turn, tool_use, or max_tokens when the reply used up max_tokens.

Open Responses API

LocalAI supports the Open Responses API specification, which provides a standardized interface for AI model interactions with support for background processing, streaming, tool calling, and advanced features like reasoning.

Endpoint: POST /v1/responses or POST /responses

Reference: https://www.openresponses.org/specification

Basic Usage

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": "Say this is a test!",
    "max_output_tokens": 1024
  }'

Reuse assistant messages

When you manage conversation history in your client, append assistant messages from a response’s output array to the next request’s input. Keep their content type as output_text. Use input_text for new user messages. Include earlier conversation items that the model still needs.

For example, submit this body to POST /v1/responses:

{
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "input": [
    {
      "type": "message",
      "role": "user",
      "content": [{"type": "input_text", "text": "What is the capital of France?"}]
    },
    {
      "type": "message",
      "role": "assistant",
      "content": [{"type": "output_text", "text": "Paris is the capital of France."}]
    },
    {
      "type": "message",
      "role": "user",
      "content": [{"type": "input_text", "text": "Which river flows through it?"}]
    }
  ]
}

WebSocket Responses

Connect to ws://localhost:8080/v1/responses (or wss:// when TLS is enabled) and send response.create messages over the WebSocket. Only one response may be in progress on a connection. Wait for response.completed or response.failed for the active response before sending the next response.create. An error that rejects an invalid or additional response.create applies only to that rejected message; it does not terminate a response that is already in progress.

Set the WebSocket-only generate field to false to prepare a request without running inference:

{
  "type": "response.create",
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "generate": false,
  "store": false,
  "input": "Say this is a test!"
}

LocalAI emits response.created followed by response.completed with no generated output. The completed response still has an ID that can continue the prepared request:

{
  "type": "response.create",
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "store": false,
  "previous_response_id": "resp_abc123",
  "input": []
}

Response IDs created with store: false are available only on the WebSocket connection that created them and are removed when that connection closes. A previous_response_id chain can contain multiple responses; LocalAI replays the complete conversation from the oldest response through the referenced response before appending the new input. The replay includes server-side MCP tool calls and their results, which are stored as function_call and function_call_output output items. Keep store: false for every descendant of a connection-local response. LocalAI rejects a store: true response that depends on connection-local history because the resulting stored chain would be broken after the WebSocket closes.

After delivery, LocalAI discards buffered stream events for store: false responses but retains their request and final response for continuation. A connection admits at most 128 local responses and accepts a new local request only while its current serialized history plus that request is at most 64 MiB. When either admission threshold is reached, LocalAI returns connection_store_limit_reached; reconnect to start a new local history. With authentication enabled, globally stored response IDs can be continued only by the identity that created them.

Request Parameters

Input Format

Input can be a simple string or an array of structured items:

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": [
      {
        "type": "message",
        "role": "user",
        "content": "What is the weather?"
      }
    ],
    "max_output_tokens": 1024
  }'

Streaming responses

Set "stream": true to receive Server-Sent Events. Each response.output_item.added event assigns an output_index to an item. Use that index and the item ID to associate later deltas and completion events with the same item.

If a request without explicit tools produces reasoning, the stream uses separate items for reasoning and answer text. Each item keeps its original index throughout the stream. The response.completed event includes both items in the same index order, followed by any automatically parsed tool calls.

Background Processing

Run requests in the background for long-running tasks:

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": "Generate a long story",
    "max_output_tokens": 4096,
    "background": true
  }'

The response will include a response ID that can be used to poll for completion:

{
  "id": "resp_abc123",
  "object": "response",
  "status": "in_progress",
  "created_at": 1234567890
}

Retrieving Background Responses

Use the GET endpoint to retrieve background responses:

# Get response by ID
curl http://localhost:8080/v1/responses/resp_abc123

# Resume streaming with query parameters
curl "http://localhost:8080/v1/responses/resp_abc123?stream=true&starting_after=10"

Canceling Background Responses

Cancel a background response that’s still in progress:

curl -X POST http://localhost:8080/v1/responses/resp_abc123/cancel

Multiple Replicas (Distributed Mode)

In distributed mode LocalAI replicates response metadata across frontend replicas, so retrieval, previous_response_id chaining and cancellation work regardless of which replica the load balancer picks:

• GET /v1/responses/{id} returns the response from any replica.
• POST /v1/responses/{id}/cancel is delegated over NATS to the replica that is actually generating, so generation really stops. If that replica is gone, the response is reported as cancelled without blocking.
• Streaming resume (?stream=true&starting_after=N) works on any replica, because the event buffer is read from the database (see Durable Storage below). Without a database (it always exists in distributed mode, so this only concerns a build that cannot reach it) the buffer is in the memory of the replica that generates, and a resume that reaches another replica returns HTTP 409 naming the owning replica.

State mode (see the distributed-state rules): the response metadata is shared state, a SyncedMap with a database Store. The table is the source of truth, a replica that starts later loads the newest rows (up to 10000), and changes reach the other replicas as deltas over NATS. The events are not sent over NATS: every replica reads them from the table. The generation itself, and its cancel function, live on one replica, the owner of the response; a cancel reaches it over NATS and through the cancel_requested flag.

Durable Storage

When LocalAI has a database (authentication enabled, or distributed mode), stored responses and the events of a background run are kept in two tables, open_responses and open_response_events. Without a database the store stays in memory, as before, and ends with the process.

With the tables:

• A response and its event buffer outlive a restart. GET /v1/responses/{id}, previous_response_id chaining and ?stream=true&starting_after=N read the tables, so any replica serves them and they work after a restart.
• The owner writes events in batches: at once for any event that is not a .delta event, and for deltas every 250 ms or 64 events. If the process dies, the deltas of the last 250 ms are lost.
• The buffer keeps the same caps as the memory store (8192 events and 64 MiB per response, oldest first). A resume below the oldest kept event returns HTTP 409 (ErrOffsetLost).
• Every read still checks the owner of the response: a caller who did not create it gets 404. An item_reference finds only an item of a response the caller created; the item of another user’s response is “not found”.
• Rows expire after OPEN_RESPONSES_STORE_TTL. With the TTL at 0 a database store applies a default of 7 days, because a table does not end with the process.

A running background response holds a lease in its row, which its owner renews every 15 seconds (the lease lasts 45 seconds). When the owner dies, for example because the pod is replaced, the lease runs out and the response is ended as failed with the error code server_restarted. A final response.failed event is added, so a client that resumes the stream gets every kept event and then the end. LocalAI does not run the request again: a generation cannot continue from a cut point, and tools may have had effects. Send the request again, with previous_response_id to keep the context. A standalone server ends the runs a previous process left when it starts. In distributed mode one frontend at a time runs the sweep.

POST /v1/responses/{id}/cancel sets a cancel_requested flag in the row. The owner checks it on every lease renewal, so a cancel works even if the bus message is lost. If the owner died before it could stop, the response ends as cancelled.

Tool Calling

Open Responses API supports function calling with tools:

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": "What is the weather in San Francisco?",
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "description": "Get the current weather",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state"
            }
          },
          "required": ["location"]
        }
      }
    ],
    "tool_choice": "auto",
    "max_output_tokens": 1024
  }'

For streaming requests with JSON tool output, LocalAI waits for the complete JSON object before emitting a completed function_call item. Arguments can span multiple tokens. Read the arguments from the response.output_item.done event before executing the tool.

Reasoning Configuration

Configure reasoning effort and summary style:

curl http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ggml-koala-7b-model-q4_0-r2.bin",
    "input": "Solve this complex problem step by step",
    "reasoning": {
      "effort": "high",
      "summary": "detailed"
    },
    "max_output_tokens": 2048
  }'

Response Format

{
  "id": "resp_abc123",
  "object": "response",
  "created_at": 1234567890,
  "completed_at": 1234567895,
  "status": "completed",
  "model": "ggml-koala-7b-model-q4_0-r2.bin",
  "output": [
    {
      "type": "message",
      "id": "msg_001",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "This is a test!",
          "annotations": [],
          "logprobs": []
        }
      ],
      "status": "completed"
    }
  ],
  "error": null,
  "incomplete_details": null,
  "temperature": 0.7,
  "top_p": 1.0,
  "presence_penalty": 0.0,
  "frequency_penalty": 0.0,
  "usage": {
    "input_tokens": 10,
    "output_tokens": 5,
    "total_tokens": 15,
    "input_tokens_details": {
      "cached_tokens": 0
    },
    "output_tokens_details": {
      "reasoning_tokens": 0
    }
  }
}

Backends

RWKV

RWKV support is available through llama.cpp (see below)

llama.cpp

llama.cpp is a popular port of Facebook’s LLaMA model in C/C++.

Features

The llama.cpp model supports the following features:

• 📖 Text generation (GPT)
• 🧠 Embeddings
• 🔥 OpenAI functions
• ✍️ Constrained grammars

Setup

LocalAI supports llama.cpp models out of the box. You can use the llama.cpp model in the same way as any other model.

Manual setup

It is sufficient to copy the ggml or gguf model files in the models folder. You can refer to the model in the model parameter in the API calls.

You can optionally create an associated YAML model config file to tune the model’s parameters or apply a template to the prompt.

Prompt templates are useful for models that are fine-tuned towards a specific prompt.

With the llama.cpp backend, streaming and non-streaming chat requests preserve the template’s grammar, stop strings, and message boundaries.

Automatic setup

LocalAI supports model galleries which are indexes of models. For instance, the huggingface gallery contains a large curated index of models from the huggingface model hub for ggml or gguf models.

For instance, if you have the galleries enabled and LocalAI already running, you can just start chatting with models in huggingface by running:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "TheBloke/WizardLM-13B-V1.2-GGML/wizardlm-13b-v1.2.ggmlv3.q2_K.bin",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.1
   }'

LocalAI will automatically download and configure the model in the model directory.

Models can be also preloaded or downloaded on demand. To learn about model galleries, check out the model gallery documentation.

YAML configuration

To use the llama.cpp backend, specify llama-cpp as the backend in the YAML file:

name: llama
backend: llama-cpp
parameters:
  # Relative to the models path
  model: file.gguf

Backend Options

The llama.cpp backend supports additional configuration options that can be specified in the options field of your model YAML configuration. These options allow fine-tuning of the backend behavior:

Example configuration with options:

name: llama-model
backend: llama
parameters:
  model: model.gguf
options:
  - use_jinja:true
  - context_shift:true
  - cache_ram:4096
  - parallel:2
  - devices:CUDA1,CUDA2,CUDA3
  - fit_params:true
  - fit_target:1024
  - slot_prompt_similarity:0.5

Note: The parallel option can also be set via the LLAMACPP_PARALLEL environment variable, and grpc_servers can be set via the LLAMACPP_GRPC_SERVERS environment variable. Options specified in the YAML file take precedence over environment variables.

An explicit parallel: 1 (or n_parallel: 1) in the model options takes precedence over LLAMACPP_PARALLEL, like any other value. The environment variable is only used when neither option is set; if it is missing or not a number, the backend uses one slot.

Hardware auto-tuning (and how to override it)

On a detected GPU, LocalAI fills a few performance-relevant defaults the model config leaves unset - a larger physical batch on NVIDIA Blackwell, and a VRAM-scaled parallel slot count for concurrent serving. Both are gated on per-device VRAM at the model’s context: when a large context already fills a single card (e.g. a 27B model with a 200k context across 2×16 GiB), the batch boost and the extra parallel slots are suppressed so they can’t tip the tighter GPU into CUDA out-of-memory.

Anything you set explicitly in the model YAML always wins, so to pin a value just set it (e.g. batch: 512 or options: ["parallel:1"]). The effective values are logged at INFO when a model loads (effective runtime tuning …). To turn the hardware auto-tuning off entirely and run llama.cpp’s stock behavior, set:

LOCALAI_DISABLE_HARDWARE_DEFAULTS=true

Server-side prompt cache (repeated system prompts)

Agents, coding assistants, and Anthropic/OpenAI-compatible CLIs typically resend the same large system prompt on every turn. The llama.cpp server can short-circuit prefill for the matching prefix by stashing idle slot KV states in host RAM and reloading them on a hit. Three settings interact:

All three are on by default since LocalAI v4.3, so the prompt cache works out of the box for the common single-slot setup. If you’re on an older release, or you’ve explicitly disabled one of them, add the following to recover the behaviour:

options:
  - cache_ram:4096       # or -1 for no limit
  - kv_unified:true
  - cache_idle_slots:true

Set cache_ram:0 to opt out of the prompt cache entirely (saves host RAM at the cost of re-prefilling repeated prompts).

cache_ram:-1 removes the limit. With idle-slot saving on, every distinct prompt then leaves its slot state in host RAM, so a workload with many different prompts (classification, ingestion) grows the backend by roughly the KV size of each prompt until the host runs out of memory.

Saving and restoring a prompt cache

The server-side prompt cache lives in RAM and is lost when the model unloads. To keep the prompt state of one slot on disk, set slot_save_path and use the slot API. A typical use is a voice assistant with a long system prompt: save the state once, then restore it when the wake word fires, so the first request does not wait for prefill.

options:
  - slot_save_path:/var/lib/localai/slots

Both calls need an admin API key when authentication is on. Without slot_save_path they return 400.

The server chooses the name of every saved file. A save returns an id, and a restore takes that id. A client cannot pick a file name or a path.

# Save slot 0 of the model. The reply contains the id.
curl http://localhost:8080/v1/slots/save \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model", "slot": 0}'

# Load that state back into slot 0
curl http://localhost:8080/v1/slots/restore \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model", "slot": 0, "id": "0123456789abcdef0123456789abcdef"}'

The save reply gives the id, the slot, the number of tokens and bytes, and the time taken in duration_ms. The message field lists the IDs of any older saves that were deleted to make room. The restore reply has the same fields. Neither reply contains a path on the host.

Keep these points in mind:

• A client cannot choose names. The backend creates a directory for each model under slot_save_path (mode 0700) and writes a manifest.json in it. A restore works only for an id in the manifest of that model. Any other id returns 404 with no saved state with this id for this model.
• The backend keeps at most 32 saves for each model. When a save would go over that, it deletes the oldest one and lists its id in message. A save larger than 8 GiB is deleted and fails.
• A restore also checks that the file is a plain file and has the size that was recorded at save time. A file that was replaced or cut short is refused.
• Keep slot_save_path private to the user that runs LocalAI. LocalAI creates it with mode 0700 when it is missing, and writes a warning to the log when an existing directory is open to other users. The checks above cannot stop a local user who can write to the directory.
• A saved state is only as fresh as its save. To change the cached prompt, save again and use the new id.
• A saved state holds the prompt state for the model and quantisation it came from. Do not restore it into a different model, quantisation or context size. The restore call fails if the state does not fit, but a state from another model is not always detected.
• A slot that is busy with a request is served after that request ends.
• The API is for llama-cpp only. Other backends return 501.

Reference

• llama

ik_llama.cpp

ik_llama.cpp is a hard fork of llama.cpp by Iwan Kawrakow that focuses on superior CPU and hybrid GPU/CPU performance. It ships additional quantization types (IQK quants), custom quantization mixes, Multi-head Latent Attention (MLA) for DeepSeek models, and fine-grained tensor offload controls - particularly useful for running very large models on commodity CPU hardware.

Features

The ik-llama-cpp backend supports the following features:

• 📖 Text generation (GPT)
• 🧠 Embeddings
• IQK quantization types for better CPU inference performance
• Multimodal models (via clip/llava)

Setup

The backend is distributed as a separate container image and can be installed from the LocalAI backend gallery, or specified directly in a model configuration. GGUF models loaded with this backend benefit from ik_llama.cpp’s optimized CPU kernels - especially useful for MoE models and large quantized models that would otherwise be GPU-bound.

YAML configuration

To use the ik-llama-cpp backend, specify it as the backend in the YAML file:

name: my-model
backend: ik-llama-cpp
parameters:
  # Relative to the models path
  model: file.gguf

The aliases ik-llama and ik_llama are also accepted.

Reference

• ik_llama.cpp

turboquant (llama.cpp fork with TurboQuant KV-cache)

llama-cpp-turboquant is a llama.cpp fork that adds the TurboQuant KV-cache quantization scheme. It reuses the upstream llama.cpp codebase and ships as a drop-in alternative backend inside LocalAI, sharing the same gRPC server sources as the stock llama-cpp backend - so any GGUF model that runs on llama-cpp also runs on turboquant.

You would pick turboquant when you want smaller KV-cache memory pressure (longer contexts on the same VRAM) or to experiment with the fork’s quantized KV representations on top of the standard cache_type_k / cache_type_v knobs already supported by upstream llama.cpp.

Features

• Drop-in GGUF compatibility with upstream llama.cpp.
• TurboQuant KV-cache quantization (see fork README for the current set of accepted cache_type_k / cache_type_v values).
• Same feature surface as the llama-cpp backend: text generation, embeddings, tool calls, multimodal via mmproj.
• Available on CPU (AVX/AVX2/AVX512/fallback), NVIDIA CUDA 12/13, AMD ROCm/HIP, Intel SYCL f32/f16, Vulkan, and NVIDIA L4T.

Setup

turboquant ships as a separate container image in the LocalAI backend gallery. Install it like any other backend:

local-ai backends install turboquant

Or pick a specific flavor for your hardware (example tags: cpu-turboquant, cuda12-turboquant, cuda13-turboquant, rocm-turboquant, intel-sycl-f16-turboquant, vulkan-turboquant).

YAML configuration

To run a model with turboquant, set the backend in your model YAML and optionally pick quantized KV-cache types:

name: my-model
backend: turboquant
parameters:
  # Relative to the models path
  model: file.gguf
# Use TurboQuant's own KV-cache quantization schemes. The fork accepts
# the standard llama.cpp types (f16, f32, q8_0, q4_0, q4_1, q5_0, q5_1)
# and adds three TurboQuant-specific ones: turbo2, turbo3, turbo4.
# turbo3 / turbo4 auto-enable flash_attention (required for turbo K/V)
# and offer progressively more aggressive compression.
cache_type_k: turbo3
cache_type_v: turbo3
context_size: 8192

The cache_type_k / cache_type_v fields map to llama.cpp’s -ctk / -ctv flags. The stock llama-cpp backend only accepts the standard llama.cpp types - to use turbo2 / turbo3 / turbo4 you need this turboquant backend, which is where the fork’s TurboQuant code paths actually take effect. Pick q8_0 here and you’re just running stock llama.cpp KV quantization; pick turbo* and you’re running TurboQuant.

Reference

• llama-cpp-turboquant
• Tracked branch: feature/turboquant-kv-cache

vLLM

vLLM is a fast and easy-to-use library for LLM inference.

LocalAI has a built-in integration with vLLM, and it can be used to run models. You can check out vllm performance here.

Setup

Create a YAML file for the model you want to use with vllm.

To setup a model, you need to just specify the model name in the YAML config file:

name: vllm
backend: vllm
parameters:
    model: "facebook/opt-125m"

The backend will automatically download the required files in order to run the model.

Usage

Use the completions endpoint by specifying the vllm backend:

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{   
   "model": "vllm",
   "prompt": "Hello, my name is",
   "temperature": 0.1, "top_p": 0.1
 }'

Passing arbitrary vLLM options with engine_args

A subset of AsyncEngineArgs is exposed as typed YAML fields (tensor_parallel_size, gpu_memory_utilization, quantization, max_model_len, dtype, trust_remote_code, enforce_eager, …). Anything else can be passed through the generic engine_args: map. Keys are forwarded verbatim to vLLM’s engine; unknown keys fail at load time with the closest valid name as a hint. Nested maps materialise into vLLM’s nested config dataclasses (SpeculativeConfig, KVTransferConfig, CompilationConfig, …).

Speculative decoding (DFlash, ngram, eagle, deepseek_mtp, …) is configured this way:

name: qwen3.5-4b-dflash
backend: vllm
parameters:
  model: Qwen/Qwen3.5-4B
context_size: 8192
max_model_len: 8192
trust_remote_code: true
quantization: fp8
template:
  use_tokenizer_template: true
engine_args:
  speculative_config:
    method: dflash
    model: z-lab/Qwen3.5-4B-DFlash
    num_speculative_tokens: 15

The shape of speculative_config follows vLLM’s SpeculativeConfig

• method picks the algorithm, the remaining keys are method-specific. Drafters from z-lab are paired with specific target models; pick the one that matches your target. The drafter loads in its native precision regardless of the target’s quantization: setting.

Another example - picking a non-default attention backend (e.g. on hardware where the default cutlass kernels aren’t supported):

engine_args:
  attention_backend: TRITON_ATTN

CLI-style engine flags in options

Engine arguments can also be written in options: with vLLM’s own CLI spelling, which is handy when copying a command line out of vLLM’s documentation:

options:
  - --quantization:gptq_marlin
  - --enable-prefix-caching
  - --kv-cache-dtype:fp8_e5m2
  - --reasoning-parser:qwen3

Rules:

• Only -- prefixed entries are treated as engine flags; everything else in options: (tool_parser:, reasoning_parser:, …) keeps its existing meaning. Both --flag:value and --flag=value are accepted.
• Dashes become underscores (--enable-prefix-caching → enable_prefix_caching) and the value is coerced to the target field’s type. A bare --flag sets a boolean field to true.
• Unknown or uncoercible flags are logged on the backend’s stderr and skipped rather than failing the load. engine_args: stays strict and is applied last, so it wins on conflict.

engine_args: remains the form to use for anything structured (nested configs, maps); options: is a convenience for flat flags.

Multi-node data parallelism

engine_args.data_parallel_size > 1 combined with the local-ai p2p-worker vllm follower lets a single model span multiple GPU nodes. See vLLM Multi-Node (Data-Parallel) for the head/follower configuration and a worked Kimi-K2.6 example.

SGLang

SGLang is a fast serving framework for LLMs and VLMs with a focus on prefix caching, speculative decoding, and multi-modal generation. LocalAI ships a gRPC backend that wraps SGLang’s async Engine, including its native function-call and reasoning parsers.

Setup

name: sglang
backend: sglang
parameters:
  model: "Qwen/Qwen3-4B"
template:
  use_tokenizer_template: true

The backend will pull the model from HuggingFace on first load.

Passing arbitrary SGLang options with engine_args

The same engine_args: map that the vLLM backend accepts is also honoured by the SGLang backend. Keys are validated against ServerArgs

• SGLang’s central configuration dataclass - and forwarded verbatim to Engine(**kwargs). Unknown keys fail at load time with the closest valid name as a hint. Unlike vLLM, ServerArgs is flat: speculative decoding fields are top-level (speculative_algorithm, speculative_draft_model_path, etc.) rather than nested under a speculative_config: dict.

The typed YAML fields shared with vLLM are mapped to their SGLang equivalents (gpu_memory_utilization → mem_fraction_static, enforce_eager → disable_cuda_graph, tensor_parallel_size → tp_size, max_model_len → context_length). Anything else, including all speculative-decoding flags, goes under engine_args:.

Speculative decoding: Gemma 4 with Multi-Token Prediction

Google publishes paired “assistant” drafters for every Gemma 4 size. The drafters use Multi-Token Prediction (MTP) to propose several candidate tokens per target step, which SGLang then verifies in parallel. Flags below are transcribed verbatim from the SGLang Gemma 4 cookbook.

For consumer GPUs in the 16-24 GB range, use E4B (8 B total / 4 B effective parameters):

name: gemma-4-e4b-mtp
backend: sglang
parameters:
  model: google/gemma-4-E4B-it
context_size: 4096
template:
  use_tokenizer_template: true
options:
  - tool_parser:gemma4
  - reasoning_parser:gemma4
engine_args:
  mem_fraction_static: 0.85
  speculative_algorithm: NEXTN
  speculative_draft_model_path: google/gemma-4-E4B-it-assistant
  speculative_num_steps: 5
  speculative_num_draft_tokens: 6
  speculative_eagle_topk: 1

For smaller cards (8-12 GB), drop to E2B (5 B total / 2 B effective) by swapping the model paths to google/gemma-4-E2B-it and google/gemma-4-E2B-it-assistant; the rest of the flags stay the same.

NEXTN is normalised to EAGLE inside ServerArgs.__post_init__, so either value works - the cookbook uses NEXTN. mem_fraction_static is the share of GPU memory SGLang reserves for the model + KV pool; 0.85 is the cookbook’s default and adapts to whatever single GPU the backend is running on.

The 31 B dense and 26 B-A4B MoE Gemma 4 variants exist in the same cookbook but require --tp-size 2, so they’re not in the gallery as single-GPU recipes.

SGLang version requirement. Gemma 4 support landed in SGLang via PR #21952. The LocalAI sglang backend pins a release that includes it; if you’ve overridden the pin to an older version, this recipe will fail with a “model architecture not recognised” error at load time.

Other speculative algorithms

speculative_algorithm: also accepts EAGLE/EAGLE3 (paired with an EAGLE-style draft head), DFLASH (block-diffusion drafters from z-lab for the Qwen3 family), STANDALONE (a smaller draft LLM verifying a larger target), and NGRAM (no draft model - pure prefix-history speculation). See SGLang’s speculative-decoding docs for the full algorithm matrix.

Tool calling and reasoning parsers

SGLang’s native parsers stream tool_calls and reasoning_content inside ChatDelta - the LocalAI Python backend wires them up per-request rather than via engine_args:. Pick a parser by name:

options:
  - tool_parser:hermes
  - reasoning_parser:deepseek_r1

The full list of registered parsers lives in sglang.srt.function_call and sglang.srt.parser.reasoning_parser.

Reasoning defaults and token budgets

Set SGLang reasoning options in the model’s options: list:

options:
  - reasoning_parser:qwen3
  - thinking_budget:512
  - reasoning_default:on
engine_args:
  enable_strict_thinking: true

thinking_budget sets a positive integer token budget for reasoning on each request. Invalid, zero, and negative values produce a warning and leave the budget unset. SGLang requires engine_args.enable_strict_thinking: true to enforce the budget. LocalAI warns if you configure a budget without that engine option. Keep the budget well below the max_tokens of your requests: if max_tokens is reached first, the budget never triggers and the whole reply can be spent on reasoning, leaving the answer empty.

effort_budgets maps the request’s reasoning_effort to its own budget, so the levels differ in how long the model may think:

options:
  - thinking_budget:256                                        # fallback: no effort, or a level not listed below
  - effort_budgets:low=256;medium=1024;high=4096;xhigh=16384;max=16384

Entries are separated by ; and use the same parsing rules as thinking_budget (invalid entries are skipped with a warning). A request whose effort is not listed falls back to thinking_budget; without either, no budget is sent. A level listed here also turns reasoning on for that request, even when reasoning_default:off is set; an explicit enable_thinking: "false" in the request metadata (also sent when the model sets reasoning.disable: true) still wins. The budget is an upper bound, not a target: the model may stop thinking earlier.

reasoning_default:on or reasoning_default:off sets the default for LocalAI’s tokenizer chat template. Request metadata enable_thinking set to "true" or "false" overrides this default. An explicit prompt bypasses tokenizer template rendering. When no default or request override is set, the template keeps its own behavior.

LocalAI signals required reasoning when the rendered prompt ends with the configured parser’s opening reasoning token. An explicit output grammar disables this detection. Configure a reasoning parser that matches your model.

The backend reads these options when it loads the model. POST /models/reload rereads model configuration files but does not update options in an already loaded backend. Restarting only the backend does not reread configuration files. Restart LocalAI after changing these options to reload both the configuration and the backend.

vllm.cpp

vllm.cpp is the LocalAI team’s C++ port of vLLM: the same continuous-batching scheduler, paged KV cache and prefix caching, with no Python at inference time. It consumes either a HuggingFace safetensors model directory or a .gguf file, and applies the model’s chat template, tool-call parsing and reasoning split engine-side.

This section is the configuration reference. For installing the backend and the ready-made models the gallery ships for it, see the vllm.cpp backend page.

Setup

name: vllm-cpp
backend: vllm-cpp
parameters:
  model: "Qwen/Qwen3-4B"
context_size: 8192
template:
  use_tokenizer_template: true

Configuring the engine with engine_args

The same engine_args: map the vLLM and SGLang backends accept is honoured here, with keys spelled exactly as vLLM’s own CLI flags - so a speculative_config or kv_transfer_config block written for vLLM works verbatim. Unknown keys are ignored rather than fatal; the engine validates the documents it is handed and reports a precise error at load.

name: qwen35-a3b
backend: vllm-cpp
parameters:
  model: "Qwen/Qwen3.5-A3B"
context_size: 16384
template:
  use_tokenizer_template: true
engine_args:
  # KV cache sizing: num_blocks * block_size tokens of cache.
  block_size: 32
  num_blocks: 1024
  # Concurrency and the per-step chunked-prefill token budget.
  max_num_seqs: 32
  max_num_batched_tokens: 8192
  # Automatic prefix caching. Omit to keep the model's own default
  # (on for dense models, off for hybrid / attention-free ones).
  enable_prefix_caching: true
  # Scheduler admission order: fcfs (default), priority, or lpm
  # (cache-aware longest-prefix-match; needs prefix caching to have any effect).
  scheduling_policy: lpm

Raising max_num_batched_tokens lets more prefill land in a single step, at the cost of decode latency for requests queued behind it. The default deliberately does not scale with max_num_seqs, which is what keeps a large concurrent prefill from blowing up the per-step activation on the hybrid architectures.

enable_prefix_caching and enable_jump_forward are tri-state at the engine boundary: omitting the key defers to a default (the model’s own capability for prefix caching, an environment variable for jump forward), while an explicit false forces the feature off. Those are genuinely different - prefix caching defaults on for dense models - so write the key only when you mean to override.

Speculative decoding

speculative_config: takes the same JSON object as vLLM’s --speculative-config. Three methods are supported.

Architecture limit. At the current engine pin, mtp and dflash are Qwen3.5 / Qwen3.6 only. The engine builds a widened speculative KV cache directly for those families rather than through the model registry, so a speculative config on any other architecture (Llama, GLM, Gemma, Mistral, …) will not work regardless of checkpoint format. ngram needs no draft weights and is not subject to this limit.

Format support. mtp and dflash now work from a .gguf target as well as safetensors. An MTP head is read from the GGUF’s nextn.* tensors when the file declares <arch>.nextn_predict_layers; a GGUF exported WITHOUT the head (converted with --no-mtp, or predating llama.cpp’s Qwen3.5 MTP support) is refused at load naming that as the reason. A DFlash draft may itself be a dflash-arch GGUF, and the target may be a GGUF too. ngram needs no draft weights and works on any format.

MTP (Multi-Token Prediction) uses a draft head shipped inside the target checkpoint’s own mtp.* tensors, so there is no second model to download. It requires a safetensors checkpoint - the mtp.* tensors do not survive GGUF conversion, and an MTP config over a .gguf model is rejected at load.

engine_args:
  speculative_config:
    method: mtp
    # Optional; defaults to the checkpoint's own head depth, which is
    # usually the right value. Must be a multiple of that depth.
    num_speculative_tokens: 1

DFlash uses a separate block-diffusion drafter that proposes a whole block of tokens in one non-autoregressive forward pass. Unlike MTP, the draft is its own checkpoint, so model: is required:

engine_args:
  speculative_config:
    method: dflash
    model: z-lab/Qwen3.6-27B-DFlash
    num_speculative_tokens: 4

The draft shares the target’s embed_tokens and lm_head, so both must come from the same model family and the target must be safetensors.

The engine does not download the draft. model: is resolved, in order, as a path as given, then as the last path segment under LocalAI’s models directory (z-lab/Qwen3.6-27B-DFlash → <models>/Qwen3.6-27B-DFlash, which is what LocalAI’s own downloader produces), then as the whole reference under the models directory. Install the draft into LocalAI first, or give an absolute path to a directory containing config.json. If none of those resolve, the load fails immediately naming every location that was tried, rather than reporting a missing checkpoint from inside the engine.

N-gram needs no draft model at all - it proposes from the prompt’s own suffix history. num_speculative_tokens is required:

engine_args:
  speculative_config:
    method: ngram
    num_speculative_tokens: 4
    prompt_lookup_min: 5
    prompt_lookup_max: 5

Auto-configuration on import. When you import a safetensors repository with backend: vllm-cpp, LocalAI reads the checkpoint’s config.json and, if it declares an MTP head (mtp_num_hidden_layers), writes speculative_config: {method: mtp} into the generated engine_args for you. An explicit speculative_config in your own config is never overwritten. Importing a DFlash draft repository is refused with a warning: a drafter cannot serve on its own, so import the target model and point speculative_config.model at the draft.

External KV cache with LMCache

kv_transfer_config: takes vLLM’s --kv-transfer-config JSON and selects an external KV-cache connector. The lm:// LMCache client lets prefill KV be stored to and reloaded from a shared lmcache.v1.server, so a prefix computed by one replica does not have to be recomputed by the next:

engine_args:
  kv_transfer_config:
    kv_connector: LMCacheConnector
    kv_role: kv_both          # required whenever kv_connector is set
    kv_connector_extra_config:
      host: 127.0.0.1
      port: 65432

kv_role is one of kv_producer (store only), kv_consumer (load only), or kv_both. An unregistered connector name, a missing role, or a malformed document fails the load with an explicit error rather than silently running without the cache.

Legacy options: list

Earlier versions configured this backend through the flat options: list, and those configs keep working. Every key in the table above is still read from there in key:value form, and engine_args wins on any key set in both:

options:
  - max_num_seqs:32
  - enable_prefix_caching:true

New configs should prefer engine_args:, which is the only place the nested speculative_config / kv_transfer_config documents can be written naturally rather than as a single-line JSON string.

Transformers

Transformers is a State-of-the-art Machine Learning library for PyTorch, TensorFlow, and JAX.

LocalAI has a built-in integration with Transformers, and it can be used to run models.

This is an extra backend - in the container images (the extra images already contains python dependencies for Transformers) is already available and there is nothing to do for the setup.

Setup

Create a YAML file for the model you want to use with transformers.

To setup a model, you need to just specify the model name in the YAML config file:

name: transformers
backend: transformers
parameters:
    model: "facebook/opt-125m"
type: AutoModelForCausalLM
quantization: bnb_4bit # One of: bnb_8bit, bnb_4bit, xpu_4bit, xpu_8bit (optional)

The backend will automatically download the required files in order to run the model.

Parameters

Type

• OVModelForCausalLM requires OpenVINO IR Text Generation models from Hugging face
• OVModelForFeatureExtraction works with any Safetensors Transformer Feature Extraction model from Huggingface (Embedding Model)

Please note that streaming is currently not implemented in AutoModelForCausalLM for Intel GPU. AMD GPU support is not implemented. Although AMD CPU is not officially supported by OpenVINO there are reports that it works: YMMV.

Embeddings

Use embeddings: true if the model is an embedding model

Inference device selection

Transformer backend tries to automatically select the best device for inference, anyway you can override the decision manually overriding with the main_gpu parameter.

Example for CUDA: main_gpu: cuda.0

Example for OpenVINO: main_gpu: AUTO:-CPU

This parameter applies to both Text Generation and Feature Extraction (i.e. Embeddings) models.

Inference Precision

Transformer backend automatically select the fastest applicable inference precision according to the device support. CUDA backend can manually enable bfloat16 if your hardware support it with the following parameter:

f16: true

Quantization

Trust Remote Code

Some models like Microsoft Phi-3 requires external code than what is provided by the transformer library. By default it is disabled for security. It can be manually enabled with: trust_remote_code: true

Maximum Context Size

Maximum context size in bytes can be specified with the parameter: context_size. Do not use values higher than what your model support.

Usage example: context_size: 8192

Auto Prompt Template

Usually chat template is defined by the model author in the tokenizer_config.json file. To enable it use the use_tokenizer_template: true parameter in the template section.

Usage example:

template:
  use_tokenizer_template: true

Custom Stop Words

Stopwords are usually defined in tokenizer_config.json file. They can be overridden with the stopwords parameter in case of need like in llama3-Instruct model.

Usage example:

stopwords:
- "<|eot_id|>"
- "<|end_of_text|>"

Usage

Use the completions endpoint by specifying the transformers model:

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{   
   "model": "transformers",
   "prompt": "Hello, my name is",
   "temperature": 0.1, "top_p": 0.1
 }'

Examples

OpenVINO

A model configuration file for openvion and starling model:

name: starling-openvino
backend: transformers
parameters:
  model: fakezeta/Starling-LM-7B-beta-openvino-int8
context_size: 8192
threads: 6
f16: true
type: OVModelForCausalLM
stopwords:
- <|end_of_turn|>
- <|endoftext|>
prompt_cache_path: "cache"
prompt_cache_all: true
template:
  chat_message: |
    {{if eq .RoleName "system"}}{{.Content}}<|end_of_turn|>{{end}}{{if eq .RoleName "assistant"}}<|end_of_turn|>GPT4 Correct Assistant: {{.Content}}<|end_of_turn|>{{end}}{{if eq .RoleName "user"}}GPT4 Correct User: {{.Content}}{{end}}

  chat: |
    {{.Input}}<|end_of_turn|>GPT4 Correct Assistant:

  completion: |
    {{.Input}}

OpenAI Functions and Tools

LocalAI supports running the OpenAI functions and tools API across multiple backends. The OpenAI request shape is the same regardless of which backend runs your model - LocalAI is responsible for extracting structured tool calls from the model’s output before returning the response.

To learn more about OpenAI functions, see also the OpenAI API blog post.

LocalAI also supports JSON mode out of the box on llama.cpp-compatible models.

💡 Check out LocalAGI for an example on how to use LocalAI functions.

Supported backends

Reasoning content (<think>...</think> blocks from DeepSeek R1, Qwen3, Gemma 4, etc.) is returned in the OpenAI reasoning_content field on the same backends. When a model both reasons and calls a tool in the same turn, see Interleaved Thinking with Tool Calls for how the reasoning survives the tool-result round trip.

Setup

llama.cpp

No configuration required - the autoparser detects the tool call format for any ggml/gguf model that was trained with tool support.

Migrating older Ministral 3 configurations

The mistralai_ministral-3-14b-reasoning-2512-multimodal gallery entry uses the model’s embedded template and llama.cpp’s native tool parser. Older installations inherit the Mistral 0.3 prompt and JSON parser, which can return tool calls such as pick_tool{...} as text.

For an existing installation, replace the model YAML’s template and function sections with:

template:
  use_tokenizer_template: true
function:
  disable_no_action: true
  automatic_tool_parsing_fallback: true
  grammar:
    disable: true

Set use_jinja:true in the existing options list:

options:
  - use_jinja:true

Remove the inherited stopwords list so llama.cpp controls the model’s end-of-turn markers. Keep your model path, mmproj, sampling settings, and MCP configuration. Reload the model after saving the YAML. Gallery changes do not rewrite installed model configurations.

vLLM / vLLM Omni

The parser must be specified explicitly because vLLM itself doesn’t auto-detect one. Pass it via the model options:

name: qwen3-8b
backend: vllm
parameters:
  model: Qwen/Qwen3-8B
options:
  - tool_parser:hermes
  - reasoning_parser:qwen3
template:
  use_tokenizer_template: true

When you import a vLLM model through the LocalAI gallery, the importer looks up the model family and pre-fills tool_parser: and reasoning_parser: for you - you only need to override them for non-standard model names.

Available tool parsers include hermes, llama3_json, llama4_pythonic, mistral, qwen3_xml, deepseek_v3, granite4, kimi_k2, glm45, and more. Available reasoning parsers include deepseek_r1, qwen3, mistral, gemma4, granite. See the upstream vLLM documentation for the full list.

With backend: vllm, vLLM 0.30 or later, and a configured reasoning parser, streamed responses separate delta.reasoning_content, delta.content, and tool calls as tokens arrive. The final chunk does not repeat the generated text. If parsing fails during a stream, the backend buffers the remaining output and sends any matching, unsent suffixes in the final chunk. Older vLLM versions keep the previous streaming behavior and can include reasoning in delta.content.

MLX / MLX-VLM

MLX backends auto-detect the right tool parser by inspecting the model’s chat template - you don’t need to set anything. Just load an MLX-quantized model that was trained with tool support:

name: qwen2.5-0.5b-mlx
backend: mlx
parameters:
  model: mlx-community/Qwen2.5-0.5B-Instruct-4bit
template:
  use_tokenizer_template: true

The gallery importer will still append tool_parser: and reasoning_parser: entries to the YAML for visibility and consistency with the other backends, but those are informational - the runtime auto-detection in the MLX backend ignores them and uses the parser matched to the chat template.

Supported parser families: hermes/json_tools, mistral, gemma4, glm47, kimi_k2, longcat, minimax_m2, pythonic, qwen3_coder, function_gemma.

Usage example

You can configure a model manually with a YAML config file in the models directory, for example:

name: gpt-3.5-turbo
parameters:
  # Model file name
  model: ggml-openllama.bin
  top_p: 80
  top_k: 0.9
  temperature: 0.1

To use the functions with the OpenAI client in python:

from openai import OpenAI

messages = [{"role": "user", "content": "What is the weather like in Beijing now?"}]
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Return the temperature of the specified region specified by the user",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "User specified region",
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "temperature unit"
                    },
                },
                "required": ["location"],
            },
        },
    }
]

client = OpenAI(
    # This is the default and can be omitted
    api_key="test",
    base_url="http://localhost:8080/v1/"
)

response =client.chat.completions.create(
    messages=messages,
    tools=tools,
    tool_choice ="auto",
    model="gpt-4",
)
#...

For example, with curl:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "What is the weather like in Beijing now?"}],
  "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Return the temperature of the specified region specified by the user",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "User specified region"
                        },
                        "unit": {
                            "type": "string",
                            "enum": ["celsius", "fahrenheit"],
                            "description": "temperature unit"
                        }
                    },
                    "required": ["location"]
                }
            }
        }
    ],
    "tool_choice":"auto"
}'

Return data：

{
    "created": 1724210813,
    "object": "chat.completion",
    "id": "16b57014-477c-4e6b-8d25-aad028a5625e",
    "model": "gpt-4",
    "choices": [
        {
            "index": 0,
            "finish_reason": "tool_calls",
            "message": {
                "role": "assistant",
                "content": "",
                "tool_calls": [
                    {
                        "index": 0,
                        "id": "16b57014-477c-4e6b-8d25-aad028a5625e",
                        "type": "function",
                        "function": {
                            "name": "get_current_weather",
                            "arguments": "{\"location\":\"Beijing\",\"unit\":\"celsius\"}"
                        }
                    }
                ]
            }
        }
    ],
    "usage": {
        "prompt_tokens": 221,
        "completion_tokens": 26,
        "total_tokens": 247
    }
}

Advanced

Use functions without grammars

The functions calls maps automatically to grammars which are currently supported only by llama.cpp, however, it is possible to turn off the use of grammars, and extract tool arguments from the LLM responses, by specifying in the YAML file no_grammar and a regex to map the response from the LLM:

name: model_name
parameters:
  # Model file name
  model: model/name

function:
  # set to true to not use grammars
  no_grammar: true
  # set one or more regexes used to extract the function tool arguments from the LLM response
  response_regex:
  - "(?P<function>\w+)\s*\((?P<arguments>.*)\)"

The response regex have to be a regex with named parameters to allow to scan the function name and the arguments. For instance, consider:

(?P<function>\w+)\s*\((?P<arguments>.*)\)

will catch

function_name({ "foo": "bar"})

Parallel tools calls

This feature is experimental and has to be configured in the YAML of the model by enabling function.parallel_calls:

name: gpt-3.5-turbo
parameters:
  # Model file name
  model: ggml-openllama.bin
  top_p: 80
  top_k: 0.9
  temperature: 0.1

function:
  # set to true to allow the model to call multiple functions in parallel
  parallel_calls: true

Use functions with grammar

It is possible to also specify the full function signature (for debugging, or to use with other clients).

The chat endpoint accepts the grammar_json_functions additional parameter which takes a JSON schema object.

For example, with curl:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "gpt-4",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.1,
     "grammar_json_functions": {
        "oneOf": [
            {
                "type": "object",
                "properties": {
                    "function": {"const": "create_event"},
                    "arguments": {
                        "type": "object",
                        "properties": {
                            "title": {"type": "string"},
                            "date": {"type": "string"},
                            "time": {"type": "string"}
                        }
                    }
                }
            },
            {
                "type": "object",
                "properties": {
                    "function": {"const": "search"},
                    "arguments": {
                        "type": "object",
                        "properties": {
                            "query": {"type": "string"}
                        }
                    }
                }
            }
        ]
    }
   }'

Grammars and function tools can be used as well in conjunction with vision APIs:

 curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "llava", "grammar": "root ::= (\"yes\" | \"no\")",
     "messages": [{"role": "user", "content": [{"type":"text", "text": "Is there some grass in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}'

💡 Examples

A full e2e example with docker-compose is available here.

Constrained Grammars

Overview

The chat endpoint supports the grammar parameter, which allows users to specify a grammar in Backus-Naur Form (BNF). This feature enables the Large Language Model (LLM) to generate outputs adhering to a user-defined schema, such as JSON, YAML, or any other format that can be defined using BNF. For more details about BNF, see Backus-Naur Form on Wikipedia.

Setup

To use this feature, follow the installation and setup instructions on the LocalAI Functions page. Ensure that your local setup meets all the prerequisites specified for the llama.cpp backend.

💡 Usage Example

The following example demonstrates how to use the grammar parameter to constrain the model’s output to either “yes” or “no”. This can be particularly useful in scenarios where the response format needs to be strictly controlled.

Example: Binary Response Constraint

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "Do you like apples?"}],
  "grammar": "root ::= (\"yes\" | \"no\")"
}'

In this example, the grammar parameter is set to a simple choice between “yes” and “no”, ensuring that the model’s response adheres strictly to one of these options regardless of the context.

Example: JSON Output Constraint

You can also use grammars to enforce JSON output format:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "Generate a person object with name and age"}],
  "grammar": "root ::= \"{\" \"\\\"name\\\":\" string \",\\\"age\\\":\" number \"}\"\nstring ::= \"\\\"\" [a-z]+ \"\\\"\"\nnumber ::= [0-9]+"
}'

Example: YAML Output Constraint

Similarly, you can enforce YAML format:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "Generate a YAML list of fruits"}],
  "grammar": "root ::= \"fruits:\" newline (\"  - \" string newline)+\nstring ::= [a-z]+\nnewline ::= \"\\n\""
}'

Advanced Usage

For more complex grammars, you can define multi-line BNF rules. The grammar parser supports:

• Alternation (|)
• Repetition (*, +)
• Optional elements (?)
• Character classes ([a-z])
• String literals ("text")

Related Features

• OpenAI Functions - Function calling with structured outputs
• Text Generation - General text generation capabilities
• Moderation - OpenAI-compatible safety classification whose response is constrained to the moderation schema

Interleaved Thinking with Tool Calls

Reasoning models can “think” before they answer. When such a model also calls a tool, the useful behaviour is for the thinking and the tool call to travel together, and for that thinking to survive the tool-result round trip. LocalAI calls this interleaved thinking: a single assistant turn carries both reasoning and tool_calls, and the client hands the reasoning back on the next turn so the model’s chain of thought is not lost when the tool result is appended.

This matters because a tool-calling loop is multi-turn. The model reasons, asks for a tool, your client runs the tool, and then you call the model again with the tool result. Without interleaved thinking the reasoning produced in the first turn is discarded, and the model has to reconstruct its plan from scratch. With it, the reasoning is echoed back and the model continues where it left off.

See also: OpenAI Functions and Tools for how tool calls are extracted per backend, Text Generation (GPT) for the chat completions basics, and Advanced model configuration for the model options referenced below.

The round-trip contract

An assistant turn that both reasons and calls a tool returns the two fields side by side:

• reasoning holds the model’s thinking.
• tool_calls holds the structured calls.
• finish_reason is tool_calls.

Your client runs the tool, then sends the conversation back with:

1. the original assistant message (including its reasoning and tool_calls), and
2. a tool role message carrying the tool result.

LocalAI reads the returned reasoning back into the model’s context so the chain is continuous.

Field naming

OpenAI chat completions (/v1/chat/completions): the response carries reasoning alongside tool_calls. On inbound assistant messages LocalAI now also accepts reasoning_content as an alias for reasoning. This alias exists because several clients (vLLM, DeepSeek, and cogito) emit the field under the name reasoning_content; either name is accepted and mapped to the same internal field.

Anthropic Messages (/v1/messages): reasoning is carried as thinking content blocks. On the local path LocalAI emits a thinking block before the tool_use block, and reads inbound thinking blocks back into reasoning. Because local models produce no cryptographic signature, LocalAI attaches a synthetic opaque signature to the emitted block, and does not validate the signature on inbound blocks. The thinking block is only emitted when the request opts in with the thinking parameter:

{
  "model": "your-reasoning-model",
  "max_tokens": 1024,
  "thinking": { "type": "enabled" },
  "messages": [
    { "role": "user", "content": "What is the weather in Rome?" }
  ]
}

Enabling reasoning per backend

Interleaved thinking requires the backend to separate the model’s thinking from its final answer. How you turn that on depends on the backend.

llama.cpp

Set the reasoning_format model option. Accepted values: none, auto, deepseek, deepseek-legacy. auto lets the backend pick based on the model’s chat template; deepseek and deepseek-legacy force the DeepSeek-style <think>...</think> extraction.

name: my-reasoning-model
backend: llama-cpp
parameters:
  model: my-reasoning-model.gguf
options:
  - reasoning_format:auto

vLLM

Set the reasoning_parser model option to vLLM’s native reasoning parser for the model family. LocalAI also ships an auto-configuration hook (core/config/hooks_vllm.go) that sets the reasoning parser and the tool-call parser together for known families, so for gallery-imported vLLM models this is often configured for you.

name: my-vllm-model
backend: vllm
options:
  - reasoning_parser:deepseek_r1
  - tool_parser:hermes

SGLang

Set both the reasoning_parser and the tool_call_parser model options.

name: my-sglang-model
backend: sglang
options:
  - reasoning_parser:deepseek-r1
  - tool_call_parser:qwen25

Worked example

Request a chat completion with a tool defined and a prompt that requires the model to reason before acting:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "your-reasoning-model",
  "messages": [
    { "role": "user", "content": "What is the weather in Rome?" }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {
            "city": { "type": "string" }
          },
          "required": ["city"]
        }
      }
    }
  ]
}'

The response message carries both the reasoning and the tool call, and finish_reason is tool_calls:

{
  "choices": [
    {
      "finish_reason": "tool_calls",
      "message": {
        "role": "assistant",
        "content": "",
        "reasoning": "Okay, the user is asking about the weather in Rome... I need to call get_weather with city Rome.",
        "tool_calls": [
          {
            "id": "call_abc123",
            "type": "function",
            "function": {
              "name": "get_weather",
              "arguments": "{\"city\":\"Rome\"}"
            }
          }
        ]
      }
    }
  ]
}

To continue, run get_weather, then send the conversation back with the assistant message (keeping its reasoning and tool_calls) followed by a tool message holding the result. LocalAI feeds the returned reasoning back into context so the model resumes its chain of thought.

Known limitations

Streaming Anthropic thinking blocks. In streaming mode, a thinking block is currently emitted only on the tool-call path that goes through the llama.cpp C++ autoparser. A plain streaming text-only turn, or a tool turn resolved through the inline token path, does not stream a thinking block. The equivalent non-streaming request does return one. Non-streaming emits thinking on all branches; only the streaming path has this gap.

Upstream llama.cpp leak on newest hybrid models. On the newest hybrid reasoning models (Qwen3.5 / Qwen3.6) there is an upstream llama.cpp bug where tool calls can leak into reasoning_content instead of being parsed into tool_calls. This is tracked upstream at ggml-org/llama.cpp discussion #23351.

Reasoning budget vs the tool call. If a reasoning model’s thinking exhausts the output budget (max_tokens) before it emits the tool call, no tool call is produced. Give the model enough max_tokens to cover both the reasoning and the call. In this case LocalAI reports finish_reason: "length".

Model Aliases

A model alias is a model name that redirects all traffic to another configured model. Declare gpt-4 as an alias of my-llama-3 and every client calling gpt-4 is served by my-llama-3 with no client reconfiguration: the clients keep their existing model name while you control what answers them on the server side.

Declaring an alias

Create a minimal config file in your models directory:

name: gpt-4
alias: my-llama-3

That is the whole config: a name (the alias clients call) and an alias key (the target that actually serves the request). An alias may add an optional profile block.

Rules and behavior

• The target (my-llama-3) must be an existing, non-alias, enabled model. You cannot point an alias at a missing model, a disabled model, or another alias (no chains).
• Aliases are 1:1. One alias maps to exactly one target.
• The target can be swapped live by editing the config file, calling the API, using the UI, or asking the assistant. No restart is required.
• Both gpt-4 and my-llama-3 appear in GET /v1/models.
• Responses echo the requested alias: a call to gpt-4 returns gpt-4 in the response model field, not the target name.
• Usage accounting records both sides: requested gpt-4, served my-llama-3.
• Aliases work for every modality (chat, embeddings, audio, images, and so on).

To serve a name from several models with automatic fallback, use a failover chain.

Profile aliases

An alias can also act as a profile of its target: a different assistant on the same loaded model. Add an optional profile block; its fields are applied on top of the target’s config for requests to the alias:

name: researcher
alias: my-llama-3
profile:
  system_prompt: You are a careful research assistant. Cite your sources.
  client_tools: drop
  mcp:
    auto: true
    remote: |
      {"mcpServers": {"search": {"url": "http://search.internal/mcp"}}}

Requests to researcher run on the same backend process as my-llama-3: the model is not loaded a second time, however many profiles point at it. A separate model config with its own backend and model file would load the weights again. Fields the profile does not set are taken from the target.

The block is opt-in. Only fields inside profile are applied: top-level fields on an alias (system_prompt, mcp, sampling parameters, templates, …) are still ignored, so an alias without a profile block behaves exactly as before. A profile block on a model that is not an alias is rejected.

MCP sessions of a profile with its own servers are kept apart from the target’s, so profiles with different tool sets do not interfere with each other. GET /v1/mcp/servers/<alias> (and the prompts and resources endpoints) report the servers a chat request for the alias would use, so a profile’s tool set can be checked without calling the model.

Evidence check

A profile with server-side MCP tools returns only the final answer to the client; the tool calls and their results stay on the server. When the client sends the conversation back on the next turn, that tool trail is missing, and a model may answer a follow-up question with a number it never looked up instead of calling the tool again. The evidence check marks such answers at the output, independently of the prompt:

name: bookkeeper
alias: my-llama-3
profile:
  mcp:
    auto: true
    remote: |
      {"mcpServers": {"ledger": {"url": "http://ledger.internal/mcp"}}}
  evidence_check:
    numbers: true
    # optional, defaults to the note below
    note: "_Note: this answer contains numbers that no tool returned in this turn — not verified._"

The rule is deterministic and applies to each request on its own:

1. If an MCP tool call succeeded on the server during this request, the answer is left unchanged. A call that failed (the tool could not be called or returned an error to LocalAI) delivers no evidence and does not count.
2. Otherwise every number in the final answer (a digit sequence, optionally with one . or , decimal part) is looked up among the numbers of the request as received: the profile’s system prompt and the content of every system, user, assistant and tool message. 3,5 and 3.5 count as the same number.
3. If at least one number of the answer is not found, the note is appended after a blank line. In a streamed response it arrives as one additional content chunk before the final chunk.

Answers that hand tool calls to the client are not checked. The check is off unless numbers: true is set, and note without numbers: true is rejected. It flags, it does not prove: a number the client happened to mention is accepted even if the answer uses it wrongly.

Source line

A research profile usually asks the model in its system prompt to cite its sources. A model can still answer from a search result without naming a single link, and the client never sees the tool results that held them. With sources: true LocalAI adds the links at the output:

name: researcher
alias: my-llama-3
profile:
  mcp:
    auto: true
    remote: |
      {"mcpServers": {"search": {"url": "http://search.internal/mcp"}}}
  evidence_check:
    sources: true
    # optional, defaults to "Sources"
    sources_label: Quellen

Like the number check it acts on /v1/chat/completions. The rule is deterministic and applies to each request on its own:

1. Only the results of MCP tool calls that succeeded on the server during this request are read. The client’s messages and failed calls are not a source.
2. If the final answer already contains an http:// or https:// URL, it is left unchanged.
3. Otherwise the distinct URLs of those tool results are collected in the order they first appear, at most five, and appended after a blank line as one line: Quellen: https://a.example/test, https://b.example/oil. Punctuation that ends a sentence is not taken as part of a URL. Parentheses inside URL paths and query parameters in JSON-encoded MCP text results are preserved.
4. If no tool result contains a URL, nothing is appended. No URL is ever made up: every link on the line was returned by a tool.

In a streamed response the line arrives as one additional content chunk before the final chunk, like the number note. Both checks can be enabled together; the number note follows the source line. sources_label without sources: true is rejected.

Links not retrieved

A model can also name a link it never opened: a plausible deep link on a known site, written from memory. The client cannot tell such a link from one a search tool returned, because it never sees the tool results. With urls: true LocalAI marks it at the output:

name: researcher
alias: my-llama-3
profile:
  mcp:
    auto: true
    remote: |
      {"mcpServers": {"search": {"url": "http://search.internal/mcp"}}}
  evidence_check:
    urls: true
    # optional; {urls} is replaced by the links, defaults to the note below
    urls_note: "_Note: this answer names links that no tool returned in this turn — not retrieved: {urls}_"

Like the other checks it acts on /v1/chat/completions, streamed or not. The rule is deterministic and applies to each request on its own:

1. Every http:// or https:// URL in the final answer is collected, bare or inside a Markdown link. Punctuation that ends a sentence is not part of it.
2. Evidence is every URL in the request as received — the profile’s system prompt and the content of every system, user and tool message — and in the results of the MCP tool calls that succeeded on the server during this request. Assistant messages are not evidence, so a link the model invented on an earlier turn is marked again when the client sends it back. Failed tool calls deliver no evidence.
3. Links are compared without scheme, fragment, a leading www. and a trailing slash; the host is compared case-insensitively, path and query exactly. A link counts as retrieved when it equals an evidence URL or is a shorter form of one at a / or ? boundary: https://a.example is backed by a retrieved https://a.example/report.pdf, but https://a.example/other.pdf is not.
4. If at least one link is not backed, the note is appended after a blank line with up to five such links. In a streamed response it arrives as one additional content chunk before the final chunk. If urls_note contains no {urls}, the links follow the note after a space.

The source line, this note and the number note can be enabled together and are appended in that order. urls_note without urls: true is rejected. The check flags, it does not prove: a link that was retrieved can still be cited for a claim the page does not make.

Requiring a tool call

A system prompt that says “always search first” does not stop a model from answering a question from its own knowledge, for example when the question comes with a picture. require_tool: true enforces the first step in the request instead:

name: researcher
alias: my-llama-3
profile:
  require_tool: true
  mcp:
    auto: true
    remote: |
      {"mcpServers": {"search": {"url": "http://search.internal/mcp"}}}

When the switch is set and a /v1/chat/completions request offers the profile’s server-side MCP tools, the first model call of the request is made with tool_choice: "required". Every follow-up call of the server-side tool loop is made with the tool_choice the client sent (or none), so after the tool results the model is free to answer. A required that stayed in place would make the model call a tool again after every result and never answer until the loop’s iteration limit.

How required is enforced depends on the model:

• With template.use_tokenizer_template (llama.cpp with the model’s own chat template), LocalAI passes tool_choice: "required" to the backend, which constrains the output to a tool call with a grammar built from the chat template’s tool-call format.
• When LocalAI builds the tool-call grammar itself, the first call’s grammar leaves out the no-action function (answer) the model would otherwise reply with directly.

The first call is not forced when:

• no server-side MCP tool was offered (MCP disabled, the caller lacks the mcp permission, or the profile has no servers),
• the client set tool_choice (other than auto) or function_call itself, which keeps precedence,
• the request does not end with a user message. A request that ends with a tool message returns the result of a tool the client ran itself; forcing another call there could bounce tool calls between client and server.

The switch applies per request, so every new user turn starts with a tool call. It enforces that a tool is called, not which one or with which arguments.

Image requests on a text-only target

A text-only model behind an alias rejects image input. vision_alias names a model that serves the alias’s requests that carry an image, while text requests stay on the target:

name: assistant
alias: my-text-llm
vision_alias: my-vision-llm
profile:
  system_prompt: You are the everyday assistant.

A chat request to assistant is served by my-vision-llm when a message contains an image part (image_url, input_image or image) and the target my-text-llm cannot accept images. The alias’s profile block is applied to my-vision-llm in the same way, so the assistant keeps its system prompt and tools. If the target accepts images itself, vision_alias is not used.

The vision model must exist and must not be an alias; a disabled vision model is refused like any disabled model. vision_alias on a config that is not an alias is rejected. At load time LocalAI warns when the vision model is missing, is an alias, or declares no image input.

Managing aliases

You can create, swap, and remove aliases from any of the management surfaces.

Web UI

Open Add Model and pick the Alias / Routing template, then set a name and a target. To re-point an existing alias, edit it and change the target.

REST API

• Create: POST /models/import
• Swap the target: PATCH /api/models/config-json/:name
• List all aliases: GET /api/aliases
• Delete: POST /models/delete/:name

Assistant and MCP

The LocalAI Assistant (and the MCP server) expose the same operations as tools: set_alias, list_aliases, and delete_model.

Aliases as deployment slots (distributed mode)

In distributed mode an alias can carry a scheduling rule. POST /api/nodes/scheduling accepts an alias for model_name, and the rule then governs whatever model the alias points at:

curl -X POST http://frontend:8080/api/nodes/scheduling \
  -H "Content-Type: application/json" \
  -d '{"model_name": "production", "node_selector": {"tier": "gpu"}, "min_replicas": 2}'

Re-point production and the placement policy follows it, so the alias behaves as a stable slot whose contents you can swap. Because a replica is shared by every name that resolves to it, only one rule may govern a given model at a time. See Scheduling a model alias.

Limits

Aliases are a static 1:1 redirect; a profile block changes only the fields listed above, not the model or its loading options. For classifier-based or load-balanced selection across several downstream models, use the intelligent router in the Middleware feature instead.

API Discovery & Instructions

LocalAI exposes a set of discovery endpoints that let external agents, coding assistants, and automation tools programmatically learn what the instance can do and how to control it - without reading documentation ahead of time.

Quick start

# 1. Discover what's available
curl http://localhost:8080/.well-known/localai.json

# 2. Browse instruction areas
curl http://localhost:8080/api/instructions

# 3. Get an API guide for a specific instruction
curl http://localhost:8080/api/instructions/config-management

Authentication and discovery

You can read these discovery surfaces without authentication. This remains true when LocalAI uses database authentication or legacy API keys:

• GET /.well-known/localai.json
• GET /api/instructions
• GET /api/instructions/{name}
• Swagger GET requests at /swagger and under /swagger/

Discovery does not grant access to the endpoints it advertises. Those endpoints require credentials unless the authentication guide lists them as a public discovery or bootstrap route. For example, GET /version requires credentials even though the well-known response includes the instance version.

Well-Known Discovery Endpoint

GET /.well-known/localai.json

Returns the instance version, all available endpoint URLs (flat and categorized), and runtime capabilities.

Example response (abbreviated):

{
  "version": "v2.28.0",
  "endpoints": {
    "chat_completions": "/v1/chat/completions",
    "models": "/v1/models",
    "models_capabilities": "/v1/models/capabilities",
    "config_metadata": "/api/models/config-metadata",
    "instructions": "/api/instructions",
    "swagger": "/swagger/index.html"
  },
  "endpoint_groups": {
    "openai_compatible": { "chat_completions": "/v1/chat/completions", "..." : "..." },
    "config_management": { "config_metadata": "/api/models/config-metadata", "..." : "..." },
    "model_management": { "..." : "..." },
    "monitoring": { "..." : "..." }
  },
  "capabilities": {
    "config_metadata": true,
    "config_patch": true,
    "vram_estimate": true,
    "model_storage": true,
    "backend_install": true,
    "mcp": true,
    "chats": true,
    "agent_jobs": true,
    "collections": true,
    "p2p": false
  }
}

The capabilities object reflects the current runtime configuration - for example, mcp is only true if MCP is enabled, chats and agent_jobs are false with --disable-chats, collections is false with --disable-collections, and backend_install is false when the server runs with --disable-backend-install (backend install and upgrade endpoints then answer 403).

Instructions API

Instructions are curated groups of related API endpoints. Each instruction maps to one or more Swagger tags and provides a focused, LLM-readable guide.

List all instructions

GET /api/instructions

curl http://localhost:8080/api/instructions

Returns a compact list of instruction areas:

{
  "instructions": [
    {
      "name": "chat-inference",
      "description": "OpenAI-compatible chat completions, text completions, and embeddings",
      "tags": ["inference", "embeddings"],
      "url": "/api/instructions/chat-inference"
    },
    {
      "name": "config-management",
      "description": "Discover, read, and modify model configuration fields with VRAM estimation",
      "tags": ["config"],
      "url": "/api/instructions/config-management"
    }
  ],
  "hint": "Fetch GET {url} for a markdown API guide. Add ?format=json for a raw OpenAPI fragment."
}

Available instructions:

Get an instruction guide

GET /api/instructions/:name

By default, returns a markdown guide suitable for LLMs and humans:

curl http://localhost:8080/api/instructions/config-management

Add ?format=json to get a raw OpenAPI fragment (filtered Swagger spec with only the relevant paths and definitions):

curl http://localhost:8080/api/instructions/config-management?format=json

Model Capabilities

GET /v1/models/capabilities

An additive, LocalAI-specific superset of /v1/models. It returns the same set of models but enriches each entry with the capabilities the model supports and the input/output modalities it accepts and produces. Use it to decide, before sending a request, whether a given model can take an image, audio, or video attachment directly - or whether the input needs converting/transcribing first.

The reported context_size uses a positive model-level value first. If the model does not set context_size, it uses Settings → Performance → Default Context Size when positive. Otherwise, it uses the backend fallback of 4096 tokens. For llama.cpp with separate KV caches, the reported value accounts for the number of parallel slots.

An entry has a description when the model configuration sets one. With authentication on, a user whose account has a model allow-list sees only the models on it.

Because it is purely additive, clients that only understand /v1/models keep working unchanged; they simply never call this route.

curl http://localhost:8080/v1/models/capabilities

{
  "object": "list",
  "data": [
    {
      "id": "qwen2.5-omni",
      "object": "model",
      "capabilities": ["chat", "vision", "tools"],
      "description": "A model that reads images and audio.",
      "input_modalities": ["text", "image", "audio"],
      "output_modalities": ["text"]
    },
    {
      "id": "parakeet",
      "object": "model",
      "capabilities": ["transcript"],
      "input_modalities": ["audio"],
      "output_modalities": ["text"]
    }
  ]
}

• capabilities - canonical usecase strings (e.g. chat, vision, transcript, tts, embeddings, image, video) plus the modifiers tools and thinking.
• input_modalities / output_modalities - the subsets of {text, image, audio, video} the model accepts and produces. LocalAI combines usecase-based inference, backend settings such as vLLM limit_mm_per_prompt, and explicit model-level known_input_modalities / known_output_modalities. The explicit fields cover distinctions a usecase cannot express, such as a video model that also accepts speech.

The same query parameters as /v1/models are honored (filter, excludeConfigured), and the same per-user model allowlist is applied when authentication is enabled.

Configuration Management APIs

These endpoints let agents discover model configuration fields, read current settings, modify them, and estimate VRAM usage.

Config metadata

GET /api/models/config-metadata

Returns structured metadata for all model configuration fields, organized by section. Each field includes its YAML path, Go type, UI type, label, description, default value, validation constraints, and available options.

# All fields
curl http://localhost:8080/api/models/config-metadata

# Filter by section
curl http://localhost:8080/api/models/config-metadata?section=parameters

Autocomplete values

GET /api/models/config-metadata/autocomplete/:provider

Returns runtime values for dynamic fields. Providers include backends, models, models:chat, models:tts, models:transcript, models:vad.

# List available backends
curl http://localhost:8080/api/models/config-metadata/autocomplete/backends

# List chat-capable models
curl http://localhost:8080/api/models/config-metadata/autocomplete/models:chat

Read model config

GET /api/models/config-json/:name

Returns the full model configuration as JSON:

curl http://localhost:8080/api/models/config-json/my-model

Update model config

PATCH /api/models/config-json/:name

Deep-merges a JSON patch into the existing model configuration. Only include the fields you want to change:

curl -X PATCH http://localhost:8080/api/models/config-json/my-model \
  -H "Content-Type: application/json" \
  -d '{"context_size": 16384, "gpu_layers": 40}'

The endpoint validates the merged config and writes it to disk as YAML.

Model storage report

GET /api/models/storage (admin)

Reports the on-disk footprint of the installed model configurations: per-model sizes with the portion shared with other models split out, every referenced file with the models referencing it, and references whose files are missing from disk.

curl http://localhost:8080/api/models/storage

VRAM estimation

POST /api/models/vram-estimate

Estimates VRAM usage for an installed model based on its weight files, context size, and GPU layer offloading:

curl -X POST http://localhost:8080/api/models/vram-estimate \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model", "context_size": 8192}'

{
  "sizeBytes": 4368438272,
  "sizeDisplay": "4.4 GB",
  "vramBytes": 6123456789,
  "vramDisplay": "6.1 GB",
  "context_note": "Estimate used default context_size=8192. The model's trained maximum context is 131072; VRAM usage will be higher at larger context sizes.",
  "model_max_context": 131072
}

Optional parameters: gpu_layers (number of layers to offload, 0 = all), kv_quant_bits (KV cache quantization, 0 = fp16).

Integration guide

A recommended workflow for agent/tool builders:

1. Discover: Fetch /.well-known/localai.json to learn available endpoints and capabilities.
2. Browse instructions: Fetch /api/instructions for an overview of instruction areas.
3. Deep dive: Fetch /api/instructions/{name} for a markdown API guide on a specific area.
4. Authenticate: Obtain credentials before you call an advertised protected endpoint.
5. Explore config: Use admin credentials with /api/models/config-metadata to understand configuration fields.
6. Interact: Use credentials with inference endpoints and configuration APIs.

Swagger UI

The full interactive API documentation is available without authentication at /swagger/index.html. Requests sent to protected endpoints from Swagger still require credentials.

Model Failover

A failover chain is a model name that is served by an ordered list of other models. LocalAI sends each request to the first healthy target. When a target fails, the request moves to the next target, and later requests stay there until the first target has recovered.

Use it to serve a model from a remote LocalAI or another OpenAI-compatible provider, and to fall back to a local model when the remote one is down.

Declaring a chain

name: assistant-llm
failover:
  targets:
    - model: argus-llm          # for example a localai-proxy or cloud-proxy model
    - model: gemma-local
      warm: true                # keep it loaded

Clients call assistant-llm. Each target is a normal model config. A chain has no backend and no parameters.model.

Optional settings, with their defaults:

failover:
  probe:
    interval: 15s     # how often an idle target is checked
    timeout: 5s
  trip:
    errors: 1         # failures within the window that mark a target down
    window: 30s
  recovery:
    probes: 3         # test requests a target must pass before it is used again
    min_dwell: 60s    # minimum time on a lower target before moving back

Rules:

• A chain needs at least 2 targets. A target can be an alias, but not another chain.
• A chain cannot also set alias or backend.
• Responses name the chain as the model. The X-LocalAI-Served-Model header names the target that served the request.
• A remote (localai-proxy or cloud-proxy) target receives its own model name, never the chain name: proxy.upstream_model, or the target name when upstream_model is empty. The health check looks for the same name.

How the target is chosen

• The active target is the first healthy target in the list.
• When a target fails, LocalAI marks it down and moves to the next target at once.
• LocalAI moves back to a higher target only when that target has passed recovery.probes test requests and the current target has been active for at least recovery.min_dwell. This stops an unstable upstream from moving traffic back and forth.
• When all targets are down, the chain is degraded. Each request still tries every target in order.

Retry inside a request

When a target fails before the response starts, LocalAI sends the same request to the next target. The client does not see the failure.

• LocalAI does not retry after the first byte of a response is sent (for example after the first streamed token). The request fails, the target is marked down, and the next request uses the next target.
• A target that is at its concurrency limit (an admission rejection) or that is disabled is skipped for that request without being marked down.
• LocalAI does not retry client errors (4xx), such as a prompt that is too long, because the next target would reject it too. A 4xx counts neither as a success nor as a failure for the target.
• Request bodies larger than 32 MiB are not retried.

When the primary did not serve the request, the response has the header X-LocalAI-Failover: fallback, or X-LocalAI-Failover: degraded when all targets were down.

Health checks

A request that succeeds counts as a check, so a busy target is almost never probed.

When a target is in more than one chain, its check settings come from the first of those chains in name order.

Warm targets

warm: true loads a local target at startup and protects it from idle and LRU eviction, so a switch does not wait for the model to load. Warm targets count toward the active backend limit (--max-active-backends) like any pinned model: LocalAI never evicts them to make room, and if they fill the limit, a new model still loads rather than being blocked.

warm applies only to local targets. On a remote (localai-proxy or cloud-proxy) target it has no effect, and LocalAI logs a warning when it loads the chain.

Realtime pipelines

A pipeline stage can name a chain:

name: assistant
pipeline:
  vad: silero-vad
  transcription: whisper-chain
  llm: assistant-llm
  tts: voice-chain

LocalAI resolves the chain for every call of the stage, in full pipelines and in transcription-only and sound-detection-only sessions. When a chain switches, the session stays open and keeps its conversation. The next turn uses the new target.

The session receives a localai.model.failover event for each chain stage when it starts (reason: initial) and each time a chain switches:

{"type":"localai.model.failover","chain":"assistant-llm","stage":"llm",
 "from":"argus-llm","to":"gemma-local","state":"fallback","reason":"trip"}

Example: stages on a remote LocalAI

This pipeline runs its transcription, LLM and TTS stages on a remote LocalAI (argus) through localai-proxy models, and uses local models when the remote instance is down. Each stage has its own chain, so one stage can fail over while the others stay remote.

# Remote targets: each one names the model on the upstream LocalAI.
name: argus-stt
backend: localai-proxy
known_usecases: [transcript]
options:
  - realtime_pipeline:asr-pipeline   # upstream pipeline for live transcription
proxy:
  upstream_url: http://argus.lan:8080
  upstream_model: parakeet
---
name: argus-llm
backend: localai-proxy
known_usecases: [chat]
proxy:
  upstream_url: http://argus.lan:8080
  upstream_model: gemma-3-12b
---
name: argus-tts
backend: localai-proxy
known_usecases: [tts]
proxy:
  upstream_url: http://argus.lan:8080
  upstream_model: kokoro
---
# One chain per stage, remote first, local second.
name: stt-chain
failover:
  targets: [{model: argus-stt}, {model: whisper-local, warm: true}]
---
name: llm-chain
failover:
  targets: [{model: argus-llm}, {model: gemma-local, warm: true}]
---
name: tts-chain
failover:
  targets: [{model: argus-tts}, {model: piper-local}]
---
name: assistant
pipeline:
  vad: silero-vad
  transcription: stt-chain
  llm: llm-chain
  tts: tts-chain

The example shows the configs as one YAML stream; put each config in its own file in the models directory. When argus stops answering, the next call of each stage fails over to the local model and the session receives a localai.model.failover event for that stage. A remote target that does not support a call (it returns Unimplemented) is skipped for that call and is not marked down.

Limits:

• After a session.update that changes the pipeline, localai.model.failover events keep describing the chains from session start.
• A chain used as a router candidate, or as the classifier-mode scoring model, is not resolved per call.

Watching failover

• GET /api/failover lists every chain, its active target and the state of each target.
• GET /api/failover/{chain} returns one chain.
• GET /api/failover/events is a server-sent event stream. The first event is snapshot with the full state. Then chain.switched and target.state events follow.
• Metrics: localai_failover_switches_total{chain,from,to,reason} and localai_failover_target_up{target}.
• With tracing on, each skipped target appears in the Traces view with the error that made LocalAI skip it.

Pinning a target

An admin can force a chain to one target, for example during maintenance:

curl -X POST http://localhost:8080/api/failover/assistant-llm/pin \
  -H 'Content-Type: application/json' -d '{"target":"gemma-local"}'
curl -X DELETE http://localhost:8080/api/failover/assistant-llm/pin

While a chain is pinned, only the pinned target serves it. Health checks continue. On a single LocalAI instance, a restart removes the pin. In distributed mode, pins persist.

In the UI

• The model editor’s template picker has a Failover Chain template that starts a new model with two empty targets. Any model can also add a failover.targets field directly: each row picks a target model and a warm toggle, with move up/down and remove controls. The toggle is available on every row; warm has no effect on a remote (localai-proxy or cloud-proxy) target, and LocalAI logs a warning when it loads a chain with one set. The field flags fewer than two targets, a duplicate target, and the chain naming itself as one of its own targets.
• Opening an existing chain in the model editor shows a health strip: the chain’s status pill, its active target and how long it has been active, and a table of every target with its kind, warm flag, status, last probe time and last error. An admin sees a Pin button on each target and an Unpin action for the chain, both behind a confirmation dialog.
• The Failover page (/app/failover, admin only) lists every chain with its status pill, active target, a small pill per target, and time since the last switch. It links each chain name to its model editor page and shows an empty state linking to the failover template when no chains exist yet. It sits under Operate → Runtime on a single install and under Operate → Swarm when distributed mode is on. On a cluster it also describes what the router and the health monitor do when a worker stops answering, and shows a preview of what would stop if a chosen node went away, worked out in the browser from the loaded replicas and the placement rules. The server does not compute that preview.
• The Installed Models list badges a model that belongs to a chain with chain → <active target>, next to the alias badge.
• All of the above update live from the same event stream as GET /api/failover/events — the strip, the overview page and the badge do not need a page refresh to reflect a chain.switched or target.state event.

Assistant and MCP

The LocalAI Assistant and local-ai mcp-server offer list_failover_chains, pin_failover_target and unpin_failover_target. Create and edit chains with the model config tools, like any other model.

Distributed mode

In distributed mode, all frontends share one failover state:

• Pins apply to the whole cluster. LocalAI stores them in the database, so they persist across restarts. A pin set on one frontend applies on all.
• Target health and the active target of each chain are shared over NATS, so all frontends converge on the same target for a chain.
• One frontend, the probe leader, runs the health checks, decides fail-over and fail-back, and loads warm targets. The leader holds a PostgreSQL advisory lock and keeps it until it stops or its database connection fails. Then another frontend takes the lock and becomes the leader: immediately when the leader shuts down or its process exits, and within about 30 seconds when the leader’s host or network fails.
• Warm targets stay loaded on the workers. The router and the replica reconciler treat them like pinned models and do not evict them.
• A frontend that starts late gets the current state within 10 seconds, because the leader sends its full state again every 10 seconds.
• If PostgreSQL is not available, no frontend holds the lock. Health checks and fail-back stop until the database is back. Requests still fail over to the next target when a target fails during the request.
• If a frontend cannot start the shared state, it logs an error and manages failover alone, as a single LocalAI instance does.
• Leadership follows the database lock, not NATS. A leader that loses its NATS connection but keeps its database connection stays the leader. Until NATS recovers, the other frontends keep the last decisions they received from it, and its new decisions do not reach them.
• Each frontend reads the pins from the database again every 30 seconds and after a NATS reconnect, and applies any change within 10 seconds. A frontend that missed a pin or an unpin catches up in this way. If a pin cannot be written to the database, LocalAI returns an error and restores the previous pin.

Limits

• Chains do not nest.
• See also model aliases and the realtime API.

vllm.cpp backend

vllm.cpp is the LocalAI team’s C++20 port of vLLM: the same continuous-batching scheduler, paged KV cache and automatic prefix caching, with no Python at inference time. LocalAI serves it through the native vllm-cpp backend, which loads either a HuggingFace safetensors directory or a .gguf file and applies the chat template, tool-call parsing and reasoning split inside the engine.

This page covers installing the backend and the models LocalAI ships ready to run on it. For the full engine_args reference (KV sizing, scheduling policy, speculative decoding, LMCache), see the vllm.cpp section of the text generation guide.

Installing

local-ai backends install vllm-cpp

Or install it from the Backends page in the web UI. Images are published for CPU, CUDA 13, Vulkan, Metal and Jetson L4T.

Which GPUs the CUDA images cover

The CUDA images are currently built for Blackwell-family architectures only: sm_120a (RTX 50 series, RTX PRO 6000 Blackwell) and sm_121a (GB10 / DGX Spark) on x86-64, and sm_121a alone on arm64. CUDA 13 is required, so there is no CUDA 12 variant.

That is narrower than vllm.cpp itself, which builds ten architectures. On an Ampere, Ada, Hopper, Jetson Orin or Jetson Thor GPU the CUDA backend installs successfully and then fails at the first request with no kernel image is available for execution on the device. Until the build widens, use the Vulkan or CPU image on those cards: vulkan-vllm-cpp builds with CUDA off entirely and gives them a GPU path, without the NVFP4 and Marlin kernels.

Ready-made models

The model gallery carries a curated set of vllm.cpp configurations. Each one arrives with the engine settings already applied, so tool calling, the reasoning split and speculative decoding work without hand-editing YAML.

local-ai models install qwen3-0.6b-vllm-cpp

The two small bf16 entries are the ones that run anywhere the backend does, including CPU. The NVFP4 entries need a Blackwell-class NVIDIA GPU on two counts: NVFP4 has no kernel on older architectures, and the CUDA images are built only for Blackwell in any case.

Sizing note: every entry sets num_blocks to give roughly one to four full contexts of KV cache, which is a starting point rather than a tuned value. KV is not free; the 4B entry, for instance, spends 144 KiB per token, so its 1024 blocks are about 4.5 GB on top of the weights. Raise num_blocks for more concurrency, lower it on a small box.

Why the 27B entries pin a revision

The Qwen3.6-27B entries pin their weights to a specific HuggingFace commit rather than tracking the repository’s default branch. This is deliberate and worth understanding before you copy one of these configs.

The upstream repository was later re-quantized in place, under the same name, from NVFP4 to FP8 W8A8. A config that names the repository without a revision therefore resolves to entirely different weights, with different numerics and different performance, and nothing about the load reports that anything changed. Pinning is what makes the entry reproducible:

artifacts:
  - name: model
    target: model
    source:
      type: huggingface
      repo: unsloth/Qwen3.6-27B-NVFP4
      revision: 890bdef7a42feba6d83b6e17a03315c694112f2a

The same reasoning applies to any quantized community repository you depend on.

Choosing between the speculative variants

Speculative decoding trades memory for decode throughput. All three Qwen3.6-27B entries serve the same weights and produce the same quality; they differ only in how tokens are proposed.

MTP drafts one token per step from a head that already lives in the target checkpoint’s own mtp.* tensors, so it costs no extra download. DFlash drafts a whole 16-token block in one non-autoregressive pass from a separate drafter, which is the larger win at the cost of a second checkpoint on disk.

Start with the plain entry if you are short on memory, and with the DFlash entry if you are not.

Tool calling

Every entry above sets use_tokenizer_template: true and disables LocalAI’s Go-side grammar path, so tool calls are detected and parsed by the engine’s own streaming parsers and arrive as real tool_calls on the OpenAI response.

The parser is normally auto-detected from the chat template, but one case cannot be: Qwen3-Coder’s tool dialect is byte-identical on the wire to another family’s, so template sniffing would pick the wrong parser. The qwen3-coder-30b-a3b-vllm-cpp entry therefore names it explicitly, and any Qwen3-Coder config you write yourself should do the same:

engine_args:
  tool_parser: qwen3_coder

Overriding config.json keys (hf_overrides)

engine_args.hf_overrides is a JSON object of top-level config.json keys that the backend merges over the model directory’s own config.json before the engine loads it, like vLLM’s --hf-overrides. The main use is to opt a published checkpoint into an engine adapter that its config does not name. For example, the Tev1 repositories declare Qwen3_5ForConditionalGeneration, and vllm.cpp serves them as decision models only when the architecture is Tev1Model:

engine_args:
  hf_overrides:
    architectures: ["Tev1Model"]

The downloaded model files do not change. At load the backend creates a private temporary directory. It writes the merged config.json there and adds a symlink for each other entry of the model directory (weights, tokenizer files, a tokenizer/ subdirectory). Then it gives that directory to the engine. When the model unloads, the backend removes the directory.

Rules:

• The merge is top-level only. An override key replaces the whole value of that key, including a nested object such as text_config.
• The model must be a directory that contains a config.json. A .gguf file or a directory without config.json fails the load.
• A value that is not a JSON object (an array, a scalar, or JSON that does not parse) fails the load. The backend does not ignore it, because loading the unchanged config would serve a different architecture than the one you configured.
• An empty object ({}) does nothing.

Named entity recognition (GLiNER2.5)

The vllm-cpp backend serves GLiNER2.5, a zero-shot NER and structured-extraction model. Point the backend at the safetensors directory and the backend exposes the TokenClassify gRPC method, which LocalAI maps to its standard NER API surface.

Labels are supplied at inference time, not baked into the model config. Set them in engine_args:

engine_args:
  ner_labels: "person,organization,location,date,time,money,quantity"
  ner_threshold: 0.5
  ner_max_width: 12

ner_labels is a comma-separated list. When omitted, the backend falls back to a built-in default set (person, organization, location, date, time, money, quantity). ner_threshold is the sigmoid cutoff (default 0.5); ner_max_width is the maximum span length in tokens (default 12).

The model runs the DeBERTa v2 encoder with disentangled attention on the host forward, which is the required contract for pooling models in vllm.cpp. A device-resident forward is tracked as a performance optimization, not a correctness gap.

Decisions API

The vllm-cpp backend serves the kev-compatible SystemOne endpoints (the Decisions API): typed choice, noul and score questions over a state text, answered by a non-generative decision model in one pass. A decision model declares known_usecases: [decisions]. See Decisions API for the request shape, the models you can install and the access rules.

The GLiNER2.5 zero-shot NER model (token_classify) also serves /v1/systemone, through the NER path, and it is the model to use for /v1/systemone/permute and /v1/systemone/separate, which decision models refuse with a 400. It derives its NER labels from the question definitions, so no ner_labels configuration is needed.

Beyond text generation

The vllm-cpp backend also serves MiniMax-H3, which generates video and audio jointly. See Video generation.

3D Generation

LocalAI can generate textured 3D meshes from a single conditioning image via the /3d/generations endpoint, powered by the trellis2cpp backend — a C++/GGML port of Microsoft TRELLIS.2 (trellis2.cpp). The output is a binary glTF (.glb) asset with PBR materials.

Generation is image-conditioned only — there is no text-prompt path. Provide a photo or rendering of a single object (ideally on a plain background) and TRELLIS.2 reconstructs a full 3D mesh from it.

For text-conditioned animated skeletons, see 3D Animation. Both workflows appear under 3D in Studio; selecting a model changes the inputs and generation options.

Setup

Install a model from the gallery:

local-ai run trellis2-4b          # full pipeline: 1024³ cascade + PBR textures (~18 GB)
# or
local-ai run trellis2-4b-geometry # 512³ untextured geometry only (~7 GB)

The backend detects which component GGUFs are present and degrades gracefully: without the texture models it produces untextured geometry, and without the fine-flow models it falls back to a coarse marching-cubes preview.

API

• Method: POST
• Endpoint: /3d/generations

Request

The request body is JSON with the following fields:

quality selects the mesh resolution: coarse is a fast marching-cubes preview, 512 the fine dual-grid mesh, 1024 the high-resolution cascade (slow — several minutes, roughly 10 GB VRAM), and auto picks the best pipeline the installed model set supports.

background controls solid-background removal on the conditioning image before generation: auto detects border-connected near-black/near-white, keep preserves the image alpha exactly, and black/white force removal of that colour.

Backend-specific params: texture_size (UV-atlas resolution hint when atlas baking is enabled) and components (tiny removes small islands, largest keeps only the biggest connected component, all — the default — keeps everything).

Response

Returns a JSON response using LocalAI’s OpenAI-style generation envelope:

Watertight print remeshing

POST /3d/remesh applies the same post-generation CGAL Alpha Wrap workflow as the trellis2.cpp demo. It accepts multipart/form-data and returns the remeshed GLB directly as model/gltf-binary:

There is intentionally no independent offset control. The enclosing offset follows the trellis2.cpp demo and is derived as detail / 30; independent tuning tends to produce puffy or degenerate wraps. Lower detail percentages retain finer features but take longer and generally produce more triangles. The output is watertight, oriented, intersection-free, and 2-manifold. For textured sources, LocalAI unwraps the replacement mesh and reprojects its PBR material onto a new UV atlas.

Source GLBs may be up to 512 MiB. This route uses its own upload limit because fine TRELLIS.2 meshes commonly exceed LocalAI’s default --upload-limit.

curl http://localhost:8080/3d/remesh \
  -F model=trellis2-4b \
  -F mesh=@generated.glb \
  -F detail=0.5 \
  --output printable.glb

Usage

Generate a 3D model from an image

curl http://localhost:8080/3d/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "trellis2-4b",
    "image": "https://example.com/photo-of-a-chair.png",
    "quality": "512"
  }'

The response contains a URL such as /generated-3d/b64123456789.glb; fetch it from the same server. The GLB is standard glTF 2.0 and opens in Blender, three.js, <model-viewer>, and most engines.

Base64 input and output

curl http://localhost:8080/3d/generations \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"trellis2-4b\",
    \"image\": \"$(base64 -w0 chair.png)\",
    \"response_format\": \"b64_json\"
  }" | jq -r '.data[0].b64_json' | base64 -d > chair.glb

WebUI

The React UI includes a 3D tab in the Studio (and a /3d page) with an interactive PBR viewer: upload or paste an image from the clipboard, pick the quality, and preview the generated mesh with orbit/pan/zoom and a wireframe toggle. Past generations are kept in the browser (IndexedDB). After generation, a single Detail slider and Apply remeshing button replace the preview with the exact watertight model that the GLB download exports; Show original switches back without regenerating.

Notes

• The 512³ pipeline takes roughly two minutes on a modern GPU; the 1024³ cascade takes around five minutes and needs about 10 GB VRAM plus a temporary host-RAM spike.
• TRELLIS2_DEVICE=cpu forces CPU inference (slow; mainly for debugging).
• The generated mesh has unoriented winding (faithful to TRELLIS.2) and is exported Y-up with vertex-PBR materials; a UV-atlas texture bake can be enabled in the backend via the T2GLB_XATLAS environment variable.

3D Animation

LocalAI supports text-to-motion with kimodo.cpp on CPU and Vulkan. In Studio → 3D, selecting an animation model replaces the image input with a motion prompt and the selected model’s animation controls.

The Studio preview plays at real-time speed using the GLB’s timestamps. Slow or delayed display frames skip ahead rather than slowing the animation. Pause and the timeline let you inspect individual poses; playback resumes from that point.

Kimodo exports an animated skeleton GLB: joint rotations and root movement, without a mesh or skin. SOMA models produce the compact 30-joint skeleton; G1 models produce 34 joints. These outputs are animation assets, not textured characters, and cannot use TRELLIS print remeshing.

Setup and runtime options

Install kimodo-soma-rp, kimodo-soma-seed, kimodo-g1-rp, or kimodo-g1-seed from the model gallery. For example:

local-ai run kimodo-soma-rp

Importing the corresponding LocalAI-io/Kimodo-*-GGML Hugging Face repository also installs the motion weights and the shared Llama-3-Kimodo text bundle. Each installation needs three files: the motion GGUF, a monolithic text-encoder GGUF, and tokenizer.gguf. Text weights and the tokenizer are shared under kimodo/text; SOMA and G1 do not download separate copies of the same encoder. SMPL-X weights are not included in the gallery because their redistribution terms differ.

The default entries use Q8_0, matching upstream’s performance default. Other text-encoder quantizations are available through the -q6_k, -q5_k, -q4_k_m, -q4_k, and -bf16 gallery entries, for example kimodo-g1-rp-q4_k_m. Quantization affects the text encoder, not the F32 motion weights. Lower-bit encoders reduce memory use but can change the generated motion. These are explicit choices, not automatically ranked alternatives to Q8_0.

The model importer also accepts a text_quantization preference with q8_0 (default), q6_k, q5_k, q4_k_m, q4_k, or bf16. To change an existing installation manually, download the selected encoder beside tokenizer.gguf and update text_bundle to its GGUF path.

Linux has CPU and Vulkan builds on amd64 and arm64. Apple Silicon uses the CPU build; this upstream engine does not implement Metal. Linux x86 CPU packages select AVX2/FMA/F16C/BMI2 where supported, with a portable fallback. Containers must expose the host GPU’s Vulkan ICD and driver libraries, not just CUDA compute devices. On NVIDIA, enable graphics driver capabilities alongside compute; on NixOS CDI installations, check that the NVIDIA Vulkan manifest is also mounted inside the container.

An installed model can override the defaults in its YAML configuration:

backend: kimodocpp
parameters:
  model: kimodo/kimodo-soma-rp-v1.1-f32.gguf
threads: 8
options:
  - text_bundle:kimodo/text/Llama-3-Kimodo-Q8_0.gguf
  - device:auto
  - text_layer_chunk:32
  - frames:150
  - steps:100
  - text_guidance:2

device accepts auto, cpu, or vulkan; explicitly requesting unavailable Vulkan fails rather than silently using CPU. threads applies to both the text encoder and motion model. text_layer_chunk accepts 1–32 and defaults to 32: all text-encoder layers stay resident across requests for maximum throughput. Set it to 8 (or another smaller value) to load fewer layers at a time when memory is limited. Full residency still executes bounded eight-layer graphs; resident weight count and execution graph size are separate concerns.

The backend keeps one native session loaded and serializes generation. It reuses text and motion weights, packed attention/LoRA paths, and upstream’s cached motion execution graphs instead of reloading and rebuilding per request. The graph cache is bounded to the most recent batch/frame shape. Cold loading therefore takes longer than subsequent generations.

Allow memory for motion weights and execution buffers as well as the encoder. BF16 text weights alone are about 14.1 GiB and should use a smaller text_layer_chunk on a 16 GiB GPU. Each loaded motion model has its own native session and resident memory, even when model files are shared on disk. Upstream’s optional KIMODO_TEXT_RESIDENT_LIMIT_MIB environment variable can also impose a residency ceiling; above it, a request for 32 layers uses streaming.

Existing text_bundle directory paths remain supported for legacy layer-file installations. They also adopt the new 32-layer default unless explicitly configured otherwise. Switching to monolithic weights does not delete the old files; remove them only after confirming no model configuration still uses them.

API

POST /3d/animate accepts named, typed conditioning inputs and model-specific parameters. Query /v1/models/capabilities and inspect three_d_operations for the selected model’s supported inputs, controls, and defaults. Animation does not universally require text: other backends may declare video or a combination of mesh and text inputs.

{
  "model": "kimodo-soma-rp",
  "inputs": {
    "prompt": {"type": "text", "data": "A person walks forward and waves."}
  },
  "params": {"frames": "150", "steps": "100", "text_guidance": "2", "seed": "42"},
  "response_format": "url"
}

The response contains data[0].url for the generated .glb, or data[0].b64_json when response_format is b64_json. The endpoint uses the existing 3D Generation permission and the selected model’s access controls. /3d/generations and /3d/remesh retain their existing mesh workflows.

Kimodo accepts one prompt with 60–150 frames at 30 FPS. The reference defaults are 150 frames, 100 sampling steps, and text guidance 2. Parameters are strings; unsupported inputs and parameters are rejected. Multi-prompt transitions and mesh retargeting are not currently exposed by this adapter.

Request traces

With tracing enabled, POST /3d/animate appears on the Traces page in both API and backend history. API traces include the JSON request, response, HTTP status, duration, and any returned error. Sensitive headers are redacted and body capture follows the configured size limit.

The backend trace includes the model and backend, text prompt, requested parameters, model-loading time (load_ms), slot-waiting time (queue_ms), inference time (inference_ms), output file size (output_bytes), and returned metadata, including the usage breakdown. Parameters describe request overrides; metadata.usage.details reports the effective frames and sampling steps. Non-text inputs record their type and presence, without copying asset contents. Trace strings follow the existing backend trace size limit.

A running backend entry appears before model loading begins. Completed entries contain the details above; failures include the error and the stage that failed. Total backend duration includes loading, waiting, and inference. Requests rejected by endpoint validation have an API trace but no animation backend trace.

Usage accounting

Kimodo reports one set of usage measurements through generic backend metadata. The gRPC Result has a metadata bytes field containing a JSON object; there is no separate gRPC usage field or model-specific usage message.

Backend metadata

The JSON stored in Result.metadata looks like this:

{
  "usage": {
    "input_units": 32,
    "output_units": 15000,
    "accounting_rule": "frame_steps_v1",
    "details": {
      "output_frames": 150,
      "sampling_steps": 100
    }
  }
}

usage is an application convention inside the generic metadata object. Other metadata keys and model-specific details can be added without changing the gRPC definitions or database schema.

For kimodo, input units are the tokens produced by its text encoder’s tokenizer, including the beginning-of-sequence (BOS) token. Output units are actual generated frames multiplied by effective sampling steps, after applying model defaults and request overrides. In the example, 150 frames × 100 steps = 15,000 output units.

HTTP response and recorded usage

For POST /3d/animate, LocalAI returns the backend object under metadata. Usage appears only at metadata.usage; there is no top-level usage summary. The following is an excerpt from either a url or b64_json response:

{
  "metadata": {
    "usage": {
      "input_units": 32,
      "output_units": 15000,
      "accounting_rule": "frame_steps_v1",
      "details": {
        "output_frames": 150,
        "sampling_steps": 100
      }
    }
  }
}

Internal accounting reads metadata.usage directly and records each successful request once. The existing token-named database columns store the unit counts:

The record’s single Metadata column retains the complete backend JSON, including the rule and frame/step breakdown. With statistics disabled, response metadata is still returned but no usage record is created.

Both unit counts must be present, nonnegative integers whose sum fits a Go int. Explicit zero counts are valid. If metadata is absent, the response omits metadata. Metadata without a usage member is returned, but creates no usage record. Malformed metadata or invalid usage counts cause metadata to be omitted and a warning to be logged; the generated asset is still returned. Failed generations produce no usage record. Counts are never estimated for backends that do not report them.

Pricing units

Kimodo’s output units are frame-steps, not text tokens. They approximate computational work rather than runtime or hardware cost. A price per million output tokens therefore means dollars per million frame-steps for this model:

cost = (input_units × input_price + output_units × output_price) / 1,000,000

This describes how to interpret the existing token-based pricing dimensions; it does not set a price or charge money automatically.

Agents

LocalAI no longer has a separate agent platform. The agents, the agent jobs and the old chat page were replaced by the chats, which keep a conversation on the server and run it with the same loop for every purpose.

The collections (the knowledge base) stay. A saved agent lists the collections it may search.

What the server did with your old data at the first start, and what it could not carry over, is on Moving from the old chat and agents.

Chats

Chats are conversations that the server keeps. You can keep typing while the model works, the model can hand work to sub-agents, long conversations are compacted for you, and the model can use the MCP servers of its model configuration.

The chats API sits beside the OpenAI-compatible /v1/chat/completions endpoint, which does not change. Use /v1 when a client brings its own history and needs the OpenAI format. Use /api/chats when the server should keep the conversation.

Permissions

With authentication on, an administrator grants features to each user in Settings → Users and keys. Plain chats are on by default. The parts that run without a person, or reach the outside world, are off by default for a user, and an administrator always has all of them:

A command-line MCP server (stdio) in your MCP connectors stays for administrators only, whatever the features. A trigger or a connector whose owner loses its feature stops running: a trigger run is skipped with the reason, and the bot says it is not available. The owner can get the feature back without losing the definition.

Upgrading. On the first start after the features were split, a one-time rule keeps what each existing user was using, and nothing else. The rule runs once; a user who registers later gets the defaults. A value an administrator stored always wins.

• agents: kept if the user has the stored agents permission (the one the old agent pool had) or owns a saved agent.
• triggers: kept if the user owns a trigger, or had agents or mcp_jobs.
• connectors: kept if the user owns a connector binding, or had agents.
• mcp_jobs: stored as it was.

Items that the conversion of the old agents makes after that first start belong to users who had the old permissions, which the rule already read.

What an administrator can see

An administrator sees what every user has set up, each with its owner, and can stop it. The view is read-only apart from switch-off and delete, and never shows a bot token or an agent’s instructions:

The agent tasks and jobs have their own view: ?all_users=true on GET /api/agent/tasks and GET /api/agent/jobs (see Agent tasks and jobs).

Switch the chats off

The operator can switch the whole surface or one part off. A part that is off starts nothing: no loop, no trigger runner, no bot. GET /api/features reports what runs (chats, agents, triggers, connectors).

LOCALAI_DISABLE_COLLECTIONS stays separate and turns off only the collections.

Start a chat and send messages

# A chat talks to one model, chosen when it starts.
curl -s localhost:8080/api/chats \
  -H 'Content-Type: application/json' \
  -d '{"model": "my-model"}'
# {"id":"3f1c...","model":"my-model","running":false,...}

curl -s localhost:8080/api/chats/3f1c.../messages \
  -H 'Content-Type: application/json' \
  -d '{"text": "Summarize the three files I mentioned."}'
# {"started":true}

# Rename a chat. This answers 409 while a turn runs.
curl -s -X PATCH localhost:8080/api/chats/3f1c... \
  -H 'Content-Type: application/json' \
  -d '{"title": "Trip plans"}'

The reply is not in that response. Read it from the event stream or from the transcript:

# Live server-sent events: text, reasoning, tool_start, tool_end, turn_end, files, ...
curl -N localhost:8080/api/chats/3f1c.../events

# The transcript, in order. Read it first, then open the event stream: the
# stream carries only what happens after you connect.
curl -s 'localhost:8080/api/chats/3f1c.../messages?after=0'

# After a dropped connection, send the id of the last event you got.
curl -N 'localhost:8080/api/chats/3f1c.../events?after=9d2e41.57'

The events of a running turn have an id, <run>.<n>, sent as the SSE id field. A client that lost its connection sends the last id back in the after query parameter, or in the Last-Event-ID header (a browser sends that header when it reconnects by itself; the header wins). The server first sends the events it missed, then the live ones. Deltas of text and reasoning that came one after the other are not sent again, and the ones the client had are cut off. The server keeps the last 2048 events or 4 MiB of each running chat, and the ring stays for 30 seconds after the turn ends, so a client that reconnects at once also gets turn_end. Events that do not come from the turn itself (files, goal, loops, cleared and the cards) have no id and are not replayed: they tell the client to read the record again.

When the server cannot give what was missed, the stream starts with a resync event. This happens when the id is of another turn, is older than the ring holds, or is not an id of this server, and in a cluster when the server that runs the turn is gone or does not answer in 2 seconds. Read the transcript, then go on: the resync event carries the id of the newest event known, when there is one, and the live events follow it. A gap in the numbers of the live events is filled from the ring in the same way.

A user can have 8 event streams open on one chat on one server; one more gets HTTP 429 until one closes. The counter localai_chat_event_resyncs_total counts the reconnects that got a resync, by reason: local (the ring of this server did not have the events), owner (the ring of the server that runs the turn did not have them), unreachable (that server did not answer) and no_owner (no other server is known to run the turn). A steady rate of local or owner says that the ring is too small for your clients’ gaps.

The transcript lists messages, tool lines and one checkpoint line per turn. Undo works per tool call, not per turn: each write_file, edit_file or move_file call is its own change set in your files, and its tool line carries a card {"kind": "files_change", "id": "<rev>:<path>"} (delete_file moves to the Trash, where you restore it). POST /api/chats/files/undo {"rev": N} undoes that change set; the chat page uses this for the Undo of its change notes (see Make LocalAI yours). To undo a whole turn, undo the changes of its tool lines, newest first. The checkpoint line marks the end of the turn; its rev is 0 and its text is empty, because the turn’s changes are already kept call by call.

A chat is named after the first line of its first message, unless you gave it a title. To rename it, see Settings, rename and model change.