Embeddings

LocalAI supports generating embeddings for text or list of tokens.

For face embeddings specifically, see the Face Recognition feature - it produces 512-d L2-normalized vectors tuned for face similarity.

For the API documentation you can refer to the OpenAI docs: https://platform.openai.com/docs/api-reference/embeddings

Model compatibility

The embedding endpoint is compatible with llama.cpp models, bert.cpp models and sentence-transformers models available in huggingface.

Using Gallery Models

LocalAI provides a model gallery with pre-configured embedding models. To use a gallery model:

1. Ensure the model is available in the gallery (check Model Gallery)
2. Use the model name directly in your API calls

Example gallery models:

• qwen3-embedding-4b - Qwen3 Embedding 4B model
• qwen3-embedding-8b - Qwen3 Embedding 8B model
• qwen3-embedding-0.6b - Qwen3 Embedding 0.6B model

Example: Using Qwen3-Embedding-4B from Gallery

curl http://localhost:8080/embeddings -X POST -H "Content-Type: application/json" -d '{
  "input": "My text to embed",
  "model": "qwen3-embedding-4b",
  "dimensions": 2560
}'

Manual Setup

Create a YAML config file in the models directory. Specify the backend and the model file.

name: text-embedding-ada-002 # The model name used in the API
parameters:
  model: <model_file>
backend: "<backend>"
embeddings: true

Huggingface embeddings

To use sentence-transformers and models in huggingface you can use the sentencetransformers embedding backend.

name: text-embedding-ada-002
backend: sentencetransformers
embeddings: true
parameters:
  model: all-MiniLM-L6-v2

The sentencetransformers backend uses Python sentence-transformers. For a list of all pre-trained models available see here: https://github.com/UKPLab/sentence-transformers#pre-trained-models

Llama.cpp embeddings

Embeddings with llama.cpp are supported with the llama-cpp backend, it needs to be enabled with embeddings set to true.

name: my-awesome-model
backend: llama-cpp
embeddings: true
parameters:
  model: ggml-file.bin

Then you can use the API to generate embeddings:

curl http://localhost:8080/embeddings -X POST -H "Content-Type: application/json" -d '{
  "input": "My text",
  "model": "my-awesome-model"
}' | jq "."

Embedding chat conversations and Go-side pooling

/v1/embeddings also accepts a chat conversation via messages (a LocalAI extension), plus a per-request pooling scheme that LocalAI applies itself to the backend’s raw per-token vectors:

curl http://localhost:8080/v1/embeddings -X POST -H "Content-Type: application/json" -d '{
  "model": "my-awesome-model",
  "messages": [
    {"role": "system", "content": "You are a support agent."},
    {"role": "user", "content": "My invoice is wrong."}
  ],
  "pooling": "decayed_mean",
  "pooling_half_life_tokens": 256
}'

• One conversation per request; the response is the standard OpenAI embeddings shape with a single data[0].embedding item.
• input and messages are mutually exclusive (400 otherwise); an unknown pooling value is also a 400.
• If the model config carries both template.chat and template.chat_message, the conversation renders exactly like a chat prompt, so the embedding matches what a chat model would actually see. Otherwise a frozen role-prefixed fallback is used (<role>: <content> lines joined by newlines, empty-content messages skipped). Non-text content parts (images, audio, video) are ignored.

pooling selects how the per-token vectors are reduced to one embedding:

Go-side schemes need raw per-token vectors from the backend. Each backend declares whether an embedding result is final or per-token; LocalAI rejects a Go-side scheme for a final vector and rejects backend pass-through for a per-token matrix instead of guessing from its shape. Older backends that do not declare a layout remain compatible with backend pooling only.

llama.cpp chooses this layout when the model is loaded. LocalAI automatically adds the pooling:none backend option when a llama.cpp model sets a Go-side parameters.pooling scheme. That raw-loaded instance can switch between mean, last, and decayed_mean per request, but it cannot switch back to backend pooling without reloading. Conversely, a backend-pooled llama.cpp instance rejects per-request Go pooling. Other backends may support Go-side pooling when they explicitly return per-token vectors.

After Go-side pooling, the vector is normalized with llama.cpp’s embd_normalize rule (default L2; configurable through options: ["embd_normalize:<n>"]).

Model-level defaults live under parameters::

name: conversation-embedder
backend: llama-cpp
embeddings: true
parameters:
  model: ggml-file.bin
  pooling: decayed_mean
  pooling_half_life_tokens: 256

Go-side pooling requires an up-to-date backend that reports its embedding layout. A legacy backend fails closed for Go-side schemes with an error asking you to rebuild or update it.

💡 Examples

• Example that uses LLamaIndex and LocalAI as embedding: here.

⚠️ Common Issues and Troubleshooting

Issue: Embedding model not returning correct results

Symptoms:

• Model returns empty or incorrect embeddings
• API returns errors when calling embedding endpoint

Common Causes:

1. Incorrect model filename: Ensure you’re using the correct filename from the gallery or your model file location.

   • Gallery models use specific filenames (e.g., Qwen3-Embedding-4B-Q4_K_M.gguf)
   • Check the Model Gallery for correct filenames

2. Context size mismatch: Ensure your context_size setting doesn’t exceed the model’s maximum context length.

   • Qwen3-Embedding-4B: max 32k (32768) context
   • Qwen3-Embedding-8B: max 32k (32768) context
   • Qwen3-Embedding-0.6B: max 32k (32768) context

3. Missing embeddings: true flag: The model configuration must have embeddings: true set.

Correct Configuration Example:

name: qwen3-embedding-4b
backend: llama-cpp
embeddings: true
context_size: 32768
parameters:
  model: Qwen3-Embedding-4B-Q4_K_M.gguf

Issue: Dimension mismatch

Symptoms:

• Returned embedding dimensions don’t match expected dimensions

Solution:

• Use the dimensions parameter in your API request to specify the output dimension
• Qwen3-Embedding models support dimensions from 32 to 2560 (4B) or 4096 (8B)

curl http://localhost:8080/embeddings -X POST -H "Content-Type: application/json" -d '{
  "input": "My text",
  "model": "qwen3-embedding-4b",
  "dimensions": 1024
}'

Issue: Model not found

Symptoms:

• API returns 404 or “model not found” error

Solution:

• Ensure the model is properly configured in the models directory
• Check that the model name in your API request matches the name field in the configuration
• For gallery models, ensure the gallery is properly loaded

Qwen3 Embedding Models Specifics

The Qwen3 Embedding series models have these characteristics:

All models support:

• User-defined output dimensions (32 to max dimensions)
• Multilingual text embedding (100+ languages)
• Instruction-tuned embedding with custom instructions