Text and retrieval
Text generation, grammars, thinking models, embeddings, reranking, moderation, forecasting and benchmarks.
Generate text with LocalAI through the OpenAI-compatible API, with llama.cpp and other backends.
Overview The chat endpoint supports the grammar parameter, which allows users to specify a grammar in Backus-Naur Form (BNF). This feature enables the Large Language Model (LLM) to generate outputs adhering to a user-defined schema, such as JSON, YAML, or any other format that can be defined using BNF. For more details about BNF, see Backus-Naur Form on Wikipedia.
Interleaved Thinking with Tool Calls
Reasoning models can “think” before they answer. When such a model also calls a tool, the useful behaviour is for the thinking and the tool call to travel together, and for that thinking to survive the tool-result round trip. LocalAI calls this interleaved thinking: a single assistant turn carries both reasoning and tool_calls, and the client hands the reasoning back on the next turn so the model’s chain of thought is not lost when the tool result is appended.
Make vector embeddings with the embeddings API, with llama.cpp or the transformers backend.
A reranking model, often referred to as a cross-encoder, is a core component in the two-stage retrieval systems used in information retrieval and natural language processing tasks. Given a query and a set of documents, it will output similarity scores. We can use then the score to reorder the documents by relevance in our RAG system to increase its overall accuracy and filter out non-relevant results.
LocalAI exposes an OpenAI-compatible text moderation endpoint at POST /v1/moderations. It uses a local text-generation model with a constrained JSON grammar, so no separate moderation service or cloud API is required. curl http://localhost:8080/v1/moderations \ -H "Content-Type: application/json" \ -d '{ "model": "your-instruct-model", "input": "Text to classify" }' input may be one string or an array of strings. The response contains one result per input with flagged, categories, category_scores, and category_applied_input_types fields. The category names match the OpenAI moderation API, including harassment, hate, illicit activity, self-harm, sexual content, and violence categories.
LocalAI forecasts numeric time series with timestral.cpp, a C++/ggml port of Mistral’s Timestral models. You send one or more series, with optional covariates, to POST /v1/forecast and get quantile trajectories for the next steps. A forecast model is not a chat model. It has its own usecase, forecast, and does not appear in the chat model picker. It shows in the gallery filter and in the /v1/models capability list. Licence. The Timestral weights use Mistral’s Modified MIT licence. You may not use them if the global consolidated monthly revenue of your company (or your employer) exceeded USD 20 million in the preceding month, unless you get a commercial licence from Mistral AI. The restriction covers derivatives, so it also applies to the GGUF conversions in the gallery. Each gallery entry installs the licence text next to the weights. The timestral.cpp code is MIT.
Use local-ai benchmark to compare text inference through a running LocalAI server. Install and configure the models first. The command sends sequential, non-streaming requests to /v1/chat/completions and reports full request latency and end-to-end completion tokens per second. Pass one or more configured model names. To compare backends, configure separate model aliases with the desired backend, then pass those aliases. Backend names alone are not model names. The command does not install, discover, or unload models; the server’s normal loading and eviction settings still apply.