Fine-tune a model

LocalAI supports fine-tuning LLMs directly through the API and Web UI. Fine-tuning is powered by pluggable backends that implement a generic gRPC interface, allowing support for different training frameworks and model types.

Supported Backends

Availability

The fine-tuning API and the training backends are part of LocalAI. The Fine-Tune page and the chat tools are in the fine-tuning add-on (see Fine-tuning is an add-on). With --addons-enabled=false every path under /api/fine-tuning answers 503. When authentication is enabled, fine-tuning is a per-user feature (default OFF). Admins can enable it for specific users via the user management API.

Quick Start

1. Start a fine-tuning job

curl -X POST http://localhost:8080/api/fine-tuning/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    "backend": "trl",
    "training_method": "sft",
    "training_type": "lora",
    "dataset_source": "yahma/alpaca-cleaned",
    "num_epochs": 1,
    "batch_size": 2,
    "learning_rate": 0.0002,
    "adapter_rank": 16,
    "adapter_alpha": 16,
    "extra_options": {
      "max_seq_length": "512"
    }
  }'

2. Monitor progress (SSE stream)

curl -N http://localhost:8080/api/fine-tuning/jobs/{job_id}/progress

3. List checkpoints

curl http://localhost:8080/api/fine-tuning/jobs/{job_id}/checkpoints

4. Export model

curl -X POST http://localhost:8080/api/fine-tuning/jobs/{job_id}/export \
  -H "Content-Type: application/json" \
  -d '{
    "export_format": "gguf",
    "quantization_method": "q4_k_m",
    "output_path": "/models/my-finetuned-model"
  }'

API Reference

Endpoints

GET /api/fine-tuning/status and GET /api/quantization/status return {"model_exports": true|false}. The value is false only when this instance does not see the models path of the add-on owner; it is the same value as the model_exports key of GET /api/features. The Web UI disables the export controls while it is false. Both routes need an authenticated user and answer 503 when add-ons are off.

Job Request Fields

Backend-Specific Options (extra_options)

TRL

DPO-specific (training_method=dpo)

GRPO-specific (training_method=grpo)

GRPO Reward Functions

GRPO training requires reward functions to evaluate model completions. Specify them via the reward_functions field (a typed array) or via extra_options["reward_funcs"] (a JSON string).

Built-in Reward Functions

Inline Custom Reward Functions

You can provide custom reward function code as a Python function body. The function receives completions (list of strings) and **kwargs, and must return list[float].

The function is given the reduced builtin set (len, int, float, str, list, dict, range, enumerate, zip, map, filter, sorted, min, max, sum, abs, round, any, all, isinstance, print) plus the re, math, json, and string modules, and is compiled and validated at job start (fail-fast on syntax errors). Treat these as ergonomics, not as an isolation boundary.

Example API Request

curl -X POST http://localhost:8080/api/fine-tuning/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "backend": "trl",
    "training_method": "grpo",
    "training_type": "lora",
    "dataset_source": "my-reasoning-dataset",
    "num_epochs": 1,
    "batch_size": 2,
    "learning_rate": 5e-6,
    "reward_functions": [
      {"type": "builtin", "name": "reasoning_accuracy_reward"},
      {"type": "builtin", "name": "format_reward"},
      {"type": "builtin", "name": "length_reward", "params": {"target_length": "200"}},
      {"type": "inline", "name": "think_presence", "code": "return [1.0 if \"<think>\" in c else 0.0 for c in completions]"}
    ],
    "extra_options": {
      "num_generations": "4",
      "max_completion_length": "256"
    }
  }'

Export Formats

GGUF Quantization Methods

q4_k_m, q5_k_m, q8_0, f16, q4_0, q5_0

Web UI

The Fine-Tune page is one page that follows the job from set-up to result. A line at the top shows where you are: set up, check, run, result.

1. Set up - Choose the base model, the dataset (a Hugging Face id or an uploaded file), the kind of training (an adapter, or the full model) and the epochs, batch size and learning rate. Method, backend, adapter settings, optimizer, evaluation, reward functions (GRPO) and extra options are under More options.
2. Check before you start - A list of what is known before the job starts, redrawn as you change the form: whether the model and dataset are set, whether a fine-tuning backend is installed, how much GPU memory (or RAM, when there is no GPU) is free, how much space is free on the models disk, and whether a token is set. LocalAI does not estimate how much memory a job needs or how long it takes, because the server cannot know that before the job starts, so the page says so. A missing model or dataset blocks the start. A warning, such as a missing backend or no GPU, does not.
3. Run - Percent, step, epoch, the server’s time estimate and tokens per second, the stages the job passes through, a chart of loss (with the evaluation loss as hollow dots), learning rate and gradient norm, and a log of what the job reported since the page opened. Stop asks whether to keep a checkpoint.
4. Result - A failed job shows the server’s message. When the message says memory ran out, the page offers two changes (a batch size of 1 and gradient checkpointing) that are applied to a copy of the setup and start nothing until you press Start. A finished job lists its checkpoints and exports the result as a model, then links to a chat with it and to Models.

Earlier jobs are listed below the form. Reuse puts a job’s setup back in the form.

A user needs the fine-tuning permission. Without it, the page says the account cannot fine-tune and who can change that.

Fine-tuning is an add-on

The Fine-Tune page is the fine-tuning add-on. It does not ship with LocalAI. An administrator installs it from Find add-ons with Install and turn on. Once it is on, it opens from My add-ons at /app/addons/fine-tuning/jobs and can be shown in the sidebar.

Open the canonical add-on page from My add-ons. Historical standalone links are not registered.

The API does not change. The page calls /api/fine-tuning/* with the signed-in user’s own session, and the server checks the fine-tuning permission on every call, so the add-on never widens what a user may do.

With --addons-enabled=false this feature is off, like every add-on. The UI shows no entry for it, /app/addons/fine-tuning/jobs leads back to /app, and every path under /api/fine-tuning answers 503. See The addons feature gate.

The add-on also offers tools to the chat: list jobs, propose a job, store a small dataset, read the status and the checkpoints of a job, stop a job and export a checkpoint as a model. A tool can only control a job that a tool started. A job started in the page is listed, but a tool cannot stop or export it. In the chats, and in an agent or a trigger whose agent names the add-on, a tool call that starts a job runs at once, because you turned the add-on on; a trigger with no agent gets no such tool; the result says what an undo does (see below). On the OpenAI-compatible API and other surfaces, a tool only proposes the action and a person confirms it in an approval card. A model or a dataset given as a local path is refused. The tools call the engine through the host operations (see Host operations), which the host setting LOCALAI_ADDON_HOST_OPS grants to this add-on by the finetune. prefix, and they run with the permissions of the person in the chat.

Undo is the safety net for a job a chat started. Starting a job keeps a change set in the add-on, and the chat shows an Undo button on it. An undo stops the job while it still runs; it does not save a checkpoint. If the job already finished, was stopped, or was exported, the answer is cannot_undo: a finished job cannot be un-run, and the engine offers no operation that removes its output, so nothing is deleted. A change set can be undone for 7 days and only once. Stopping a job, exporting a checkpoint and storing a dataset cannot be undone, and their results say so with the reason. The undo is the person’s own action from a browser tab; no model, agent or API key can call it. The change set lives in the add-on’s own data, with the rest of its journal, so it travels with the add-on’s backups and goes with its data when that is purged. In a cluster the undo goes through the add-on’s owner like every other call.

Dataset Formats

Datasets should follow standard HuggingFace formats:

• SFT: Alpaca format (instruction, input, output fields) or ChatML/ShareGPT
• DPO: Preference pairs (prompt, chosen, rejected fields)
• GRPO: Prompts with reward signals

Supported file formats: .json, .jsonl, .csv

Architecture

Fine-tuning uses the same gRPC backend architecture as inference:

1. Proto layer: FineTuneRequest, FineTuneProgress (streaming), StopFineTune, ListCheckpoints, ExportModel
2. Python backends: Each backend implements the gRPC interface with its specific training framework
3. Go service: Manages job lifecycle, routes API requests to backends
4. REST API: HTTP endpoints with SSE progress streaming
5. React UI: Configuration form, real-time training monitor, export panel

Quantize a model

LocalAI supports model quantization directly through the API and Web UI. Quantization converts HuggingFace models to GGUF format and compresses them to smaller sizes for efficient inference with llama.cpp.

Supported Backends

Quick Start

1. Start a quantization job

curl -X POST http://localhost:8080/api/quantization/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "model": "unsloth/functiongemma-270m-it",
    "quantization_type": "q4_k_m"
  }'

2. Monitor progress (SSE stream)

curl -N http://localhost:8080/api/quantization/jobs/{job_id}/progress

3. Download the quantized model

curl -o model.gguf http://localhost:8080/api/quantization/jobs/{job_id}/download

4. Or import it directly into LocalAI

curl -X POST http://localhost:8080/api/quantization/jobs/{job_id}/import \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-quantized-model"
  }'

API Reference

Endpoints

GET /api/fine-tuning/status and GET /api/quantization/status return {"model_exports": true|false}. The value is false only when this instance does not see the models path of the add-on owner; it is the same value as the model_exports key of GET /api/features. The Web UI disables the import controls while it is false. Both routes need an authenticated user and answer 503 when add-ons are off.

Job Request Fields

Extra Options

Import Request Fields

Job Status Values

Progress Stream

The GET /api/quantization/jobs/:id/progress endpoint returns Server-Sent Events (SSE) with JSON payloads:

{
  "job_id": "abc-123",
  "progress_percent": 65.0,
  "status": "quantizing",
  "message": "[ 234/ 567] quantizing blk.5.attn_k.weight ...",
  "output_file": "",
  "extra_metrics": {}
}

When the job completes, output_file contains the path to the quantized GGUF file and extra_metrics includes file_size_mb.

Quantization Types

The UI also supports entering a custom quantization type string for any format supported by llama-quantize.

Web UI

The Quantize page uses the same page pattern as fine-tuning: set up, check, run, result.

1. Set up - Choose the model and the quantization type (a preset, or a custom type). The backend and the Hugging Face token are under More options.
2. Check before you start - Whether a model and type are set, whether a quantization backend is installed, the free RAM and the free space on the models disk. The server reports no model size before the job starts, so the page does not estimate one.
3. Run - Percent, the stage (downloading, converting, quantizing), and a log of what the job reported since the page opened. Stop ends the job.
4. Result - A failed job shows the server’s message. A finished job shows the output file, downloads it, or imports it into LocalAI under a name you choose, then links to a chat with it and to Models.

Earlier jobs are listed below the form, with Reuse and Delete (after a confirmation).

A user needs the quantization permission. Without it, the page says the account cannot quantize.

Quantization is an add-on

The Quantize page is the quantization add-on. It does not ship with LocalAI. An administrator installs it from Find add-ons with Install and turn on. Once it is on, it opens from My add-ons at /app/addons/quantization/jobs and can be shown in the sidebar.

Open the canonical add-on page from My add-ons. Historical standalone links are not registered.

The API does not change. The page calls /api/quantization/* with the signed-in user’s own session, and the server checks the quantization permission on every call.

With --addons-enabled=false this feature is off, like every add-on. The UI shows no entry for it, /app/addons/quantization/jobs leads back to /app, and every path under /api/quantization answers 503. See The addons feature gate.

The add-on also offers tools to the chat: list jobs, propose a job, read the status of a job, stop a job and import the result as a model. A tool can only control a job that a tool started. A job started in the page is listed, but a tool cannot stop or import it. In the chats, an agent and a trigger, a tool call that starts a job runs at once, because you turned the add-on on for that chat; the result says what an undo does (see below). On the OpenAI-compatible API and other surfaces, a tool only proposes the action and a person confirms it in an approval card. A model given as a local path is refused. The tools call the engine through the host operations (see Host operations), which the host setting LOCALAI_ADDON_HOST_OPS grants to this add-on by the quantize. prefix, and they run with the permissions of the person in the chat.

Undo is the safety net for a job a chat started. Starting a job keeps a change set in the add-on, and the chat shows an Undo button on it. An undo stops the job while it still runs. If the job already finished, was stopped, or was imported, the answer is cannot_undo: a finished job cannot be un-run, and the engine offers no operation that removes its files, so nothing is deleted. A change set can be undone for 7 days and only once. Stopping a job and importing the result cannot be undone, and their results say so with the reason. The undo is the person’s own action from a browser tab; no model, agent or API key can call it. The change set lives in the add-on’s own data, with the rest of its journal, so it travels with the add-on’s backups and goes with its data when that is purged. In a cluster the undo goes through the add-on’s owner like every other call.

Architecture

Quantization uses the same gRPC backend architecture as fine-tuning:

1. Proto layer: QuantizationRequest, QuantizationProgress (streaming), StopQuantization
2. Python backend: Downloads model, runs convert_hf_to_gguf.py and llama-quantize
3. Go service: Manages job lifecycle, state persistence, async import
4. REST API: HTTP endpoints with SSE progress streaming
5. React UI: Configuration form, real-time progress monitor, download/import panel