Model Quantization

LocalAI supports model quantization directly through the API and Web UI. Quantization converts HuggingFace models to GGUF format and compresses them to smaller sizes for efficient inference with llama.cpp.

Supported Backends

Quick Start

1. Start a quantization job

curl -X POST http://localhost:8080/api/quantization/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "model": "unsloth/functiongemma-270m-it",
    "quantization_type": "q4_k_m"
  }'

2. Monitor progress (SSE stream)

curl -N http://localhost:8080/api/quantization/jobs/{job_id}/progress

3. Download the quantized model

curl -o model.gguf http://localhost:8080/api/quantization/jobs/{job_id}/download

4. Or import it directly into LocalAI

curl -X POST http://localhost:8080/api/quantization/jobs/{job_id}/import \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-quantized-model"
  }'

API Reference

Endpoints

GET /api/fine-tuning/status and GET /api/quantization/status return {"model_exports": true|false}. The value is false only when this instance does not see the models path of the add-on owner; it is the same value as the model_exports key of GET /api/features. The Web UI disables the import controls while it is false. Both routes need an authenticated user and answer 503 when add-ons are off.

Job Request Fields

Extra Options

Import Request Fields

Job Status Values

Progress Stream

The GET /api/quantization/jobs/:id/progress endpoint returns Server-Sent Events (SSE) with JSON payloads:

{
  "job_id": "abc-123",
  "progress_percent": 65.0,
  "status": "quantizing",
  "message": "[ 234/ 567] quantizing blk.5.attn_k.weight ...",
  "output_file": "",
  "extra_metrics": {}
}

When the job completes, output_file contains the path to the quantized GGUF file and extra_metrics includes file_size_mb.

Quantization Types

The UI also supports entering a custom quantization type string for any format supported by llama-quantize.

Web UI

The Quantize page uses the same page pattern as fine-tuning: set up, check, run, result.

1. Set up - Choose the model and the quantization type (a preset, or a custom type). The backend and the Hugging Face token are under More options.
2. Check before you start - Whether a model and type are set, whether a quantization backend is installed, the free RAM and the free space on the models disk. The server reports no model size before the job starts, so the page does not estimate one.
3. Run - Percent, the stage (downloading, converting, quantizing), and a log of what the job reported since the page opened. Stop ends the job.
4. Result - A failed job shows the server’s message. A finished job shows the output file, downloads it, or imports it into LocalAI under a name you choose, then links to a chat with it and to Models.

Earlier jobs are listed below the form, with Reuse and Delete (after a confirmation).

A user needs the quantization permission. Without it, the page says the account cannot quantize.

Quantization is an add-on

The Quantize page is the quantization add-on. It does not ship with LocalAI. An administrator installs it from Find add-ons with Install and turn on. Once it is on, it opens from My add-ons at /app/addons/quantization/jobs and can be shown in the sidebar.

Open the canonical add-on page from My add-ons. Historical standalone links are not registered.

The API does not change. The page calls /api/quantization/* with the signed-in user’s own session, and the server checks the quantization permission on every call.

With --addons-enabled=false this feature is off, like every add-on. The UI shows no entry for it, /app/addons/quantization/jobs leads back to /app, and every path under /api/quantization answers 503. See The addons feature gate.

The add-on also offers tools to the chat: list jobs, propose a job, read the status of a job, stop a job and import the result as a model. A tool can only control a job that a tool started. A job started in the page is listed, but a tool cannot stop or import it. In the chats, an agent and a trigger, a tool call that starts a job runs at once, because you turned the add-on on for that chat; the result says what an undo does (see below). On the OpenAI-compatible API and other surfaces, a tool only proposes the action and a person confirms it in an approval card. A model given as a local path is refused. The tools call the engine through the host operations (see Host operations), which the host setting LOCALAI_ADDON_HOST_OPS grants to this add-on by the quantize. prefix, and they run with the permissions of the person in the chat.

Undo is the safety net for a job a chat started. Starting a job keeps a change set in the add-on, and the chat shows an Undo button on it. An undo stops the job while it still runs. If the job already finished, was stopped, or was imported, the answer is cannot_undo: a finished job cannot be un-run, and the engine offers no operation that removes its files, so nothing is deleted. A change set can be undone for 7 days and only once. Stopping a job and importing the result cannot be undone, and their results say so with the reason. The undo is the person’s own action from a browser tab; no model, agent or API key can call it. The change set lives in the add-on’s own data, with the rest of its journal, so it travels with the add-on’s backups and goes with its data when that is purged. In a cluster the undo goes through the add-on’s owner like every other call.

Architecture

Quantization uses the same gRPC backend architecture as fine-tuning:

1. Proto layer: QuantizationRequest, QuantizationProgress (streaming), StopQuantization
2. Python backend: Downloads model, runs convert_hf_to_gguf.py and llama-quantize
3. Go service: Manages job lifecycle, state persistence, async import
4. REST API: HTTP endpoints with SSE progress streaming
5. React UI: Configuration form, real-time progress monitor, download/import panel