vllm.cpp backend

vllm.cpp is the LocalAI team’s C++20 port of vLLM: the same continuous-batching scheduler, paged KV cache and automatic prefix caching, with no Python at inference time. LocalAI serves it through the native vllm-cpp backend, which loads either a HuggingFace safetensors directory or a .gguf file and applies the chat template, tool-call parsing and reasoning split inside the engine.

This page covers installing the backend and the models LocalAI ships ready to run on it. For the full engine_args reference (KV sizing, scheduling policy, speculative decoding, LMCache), see the vllm.cpp section of the text generation guide.

Installing

local-ai backends install vllm-cpp

Or install it from the Backends page in the web UI. Images are published for CPU, CUDA 13, Vulkan, Metal and Jetson L4T.

Which GPUs the CUDA images cover

The CUDA images are currently built for Blackwell-family architectures only: sm_120a (RTX 50 series, RTX PRO 6000 Blackwell) and sm_121a (GB10 / DGX Spark) on x86-64, and sm_121a alone on arm64. CUDA 13 is required, so there is no CUDA 12 variant.

That is narrower than vllm.cpp itself, which builds ten architectures. On an Ampere, Ada, Hopper, Jetson Orin or Jetson Thor GPU the CUDA backend installs successfully and then fails at the first request with no kernel image is available for execution on the device. Until the build widens, use the Vulkan or CPU image on those cards: vulkan-vllm-cpp builds with CUDA off entirely and gives them a GPU path, without the NVFP4 and Marlin kernels.

Ready-made models

The model gallery carries a curated set of vllm.cpp configurations. Each one arrives with the engine settings already applied, so tool calling, the reasoning split and speculative decoding work without hand-editing YAML.

local-ai models install qwen3-0.6b-vllm-cpp

The two small bf16 entries are the ones that run anywhere the backend does, including CPU. The NVFP4 entries need a Blackwell-class NVIDIA GPU on two counts: NVFP4 has no kernel on older architectures, and the CUDA images are built only for Blackwell in any case.

Sizing note: every entry sets num_blocks to give roughly one to four full contexts of KV cache, which is a starting point rather than a tuned value. KV is not free; the 4B entry, for instance, spends 144 KiB per token, so its 1024 blocks are about 4.5 GB on top of the weights. Raise num_blocks for more concurrency, lower it on a small box.

Why the 27B entries pin a revision

The Qwen3.6-27B entries pin their weights to a specific HuggingFace commit rather than tracking the repository’s default branch. This is deliberate and worth understanding before you copy one of these configs.

The upstream repository was later re-quantized in place, under the same name, from NVFP4 to FP8 W8A8. A config that names the repository without a revision therefore resolves to entirely different weights, with different numerics and different performance, and nothing about the load reports that anything changed. Pinning is what makes the entry reproducible:

artifacts:
  - name: model
    target: model
    source:
      type: huggingface
      repo: unsloth/Qwen3.6-27B-NVFP4
      revision: 890bdef7a42feba6d83b6e17a03315c694112f2a

The same reasoning applies to any quantized community repository you depend on.

Choosing between the speculative variants

Speculative decoding trades memory for decode throughput. All three Qwen3.6-27B entries serve the same weights and produce the same quality; they differ only in how tokens are proposed.

MTP drafts one token per step from a head that already lives in the target checkpoint’s own mtp.* tensors, so it costs no extra download. DFlash drafts a whole 16-token block in one non-autoregressive pass from a separate drafter, which is the larger win at the cost of a second checkpoint on disk.

Start with the plain entry if you are short on memory, and with the DFlash entry if you are not.

Tool calling

Every entry above sets use_tokenizer_template: true and disables LocalAI’s Go-side grammar path, so tool calls are detected and parsed by the engine’s own streaming parsers and arrive as real tool_calls on the OpenAI response.

The parser is normally auto-detected from the chat template, but one case cannot be: Qwen3-Coder’s tool dialect is byte-identical on the wire to another family’s, so template sniffing would pick the wrong parser. The qwen3-coder-30b-a3b-vllm-cpp entry therefore names it explicitly, and any Qwen3-Coder config you write yourself should do the same:

engine_args:
  tool_parser: qwen3_coder

Overriding config.json keys (hf_overrides)

engine_args.hf_overrides is a JSON object of top-level config.json keys that the backend merges over the model directory’s own config.json before the engine loads it, like vLLM’s --hf-overrides. The main use is to opt a published checkpoint into an engine adapter that its config does not name. For example, the Tev1 repositories declare Qwen3_5ForConditionalGeneration, and vllm.cpp serves them as decision models only when the architecture is Tev1Model:

engine_args:
  hf_overrides:
    architectures: ["Tev1Model"]

The downloaded model files do not change. At load the backend creates a private temporary directory. It writes the merged config.json there and adds a symlink for each other entry of the model directory (weights, tokenizer files, a tokenizer/ subdirectory). Then it gives that directory to the engine. When the model unloads, the backend removes the directory.

Rules:

• The merge is top-level only. An override key replaces the whole value of that key, including a nested object such as text_config.
• The model must be a directory that contains a config.json. A .gguf file or a directory without config.json fails the load.
• A value that is not a JSON object (an array, a scalar, or JSON that does not parse) fails the load. The backend does not ignore it, because loading the unchanged config would serve a different architecture than the one you configured.
• An empty object ({}) does nothing.

Named entity recognition (GLiNER2.5)

The vllm-cpp backend serves GLiNER2.5, a zero-shot NER and structured-extraction model. Point the backend at the safetensors directory and the backend exposes the TokenClassify gRPC method, which LocalAI maps to its standard NER API surface.

Labels are supplied at inference time, not baked into the model config. Set them in engine_args:

engine_args:
  ner_labels: "person,organization,location,date,time,money,quantity"
  ner_threshold: 0.5
  ner_max_width: 12

ner_labels is a comma-separated list. When omitted, the backend falls back to a built-in default set (person, organization, location, date, time, money, quantity). ner_threshold is the sigmoid cutoff (default 0.5); ner_max_width is the maximum span length in tokens (default 12).

The model runs the DeBERTa v2 encoder with disentangled attention on the host forward, which is the required contract for pooling models in vllm.cpp. A device-resident forward is tracked as a performance optimization, not a correctness gap.

Decisions API

The vllm-cpp backend serves the kev-compatible SystemOne endpoints (the Decisions API): typed choice, noul and score questions over a state text, answered by a non-generative decision model in one pass. A decision model declares known_usecases: [decisions]. See Decisions API for the request shape, the models you can install and the access rules.

The GLiNER2.5 zero-shot NER model (token_classify) also serves /v1/systemone, through the NER path, and it is the model to use for /v1/systemone/permute and /v1/systemone/separate, which decision models refuse with a 400. It derives its NER labels from the question definitions, so no ner_labels configuration is needed.

Beyond text generation

MiniMax-H3, which generates video and audio jointly, is supported through the stable-diffusion.cpp backend (the -ggml gallery entries), not through vllm-cpp. The vllm-cpp video path has no gallery entries because it does not finish a clip in a practical time. See Video generation.

RKLLM on Rockchip NPUs

RKLLM runs converted language models on supported Rockchip NPUs. Its server already exposes an OpenAI-compatible chat-completions API, so LocalAI can put its authentication, routing, usage tracking, and web UI in front of the board with the cloud-proxy backend.

This integration does not install or convert RKLLM models. Model conversion is a separate host-side step using RKLLM-Toolkit, and the resulting .rkllm file must match the target SoC.

Requirements

• A Linux Rockchip board supported by the installed RKLLM runtime. Upstream RKLLM 1.3.0 accepts rk3588, rk3576, rv1126b, and rk3562 in its server demo.
• A model converted to .rkllm for that target.
• The upstream RKLLM Flask server running on the board and reachable from LocalAI.

RK3566 boards, including Quartz64 models with that SoC, are not in the current upstream RKLLM server target list. LocalAI cannot add support for a SoC that the RKLLM runtime does not support.

Start the RKLLM server

Follow the upstream rkllm_server_demo instructions to deploy the runtime library, server, and converted model to the board. For example, from an RKLLM checkout on the host:

cd examples/rkllm_server_demo
./build_rkllm_server_flask.sh \
  --workshop /userdata/rkllm-server \
  --model_path /userdata/models/qwen3.rkllm \
  --platform rk3588 \
  --adb_device YOUR_DEVICE_SERIAL

The helper starts the server on port 8080. Confirm it is reachable before configuring LocalAI:

curl http://ROCKCHIP_BOARD_IP:8080/v1/models

Configure LocalAI

Create models/rkllm.yaml:

name: rockchip-rkllm
backend: cloud-proxy

proxy:
  mode: passthrough
  provider: openai
  upstream_url: http://ROCKCHIP_BOARD_IP:8080/v1/chat/completions
  upstream_model: rkllm
  request_timeout_seconds: 600

# The upstream is on the local network, so cloud-egress PII filtering is not
# enabled by default in this example. Enable it if your deployment needs it.
pii:
  enabled: false

No API key setting is required by the upstream demo server. If you expose the board beyond a trusted network, put an authenticated reverse proxy in front of it and configure the corresponding key through api_key_env or api_key_file.

Start LocalAI, then use the local model name with any OpenAI-compatible client:

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "rockchip-rkllm",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

LocalAI forwards the request and the streaming response without translating the wire format. Sampling fields supported by the upstream server, including temperature, top_p, top_k, max_tokens, repeat_penalty, and enable_thinking, pass through unchanged.

Limitations

• The upstream demo serves chat completions and model listing only. Embeddings, image generation, audio APIs, and legacy text completions are not available.
• Model conversion and runtime installation remain upstream RKLLM operations.
• Upstream serializes inference for a loaded model and returns HTTP 503 while it is busy. Scale with multiple boards and LocalAI routing if concurrent inference is required.
• The upstream demo returns model-generated tool calls as markup in message.content; passthrough mode does not convert that markup into OpenAI message.tool_calls. Clients that use tools must parse the model-specific format or put an adapter in front of the RKLLM server.

See Cloud passthrough proxy for the full proxy configuration, authentication, routing, and PII options.

Strata backend (experimental)

Strata is an MIT-licensed inference engine for large mixture-of-experts models on consumer hardware. LocalAI exposes it through the experimental strata backend. The backend starts the Strata engine as a child process and talks to it over the engine’s line protocol. Text generation only.

Strata is developed by its own authors, not by the LocalAI project.

Installing

local-ai backends install strata

LocalAI picks the image that fits the machine. The image already contains everything the backend needs: the engine, a portable Python (with numpy) for the tokenizer helper and the pack tool, and the Strata Python modules. You do not build or download an engine yourself.

Supported hardware

These targets have no Strata engine, so there is no image for them: CPU only, Vulkan, NVIDIA Jetson (JetPack 6, CUDA 12 on arm64), other AMD GPU families, and Intel Macs. On those hosts local-ai backends install strata reports that no variant fits the machine.

The images are large because each engine package carries its own GPU runtime libraries (CUDA, ROCm or oneAPI). The ROCm image carries four engines, one per GPU family.

DeepSeek V4.1-Flash runs on a second engine, strata-dsv41, which Strata ships as a separate package. The images carry it next to the Qwen engine where Strata publishes one:

Licensing

The engine and the Strata Python modules are MIT licensed. The engine packages published by Strata also bundle runtime libraries from NVIDIA (CUDA), AMD (ROCm) and Intel (oneAPI), which keep their own licences. The images ship the upstream packages unchanged.

Models

Strata does not run a single GGUF file. A model is a directory that holds the GGUF files and a Strata pack (an index, the dense weights and the tokenizer). The model gallery installs all of them in one step, with the strata backend, the model’s own chat template, and the vision projector for images. Every file is pinned to a Hugging Face commit and checked by SHA-256.

For the GGUF models the gallery downloads only the GGUF files. The backend builds the pack with Strata’s own tools the first time the model loads (the pack_gguf option; see backend/go/strata/README.md). This takes a few minutes and a few GB of disk. A finished pack is never built again. If the build fails, the load fails with the last lines of the tool.

local-ai models install qwen3.8-flash-next-ista-iq2_xs

Qwen3.8-Flash-Next (125B parameters, mixture of experts, Qwen Community License 1.0). The n-gram table (about 29 GB in the ISTA-DASLab files) is read from disk, so use an SSD. RAM is what the experts need plus about 10 GB; the entries map the experts from the files, so a machine with less RAM still runs, slower. Add --resident-experts to engine_args to copy the experts into RAM when there is room.

The ISTA-DASLab, Unsloth and Gyro files are those of the Hugging Face repositories of the same names. The Unsloth and Gyro files need the pack tool’s --compat-bf16 option, which the entries set. NVFP4 packs run on NVIDIA GPUs only. The MTP draft layer is not configured in any entry. The context size is 32768 tokens by default; set context_size for more. For less RAM with the Unsloth files, add --resident-budget-gib N to engine_args, as described in the Strata model guide.

Qwen3.6-35B-A3B (35B parameters, 3B active, Apache License 2.0). Unsloth’s GGUF with the MTP layer, which the engine uses for speculative decoding. It runs on NVIDIA and AMD GPUs; the Intel Arc and Apple builds of Strata do not run it. The engine keeps every expert it can in VRAM and computes the rest on the CPU. On the first load the backend only exports the tokenizer from the GGUF file (pack_tool:tokenizer).

Licences. Qwen3.8-Flash-Next models are under the Qwen Community License 1.0: free use, but it asks for a separate licence from Qwen if you run a Model as a Service or AI Work Assistant business with it, and for the model name to be shown in products above its user and revenue limits. Swift 1.5 adds UkisAI’s Swift Open License 1.0. The NVIDIA checkpoint adds the NVIDIA Open Model License Agreement. Qwen3.6 is Apache 2.0. Read the LICENSE file of the repository before you offer a model to other people. The ISTA-DASLab model cards also carry an Apache 2.0 tag; their text says the weights inherit the base model’s licence, and the entries follow the text.

DeepSeek-V4.1-Flash-1017 (763B parameters, 384 experts per layer, MIT License). It runs on the strata-dsv41 engine, with the model’s own V4.1 chat template, its reasoning trace and its DSML tool calls (the same prompt bytes as DeepSeek’s reference encoder). The quantization is ISTA-DAS’s GSQ-RCO; the draft model for speculative decoding (DSpark) is Unsloth’s. Text only: this backend does not send images to the DeepSeek engine. Each entry needs a GPU with 12 GB of VRAM or more, and Strata measured it with 46 to 64 GB of RAM; with less it runs slowly. The 111 GB n-gram table stays on disk and is read for every token, so use an NVMe drive. The table and the 8 GB draft model are downloaded once into strata/models/deepseek-v4.1-flash-shared and used by every DeepSeek entry.

Strata measured the quality against the original model: the 2.25 set is 7% worse in perplexity, the 2.00 set 16% and the 1.75 set 21%; the Coder set is about 48% worse overall and is meant for code and agent work. Free disk space is the download plus about 1 GB. The default context is 8192 tokens, the size Strata checked; the engine reads up to 1M, and the KV cache takes GPU memory from the experts, so raise context_size with care.

The DeepSeek options of the backend are family:deepseek41 (set by these entries), dspark (auto, the default, on, off or none), experts (5 for faster answers, 6 for full quality; the engine decides when it is not set) and mmap_experts. See backend/go/strata/README.md. These entries were not run on a real engine: the pack build was run on the real file headers, and the prompts were compared with Strata’s reference tokens.

Not in the gallery:

• NVIDIA’s retrained FP4 checkpoint (NVFP4-QAD). Strata converts NVIDIA’s 102 GB of files on the user’s machine with a tool that needs a heavy Python stack.
• OrcaRouter’s uncensored IQ3_XXS. Upstream documents it as a manual conversion.

You can still run these by building the pack with Strata and writing a model config by hand (see backend/go/strata/README.md).

The gallery entries are experimental. The backend has not yet been run against a real engine and model, and the pack build has not been run on a real model file. For DeepSeek, the pack tool was run on the real GGUF headers only.

Limits

No embeddings, no grammar, no video or audio input, one request at a time. DeepSeek models take no images and no logprobs. See backend/go/strata/README.md for the full list, including images, logprobs, tool calls and reasoning, which the backend supports.

Versions

The Strata release and the SHA-256 of every engine package, DeepSeek packages included, are pinned in backend/go/strata/Makefile. A nightly job (bump-strata in .github/workflows/bump_deps.yaml) opens a pull request when Strata publishes a release. The backend speaks an unversioned line protocol, so review each bump against backend/go/strata/README.md.

audio.cpp backend

audio.cpp is a GGML audio inference framework covering roughly thirty model families in one engine. LocalAI exposes it through the native C++ audio-cpp backend, so a single installed backend can serve text to speech, voice cloning, transcription, forced alignment, voice activity detection, speaker diarization, source separation, voice conversion and music generation, depending only on which model you point it at.

audio.cpp is developed by ShugoAI LLC, not by the LocalAI project. Model packages live in huggingface.co/audio-cpp/audio.cpp-gguf.

Installing

local-ai backends install audio-cpp

Or install it from the Backends page in the web UI. audio-cpp is a preference-only backend: LocalAI never picks it automatically during model import, because the only reliable signal for an audio.cpp model is the audiocpp.model_spec.family key stored inside the GGUF, which cannot be read from a remote repository, and the upstream GGUF repository hosts every family in one place. Set backend: audio-cpp in the model YAML, or select it explicitly in the import form.

The bundled audio-cpp gallery entries set backend:best in options to select an available compute backend, with CPU as the fallback. To force CPU execution, replace that option with backend:cpu. Model configurations that omit this option still default to CPU — unless the AUDIOCPP_DEFAULT_BACKEND environment variable is set on the backend process, which supplies the fallback for exactly those models (an explicit backend: option always wins). Set it to best in a deployment to give hand-written model configurations the same accelerator selection the gallery entries get.

Sortformer installation

Install the Sortformer model for speaker diarization from the gallery:

local-ai models install audio-cpp-sortformer-diarization

If installation fails with SHA mismatch, compare the reported metadata checksum with the current gallery entry. An outdated gallery checksum can reject a complete download. Report a persistent mismatch with both checksum values and the model name.

Stable Audio SFX installation

Install the Stable Audio 3 Small SFX model for text-to-sound generation:

local-ai models install audio-cpp-stable-audio-sfx

The gallery pins the Q8_0 file to an immutable upstream revision that matches its SHA256 checksum. Changes to the upstream main branch do not change this download. Use this model with /v1/sound-generation and set duration_seconds to control the sound length.

What it serves

One model serves one family, and a family advertises the tasks it can perform. The backend routes each LocalAI endpoint onto an audio.cpp task and refuses, with the family’s real capability list in the message, when the loaded model cannot serve it.

A supplied speaker clip routes TTS to clon; free-form instructions route it to vdes. Singing voice conversion (svc) is never chosen automatically, because no request signal means “this input is singing”; pin it with task:svc.

Not served, and why

These RPCs return UNIMPLEMENTED with an explanation rather than a degraded answer:

• AudioEncode / AudioDecode: audio.cpp has no codec task kind, so no family can be asked to turn PCM into codec frames or back.
• AudioTransformStream: no family advertises streaming for any task AudioTransform routes to (sep, vc, svc, s2s). Upstream advertises streaming for tts, asr and vad only, so a streaming transform here would be a buffered offline call in disguise.
• AudioToAudioStream: LocalAI’s contract is an OpenAI-Realtime shaped audio conversation with a system prompt and a tool loop. audio.cpp’s s2s is offline clip-to-clip processing against a target voice.
• VoiceEmbed: no family advertises the spk (speaker recognition) task, so nothing in the engine can produce a speaker embedding. Use the voice recognition backends instead.

Model YAML

None of the option names are guessable, so here is the full shape:

name: supertonic
backend: audio-cpp
parameters:
  model: supertonic-3-orig.gguf
known_usecases:
  - FLAG_TTS
options:
  # Optional for an audio.cpp GGUF: the family is read from the file's
  # audiocpp.model_spec.family metadata. REQUIRED for a safetensors file or a
  # package directory, which carry no such key.
  - family:supertonic
  # Optional: pins the audio.cpp task instead of routing it from the RPC.
  - task:tts
  # ggml compute backend inside audio.cpp (not the LocalAI backend above).
  - backend:cuda
  - device:0
  - threads:8
  # Give up on a queued request instead of piling up behind a wedged run.
  - busy_timeout_ms:120000
  # Per-family knobs, passed through to the upstream loader and session.
  - load.vibevoice.lora:/models/my-adapter
  - session.seed_vc.weight_type:q8_0

Every entry is key:value, split on the first colon so values may contain more (paths, for instance). An unknown key fails the load with INVALID_ARGUMENT and a message listing the keys that exist, rather than being silently ignored.

Option reference

Numeric options are validated at load time: a non-integer or negative value is refused before the model is read, not after.

The load. and session. namespaces

Upstream families advertise their own options in two buckets, and the prefix decides which bucket an entry lands in. The key that follows the prefix is passed through verbatim, and upstream names those keys after the family, so both segments are needed:

options:
  # Loader options (upstream "load options"):
  - load.ace_step.dit_model_path:acestep-v15-turbo
  - load.vevo2.whisper_model_path:/models/whisper-dir
  - load.vibevoice.lora:/models/my-adapter
  # Session options (upstream "session options"):
  - session.seed_vc.weight_type:q8_0

The names each family accepts are declared upstream in its loader, and audio.cpp’s own CLI prints them under “Model session options” and “Model load options” when asked for a model’s help. Options that upstream lists as request options are not configurable here: they come from the API request, not from the model YAML.

Voice activity detection with no download

The backend package ships upstream’s silero_vad and marblenet_vad assets, so VAD works with nothing to fetch. bundled:<name> resolves to the asset directory inside the installed backend. Those assets are directories rather than GGUF files, so the family cannot be inferred and family: is required:

name: silero-vad
backend: audio-cpp
parameters:
  model: bundled:silero_vad
known_usecases:
  - FLAG_VAD
options:
  - family:silero_vad

Then call it like any other VAD model, as described in Voice Activity Detection.

Source separation stems

Separation families such as htdemucs and mel_band_roformer produce several named outputs from one run. Running inference once per stem would cost a full separation each time, so the backend runs once, writes every stem to a sibling file next to the main output, and puts the selected one in the response body. params[stem] picks which one:

curl http://localhost:8080/audio/transformations \
  -F model=htdemucs -F audio=@song.wav -F 'params[stem]=drums' -o drums.wav

Without params[stem] the body carries vocals when the model produces it, and the first output otherwise. An unknown stem name is refused with a message listing the stems the model really has, rather than quietly returning a different one. Every stem, including the one in the body, is listed in the X-Audio-Stems response header. See Audio Transform for the full endpoint contract.

params[stem] against a model that produces a single unnamed output is refused rather than ignored.

Text-conditioned transforms

s2s is a text-and-prosody route, not an audio-only one: vevo2 resynthesises a line in another voice and reads the line itself, refusing the run without it. AudioTransform has no text field, so the text arrives as params[text] (or params[target_text], the name vevo2’s own option table uses) and the backend lifts it into the engine’s text input. params[language] rides along when the line is not English.

curl http://localhost:8080/audio/transformations \
  -F model=audio-cpp-vevo2-speech-to-speech \
  -F audio=@source.wav -F reference=@target-speaker.wav \
  -F 'params[text]=The quick brown fox jumps over the lazy dog.' -o converted.wav

A model that takes no text ignores the keys, so separation and plain voice conversion are unaffected.

Pinned tasks

The upstream turn task performs turn detection. LocalAI has no RPC for this task, so requests to a model that only advertises turn are refused. It is not routed through voice activity detection (vad). The upstream wake (wake-word detection) and cls (audio classification) tasks are reported in capability diagnostics. They cannot serve LocalAI audio RPCs, including VAD, even when selected with the task option.

Two routes are unreachable by auto-routing and need task: in the model config, because nothing in a request distinguishes them from a task the same family also advertises:

• task:svc for singing voice conversion. seed_vc and vevo2 advertise both svc and ordinary vc, and no request signal means “this input is singing”, so vc always wins.
• task:s2s for speech to speech, for the same reason against the same vc.

The gallery entries that ship these routes carry the pin already. Removing it gives plain voice conversion from the same weights.

Family notes

• Fish Audio voice cloning: save a reference clip with its transcript in the Voice Library (the Voice add-on, Speech voices), then select Use in Text to Speech. The backend accepts params.ref_text as an alias for params.reference_text in both ordinary and streaming speech requests. If you supply both parameters, reference_text takes precedence. For direct requests with a reference file in voice, supply its transcript in one of these parameters.
• YuE2: the audio-cpp-yue2-3b gallery entry installs a model directory containing the Q8_0 main model, the F16 VAE, and four required sidecar files. Each request requires a non-empty params.style string. Set the params.cot string to off, melody, or full. The model and GGUF package are licensed under CC-BY-NC-4.0 for non-commercial use.
• Supertonic: use the orig GGUF package, whose weights are f32. The f16 package was observed to reach ggml_concat with mismatched operand types and take the backend process down with SIGABRT on the first request, rather than returning an error. Upstream’s docs/gguf.md leaves supertonic’s 16-bit column untested and records its q8_0 as “No (unsupported weight dtype)”, which says the format is unusable rather than that it crashes. The backend refuses both at load time with a message naming the offending tensor, so neither reaches that abort.
• Live transcription latency: AudioTranscriptionLive needs a family advertising asr in streaming mode. At the pinned upstream revision that is nemotron_asr, higgs_audio_stt and voxtral_realtime. higgs_audio_stt and voxtral_realtime decode incrementally and emit partial text as chunks arrive. nemotron_asr only buffers in its chunk handler and decodes everything in finalize(), so with that family every delta arrives after the client closes its send side. Pick accordingly if you need partials during speech.
• Voice conversion vs singing: seed_vc and vevo2 advertise both vc and svc, and automatic routing always picks vc. Pin task:svc for singing voice conversion.
• Cloning-only families: chatterbox advertises clon and vc, and no plain tts at all, so every /v1/audio/speech request against it must carry a reference clip in voice. Without one the request is refused with INVALID_ARGUMENT naming that field, rather than failing deep inside the model.

Acceleration

The ROCm image includes kernels for these GPU targets: gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101, gfx1102, gfx1151, gfx1200 and gfx1201. Set the model option to backend:hip or backend:rocm. There is no Vulkan image because the container would have no Vulkan ICD. BUILD_TYPE=vulkan still works when you build the backend yourself.

NeMo-Speech.cpp backend

NeMo-Speech.cpp is NVIDIA’s Apache-2.0 C++/ggml runtime for the Nemotron Speech models. LocalAI exposes it through the native nemo-speech-cpp backend, which serves four model families from one installed backend: transcription, speaker diarization, speech synthesis and text translation.

NeMo-Speech.cpp is developed by NVIDIA, not by the LocalAI project.

Installing

local-ai backends install nemo-speech-cpp

Or install it from the Backends page in the web UI. nemo-speech-cpp is a preference-only backend: LocalAI never picks it automatically during model import, because the family is decided by the GGUF’s general.architecture key, which cannot be read from a remote repository, and because a translation model carries an ordinary LLM architecture with no NeMo-specific marker at all. Set backend: nemo-speech-cpp in the model YAML, or select it explicitly in the import form.

How the family is chosen

The backend reads general.architecture from the GGUF at load time and picks the family from it. Nothing else in the config selects it.

Two rows need explaining.

The auxiliary architectures are converted NeMo components that attach to a primary model and cannot run on their own. Pointing parameters.model at one fails the load with a message naming the option it belongs on: the codec belongs on codec_model of a magpietts model, the VAD on vad_model of an asr model, the punctuation model on pnc_model of an asr model.

Everything else is translation, and that is deliberate rather than a fallback that happens to catch it. Riva-Translate GGUFs are produced by llama.cpp’s converter and carry an ordinary LLM architecture such as qwen3, so there is no NeMo-specific string to match on. Selecting this backend explicitly is the signal that the model is meant for it.

A request for the wrong family is refused with UNIMPLEMENTED naming the family the model was loaded as, rather than failing somewhere inside the runtime.

Model YAML

Options are key:value entries in the options: list, split on the first colon so a value may contain more. Every path option is resolved relative to the models directory when it is not absolute. An unknown key is ignored rather than rejected, so a config written for a newer backend still loads on an older one.

Transcription

name: nemotron-asr
backend: nemo-speech-cpp
parameters:
  model: asr.gguf
known_usecases:
  - FLAG_TRANSCRIPT
options:
  # All optional. Each attaches a converted NeMo component to the recognizer.
  - vad_model:vad.gguf
  - pnc_model:pnc-bert-base-en.q8_0.gguf
  - itn_dir:sparrowhawk_grammars
  - diar_model:diarization.gguf
  - language_code:en-US
  # GPU device index. Omit it, or set -1, to run on the CPU.
  - gpu:0

Attaching diar_model is what turns on per-word speaker tags: the backend asks for diarization only when the recognizer was created with a diarization model, because the runtime refuses a request for it otherwise. Transcript segments are then cut at each change of speaker. Without it there is a single unlabelled segment. Word-level timings are returned only when the request asks for them with timestamp_granularities[]=word, following the OpenAI contract.

language_code is the model-level default. A per-request language wins over it, and both may be left empty, which the runtime reads as the model’s own default.

Diarization

A sortformer model is standalone: this pipeline has no ASR in it, so segments carry speaker labels and timings but no text.

name: sortformer-diarization
backend: nemo-speech-cpp
parameters:
  model: diarization.gguf
known_usecases:
  - FLAG_DIARIZATION
options:
  - gpu:0

Sortformer is end to end and its speaker capacity is fixed by the checkpoint, so num_speakers, min_speakers, max_speakers and clustering_threshold have nothing to map onto and are logged and dropped. include_text is dropped for the same reason: there is no ASR in this pipeline. min_duration_on and min_duration_off are honoured. To get speaker labels on a transcript, use an asr model with diar_model instead.

Text to speech

name: magpie-tts
backend: nemo-speech-cpp
parameters:
  model: magpie-tts/magpietts.gguf
known_usecases:
  - FLAG_TTS
options:
  # Both are required, and both are auto-discovered when unset (see below).
  - codec_model:magpie-tts/nanocodec.gguf
  - tokenizer_dir:magpie-tts/extracted
  # Optional Sparrowhawk text-normalization grammars applied to the input text.
  - tn_dir:tts_grammars
  - language_code:en-US
  - gpu:0

codec_model and tokenizer_dir are both required, and both are discovered from the model’s own directory when they are not set: a sibling file whose name contains nanocodec or nano-codec becomes the codec, and a sibling directory named extracted becomes the tokenizer directory. If discovery finds nothing, the load fails naming the option to set. That is a hard error rather than a warning because the alternative is a synthesizer that loads and emits noise.

Voices are selected by the request’s voice field. A non-negative integer is used as a speaker index; anything else is passed through as a voice name. MagpieTTS conditions on a speaker, not on a prose style, so instructions has nothing to map onto and is logged and ignored. Per-request params are read for seed, steps, top_k, temperature and cfg_scale; anything else is left at the synthesizer’s configured value.

The TTS runtime has a three-way CPU/CUDA/auto preference rather than a device index, so gpu with a non-negative value means “let the runtime choose” here, and any negative value (-1 is the default) pins it to the CPU.

Translation

name: riva-translate
backend: nemo-speech-cpp
parameters:
  model: translate.q8_0.gguf
known_usecases:
  - FLAG_COMPLETION
  - FLAG_CHAT
options:
  - source_language:en
  - target_language:de
  - gpu:0

Both flags run through the same pair of RPCs, Predict and PredictStream, which is why both are listed for this backend in core/config/backend_capabilities.go. FLAG_COMPLETION is the plain shape: prompt in, translation out. FLAG_CHAT adds three things, and each turn of the conversation is translated on its own: the model becomes eligible as the default chat model when a request names none, it appears in the web UI’s chat model picker, and it stays visible under the gallery’s Chat filter (completion is not a gallery filter, chat is). Neither flag gates a request that names the model explicitly.

Option reference

Translating

Translation is reached through the text endpoints: the prompt is the text to translate, and the languages come from source_language and target_language.

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "riva-translate",
  "messages": [{"role": "user", "content": "The quick brown fox jumps over the lazy dog."}]
}'

A leading [src->tgt] prefix on the prompt overrides the configured pair for that one request:

{"role": "user", "content": "[en->zh-cn] The quick brown fox jumps over the lazy dog."}

Either side may be left out to keep the model-level default for it, so [->de] changes only the target. Three-segment pair tags such as en-zh-cn, en-pt-br and zh-tw-en parse correctly. The prefix is stripped before the text reaches the model.

The C API takes a source and a target language and has no free-form generation entry point, so there is no prompt in the LLM sense. Sampling parameters, tools, grammars and attached media have no equivalent and are ignored; the structural ones (grammar, tools, images, videos, audios, negative_prompt, logprobs) are named in the log when a request sets them. Streaming works, but the runtime returns the finished translation in one piece, so the whole result arrives as a single chunk rather than token by token.

Streaming transcription

Streaming transcription and the realtime live path both emit a delta per finalized utterance. Interim hypotheses are produced by the runtime and deliberately dropped: the wire contract defines delta as newly finalized text that consumers concatenate, and the runtime rewrites a final rather than extending its interims (it runs ITN and formatting stripping on the final only). Forwarding interims would assemble “he”, “hell”, “hello”, “Hello.” into hehellhelloHello. rather than into the transcript, and no diffing trick recovers it.

The cost is latency: the first delta of an utterance arrives at its endpoint rather than mid-word. That is the trade this backend takes.

Acceleration

There is no AMD ROCm and no Intel SYCL support, because upstream NeMo-Speech.cpp has no HIP and no SYCL backend, so there is nothing to build against. A host reporting either capability gets the CPU build, which is the honest answer rather than a broken image. If you want NeMo ASR on an AMD or Intel GPU, use parakeet-cpp instead, which is described on the Audio to text page.

Limitations

• Text normalization is Linux only. The macOS build ships without it: the Sparrowhawk/OpenFST stack assumes a GNU toolchain, and the gcc-12 pin it needs (OpenFST’s templates fail to compile on gcc-13 and gcc-14 at -O2) has no macOS equivalent. One build flag covers both directions, so on macOS both itn_dir (inverse text normalization on ASR output) and tn_dir (text normalization on TTS input) do nothing. A tn_dir set on a macOS build logs a warning and carries on with normalization disabled. pnc_model is unaffected: punctuation is always compiled in.
• Interim streaming results are suppressed, as described above. This costs latency.
• Translation runs at the library’s default limits: 1024 tokens of context and 256 new tokens per call. Neither is configurable from the model YAML, because both are create-time settings on the translator and raising the context costs one context-sized KV cache per pooled context. The two limits fail differently. Input longer than the context is rejected, with nmt: prompt too long (N tokens) for context 1024 surfacing as a failed request, so you will know. Output longer than 256 tokens is silently cut: generation simply stops at the limit and the truncated translation is returned as if it were complete. Translate a sentence or a paragraph at a time rather than a whole document.

Converting models

Upstream has a single conversion entry point for every family. SOURCE may be a .nemo archive, an extracted NeMo checkpoint, a local Hugging Face directory, or a Hugging Face repository ID.

git clone https://github.com/NVIDIA/NeMo-Speech.cpp
cd NeMo-Speech.cpp
pip install -r requirements.txt

python3 convert_model.py nvidia/nemotron-speech-streaming-en-0.6b \
    --outfile /models/asr.gguf

Copy the resulting GGUF into your models directory and write the YAML above against it.

Text to speech is the one family that needs more than a single conversion. It loads two GGUFs, the MagpieTTS token generator and the NanoCodec decoder, and it needs the tokenizer assets that live inside the MagpieTTS .nemo archive rather than in the GGUF. Extract the archive and keep the extracted directory next to the converted model, so that tokenizer_dir (or the extracted/ auto-discovery) has something to point at, and put the codec GGUF in the same directory so codec_model (or its auto-discovery) finds it.

# 1. MagpieTTS: download the .nemo, extract it for the tokenizer, convert it
hf download nvidia/magpie_tts_multilingual_357m --revision v2602 \
    --local-dir /models/magpie-tts
mkdir -p /models/magpie-tts/extracted
tar -xf /models/magpie-tts/magpie_tts_multilingual_357m.nemo \
    -C /models/magpie-tts/extracted
python3 convert_model.py /models/magpie-tts/extracted \
    --outfile /models/magpie-tts/magpietts.gguf

# 2. NanoCodec: no tokenizer, it is a codec decoder. The filename carries
#    "nanocodec" so the auto-discovery in the YAML above picks it up.
hf download nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps \
    --local-dir /models/magpie-tts/nano-codec
python3 convert_model.py \
    /models/magpie-tts/nano-codec/nemo-nano-codec-22khz-1.89kbps-21.5fps.nemo \
    --outfile /models/magpie-tts/nanocodec.gguf

Skipping the codec leaves the model unloadable: the backend fails with no NanoCodec GGUF found next to ... naming the codec_model option.

Translation additionally uses the pinned llama.cpp converter, which has to be initialized first:

git submodule update --init llama.cpp
pip install -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt
python3 convert_model.py nvidia/Riva-Translate-4B-Instruct-v2 \
    --outfile /models/translate.q8_0.gguf --outtype q8_0

See upstream’s model conversion guide for the per-architecture defaults and options.

Whisper-Medusa

The whisper-medusa backend serves aiola’s Whisper-Medusa checkpoints through LocalAI’s OpenAI-compatible audio transcription endpoint. Whisper-Medusa uses multiple decoding heads to predict several tokens per decoding step.

Model configuration

Install the whisper-medusa backend from the backend gallery, then create a model configuration such as:

name: whisper-medusa
backend: whisper-medusa
parameters:
  model: aiola/whisper-medusa-linear-libri
options:
  - language:en
  - regulation_start:140
  - regulation_factor:1.01

Transcribe a clip with the standard endpoint:

curl http://localhost:8080/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=whisper-medusa \
  -F language=en

language in the request overrides the configured default. The regulation options control the exponential length penalty passed to upstream generation.

The backend currently ships Linux CPU and NVIDIA CUDA 12 images. It is not advertised for macOS, ROCm, Intel GPU, or Jetson because upstream pins PyTorch 2.2.2 and has not published compatibility for those platforms.