Audio to Text

Audio to text models are models that can generate text from an audio file.

The transcription endpoint allows to convert audio files to text. The endpoint supports multiple backends:

• whisper.cpp: A C++ library for audio transcription (default)
• moonshine: Ultra-fast transcription engine optimized for low-end devices
• faster-whisper: Fast Whisper implementation with CTranslate2
• WhisperX: Whisper transcription with word alignment and optional speaker diarization. Set HF_TOKEN and pass diarize=true to load WhisperX’s gated pyannote diarization pipeline.
• parakeet-cpp: A C++/ggml port of NVIDIA NeMo Parakeet (FastConformer TDT/CTC/RNNT/hybrid). Runs quantized GGUFs on CPU or GPU, emits word-level timestamps, and supports cache-aware streaming (the realtime_eou model surfaces end-of-utterance events). The same backend also loads Nemotron-3-Diarization (/v1/audio/diarization) and CED sound models (/v1/audio/classification), and can attach either as a companion to a transcription model.
• llama-cpp: Route transcription to any multimodal-audio GGUF model served by the llama-cpp backend (e.g. Qwen3-ASR, Voxtral, Qwen2-Audio). Under the hood the request is converted into a chat completion with the audio attached via the model’s audio encoder - the same path the upstream llama.cpp server uses. Set backend: llama-cpp in the model YAML and point mmproj at the matching audio encoder.
• voxtral: Voxtral-family models served by a dedicated backend
• NeMo-Speech.cpp: NVIDIA’s C++/ggml runtime for the Nemotron Speech models. Serves offline, streaming and live transcription, with VAD, punctuation, inverse text normalization and Sortformer speaker tags attached through model options, and covers diarization, speech synthesis and translation from the same backend. See the NeMo-Speech.cpp backend page for the model options.
• audio.cpp: Multi-family GGML audio engine. Serves transcription and forced alignment from families such as nemotron_asr, qwen3_asr, citrinet_asr, higgs_audio_stt and voxtral_realtime, and covers diarization, VAD, TTS and source separation from the same backend. See the audio.cpp backend page for the model options.

The endpoint input supports all the audio formats supported by ffmpeg.

Looking for “who spoke when” instead of a flat transcript? See Speaker Diarization - /v1/audio/diarization returns time-stamped speaker segments and supports the rttm format used by pyannote.metrics.

Usage

Once LocalAI is started and whisper models are installed, you can use the /v1/audio/transcriptions API endpoint.

For instance, with cURL:

curl http://localhost:8080/v1/audio/transcriptions -H "Content-Type: multipart/form-data" -F file="@<FILE_PATH>" -F model="<MODEL_NAME>"

When usage statistics are enabled, successful requests to /v1/audio/transcriptions and /audio/transcriptions contribute request counts and elapsed time for the requested model name, including an alias. This includes streaming responses. Token counts remain zero because transcription results do not provide canonical token usage. Failed transcriptions, including errors reported within a stream, do not count as successful usage.

When API tracing is enabled, multipart transcription requests appear in Traces with the model name, response, status, and elapsed time. The captured request body contains metadata marked multipart upload omitted; API traces do not store the uploaded audio. Response capture follows the configured trace size limit. Backend traces have their own payload capture behavior.

Example

Download one of the models from here in the models folder, and create a YAML file for your model:

name: whisper-1
backend: whisper
parameters:
  model: whisper-en

The transcriptions endpoint then can be tested like so:

## Get an example audio file
wget --quiet --show-progress -O gb1.ogg https://upload.wikimedia.org/wikipedia/commons/1/1f/George_W_Bush_Columbia_FINAL.ogg

## Send the example audio file to the transcriptions endpoint
curl http://localhost:8080/v1/audio/transcriptions -H "Content-Type: multipart/form-data" -F file="@$PWD/gb1.ogg" -F model="whisper-1"

Result:

{
  "segments":[{"id":0,"start":0,"end":9640000000,"text":" My fellow Americans, this day has brought terrible news and great sadness to our country.","tokens":[50364,1222,7177,6280,11,341,786,575,3038,6237,2583,293,869,22462,281,527,1941,13,50846]},{"id":1,"start":9640000000,"end":15960000000,"text":" At 9 o'clock this morning, Mission Control and Houston lost contact with our Space Shuttle","tokens":[1711,1722,277,6,9023,341,2446,11,20170,12912,293,18717,2731,3385,365,527,8705,13870,10972,51162]},{"id":2,"start":15960000000,"end":16960000000,"text":" Columbia.","tokens":[17339,13,51212]},{"id":3,"start":16960000000,"end":24640000000,"text":" A short time later, debris was seen falling from the skies above Texas.","tokens":[316,2099,565,1780,11,21942,390,1612,7440,490,264,25861,3673,7885,13,51596]},{"id":4,"start":24640000000,"end":27200000000,"text":" The Columbia's lost.","tokens":[440,17339,311,2731,13,51724]},{"id":5,"start":27200000000,"end":29920000000,"text":" There are no survivors.","tokens":[821,366,572,18369,13,51860]},{"id":6,"start":29920000000,"end":32920000000,"text":" And board was a crew of seven.","tokens":[50364,400,3150,390,257,7260,295,3407,13,50514]},{"id":7,"start":32920000000,"end":39780000000,"text":" Colonel Rick Husband, Lieutenant Colonel Michael Anderson, Commander Laurel Clark, Captain","tokens":[28478,11224,21282,4235,11,28412,28478,5116,18768,11,20857,27270,75,18572,11,10873,50857]},{"id":8,"start":39780000000,"end":50020000000,"text":" David Brown, Commander William McCool, Dr. Cooltna Chavla, and Elon Ramon, a Colonel","tokens":[4389,8030,11,20857,6740,4050,34,1092,11,2491,13,8561,83,629,761,706,875,11,293,28498,9078,266,11,257,28478,51369]},{"id":9,"start":50020000000,"end":52800000000,"text":" in the Israeli Air Force.","tokens":[294,264,19974,5774,10580,13,51508]},{"id":10,"start":52800000000,"end":58480000000,"text":" These men and women assumed great risk in the service to all humanity.","tokens":[1981,1706,293,2266,15895,869,3148,294,264,2643,281,439,10243,13,51792]},{"id":11,"start":58480000000,"end":63120000000,"text":" And an age when Space Flight has come to seem almost routine.","tokens":[50364,400,364,3205,562,8705,28954,575,808,281,1643,1920,9927,13,50596]},{"id":12,"start":63120000000,"end":68800000000,"text":" It is easy to overlook the dangers of travel by rocket and the difficulties of navigating","tokens":[467,307,1858,281,37826,264,27701,295,3147,538,13012,293,264,14399,295,32054,50880]},{"id":13,"start":68800000000,"end":72640000000,"text":" the fierce outer atmosphere of the Earth.","tokens":[264,25341,10847,8018,295,264,4755,13,51072]},{"id":14,"start":72640000000,"end":78040000000,"text":" These astronauts knew the dangers and they faced them willingly.","tokens":[1981,28273,2586,264,27701,293,436,11446,552,44675,13,51342]},{"id":15,"start":78040000000,"end":83040000000,"text":" Knowing they had a high and noble purpose in life.","tokens":[25499,436,632,257,1090,293,20171,4334,294,993,13,51592]},{"id":16,"start":83040000000,"end":90800000000,"text":" Because of their courage and daring and idealism, we will miss them all the more.","tokens":[50364,1436,295,641,9892,293,43128,293,7157,1434,11,321,486,1713,552,439,264,544,13,50752]},{"id":17,"start":90800000000,"end":96560000000,"text":" All Americans today are thinking as well of the families of these men and women who have","tokens":[1057,6280,965,366,1953,382,731,295,264,4466,295,613,1706,293,2266,567,362,51040]},{"id":18,"start":96560000000,"end":100440000000,"text":" been given this sudden shock in grief.","tokens":[668,2212,341,3990,5588,294,18998,13,51234]},{"id":19,"start":100440000000,"end":102400000000,"text":" You're not alone.","tokens":[509,434,406,3312,13,51332]},{"id":20,"start":102400000000,"end":105440000000,"text":" Our entire nation agrees with you.","tokens":[2621,2302,4790,26383,365,291,13,51484]},{"id":21,"start":105440000000,"end":112360000000,"text":" And those you loved will always have the respect and gratitude of this country.","tokens":[400,729,291,4333,486,1009,362,264,3104,293,16935,295,341,1941,13,51830]},{"id":22,"start":112360000000,"end":116600000000,"text":" The cause in which they died will continue.","tokens":[50364,440,3082,294,597,436,4539,486,2354,13,50576]},{"id":23,"start":116600000000,"end":124240000000,"text":" Man kind is led into the darkness beyond our world by the inspiration of discovery and the","tokens":[2458,733,307,4684,666,264,11262,4399,527,1002,538,264,10249,295,12114,293,264,50958]},{"id":24,"start":124240000000,"end":127000000000,"text":" longing to understand.","tokens":[35050,281,1223,13,51096]},{"id":25,"start":127000000000,"end":131160000000,"text":" Our journey into space will go on.","tokens":[2621,4671,666,1901,486,352,322,13,51304]},{"id":26,"start":131160000000,"end":136480000000,"text":" In the skies today, we saw destruction and tragedy.","tokens":[682,264,25861,965,11,321,1866,13563,293,18563,13,51570]},{"id":27,"start":136480000000,"end":142080000000,"text":" As farther than we can see, there is comfort and hope.","tokens":[1018,20344,813,321,393,536,11,456,307,3400,293,1454,13,51850]},{"id":28,"start":142080000000,"end":149800000000,"text":" In the words of the prophet Isaiah, lift your eyes and look to the heavens who created","tokens":[50364,682,264,2283,295,264,18566,27263,11,5533,428,2575,293,574,281,264,26011,567,2942,50750]},{"id":29,"start":149800000000,"end":151640000000,"text":" all these.","tokens":[439,613,13,50842]},{"id":30,"start":151640000000,"end":159960000000,"text":" He who brings out the story hosts one by one and calls them each by name because of his great","tokens":[634,567,5607,484,264,1657,21573,472,538,472,293,5498,552,1184,538,1315,570,295,702,869,51258]},{"id":31,"start":159960000000,"end":163400000000,"text":" power and mighty strength.","tokens":[1347,293,21556,3800,13,51430]},{"id":32,"start":163400000000,"end":166400000000,"text":" Not one of them is missing.","tokens":[1726,472,295,552,307,5361,13,51580]},{"id":33,"start":166400000000,"end":173600000000,"text":" The same creator who names the stars also knows the names of the seven souls we mourn","tokens":[50364,440,912,14181,567,5288,264,6105,611,3255,264,5288,295,264,3407,16588,321,22235,77,50724]},{"id":34,"start":173600000000,"end":175600000000,"text":" today.","tokens":[965,13,50824]},{"id":35,"start":175600000000,"end":183160000000,"text":" The crew of the shuttle Columbia did not return safely to earth yet we can pray that all","tokens":[440,7260,295,264,26728,17339,630,406,2736,11750,281,4120,1939,321,393,3690,300,439,51202]},{"id":36,"start":183160000000,"end":185840000000,"text":" are safely home.","tokens":[366,11750,1280,13,51336]},{"id":37,"start":185840000000,"end":192600000000,"text":" May God bless the grieving families and may God continue to bless America.","tokens":[1891,1265,5227,264,48454,4466,293,815,1265,2354,281,5227,3374,13,51674]},{"id":38,"start":196400000000,"end":206400000000,"text":" [BLANK_AUDIO]","tokens":[50364,542,37592,62,29937,60,50864]}],
  "text":"My fellow Americans, this day has brought terrible news and great sadness to our country. At 9 o'clock this morning, Mission Control and Houston lost contact with our Space Shuttle Columbia. A short time later, debris was seen falling from the skies above Texas. The Columbia's lost. There are no survivors. And board was a crew of seven. Colonel Rick Husband, Lieutenant Colonel Michael Anderson, Commander Laurel Clark, Captain David Brown, Commander William McCool, Dr. Cooltna Chavla, and Elon Ramon, a Colonel in the Israeli Air Force. These men and women assumed great risk in the service to all humanity. And an age when Space Flight has come to seem almost routine. It is easy to overlook the dangers of travel by rocket and the difficulties of navigating the fierce outer atmosphere of the Earth. These astronauts knew the dangers and they faced them willingly. Knowing they had a high and noble purpose in life. Because of their courage and daring and idealism, we will miss them all the more. All Americans today are thinking as well of the families of these men and women who have been given this sudden shock in grief. You're not alone. Our entire nation agrees with you. And those you loved will always have the respect and gratitude of this country. The cause in which they died will continue. Man kind is led into the darkness beyond our world by the inspiration of discovery and the longing to understand. Our journey into space will go on. In the skies today, we saw destruction and tragedy. As farther than we can see, there is comfort and hope. In the words of the prophet Isaiah, lift your eyes and look to the heavens who created all these. He who brings out the story hosts one by one and calls them each by name because of his great power and mighty strength. Not one of them is missing. The same creator who names the stars also knows the names of the seven souls we mourn today. The crew of the shuttle Columbia did not return safely to earth yet we can pray that all are safely home. May God bless the grieving families and may God continue to bless America. [BLANK_AUDIO]"
}

You can also specify the response_format parameter to be one of lrc, srt, vtt, text, json or verbose_json (default):

## Send the example audio file to the transcriptions endpoint
curl http://localhost:8080/v1/audio/transcriptions -H "Content-Type: multipart/form-data" -F file="@$PWD/gb1.ogg" -F model="whisper-1" -F response_format="srt"

Result (first few lines):

1
00:00:00,000 --> 00:00:09,640
My fellow Americans, this day has brought terrible news and great sadness to our country.

2
00:00:09,640 --> 00:00:15,960
At 9 o'clock this morning, Mission Control and Houston lost contact with our Space Shuttle

3
00:00:15,960 --> 00:00:16,960
Columbia.

4
00:00:16,960 --> 00:00:24,640
A short time later, debris was seen falling from the skies above Texas.

5
00:00:24,640 --> 00:00:27,200
The Columbia's lost.

6
00:00:27,200 --> 00:00:29,920
There are no survivors.

Supported request parameters

In addition to file and model, the endpoint accepts the following multipart form fields, matching the OpenAI audio transcription API:

If speaker diarization fails after transcription succeeded, the WhisperX backend logs the error and returns the transcript without speaker labels. Other transcription failures return an error instead of an empty transcript. Diarization still requires HF_TOKEN.

The response body for verbose_json includes text, language, duration, and segments[] (with speaker populated when diarization is enabled).

Streaming transcriptions

Set -F stream=true to receive token-by-token SSE events as the backend produces them. The event shape matches the OpenAI streaming transcription format:

curl -N http://localhost:8080/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@sample.wav" \
  -F model="whisper-1" \
  -F stream=true

data: {"type":"transcript.text.delta","delta":"And so, my"}

data: {"type":"transcript.text.delta","delta":" fellow Americans..."}

data: {"type":"transcript.text.done","text":"And so, my fellow Americans..."}

data: [DONE]

Backends that do not natively stream tokens fall back to emitting one delta plus a done event with the full text - the SSE contract is identical either way.

Using the llama-cpp backend with an audio-capable model

Any GGUF model whose mmproj contains an audio encoder can be used for transcription via the llama-cpp backend. This reuses the model’s own audio front-end rather than shelling out to whisper.cpp, which is useful when you want a single backend serving both chat-with-audio and transcription.

Example using ggml-org/Qwen3-ASR-0.6B-GGUF:

name: qwen3-asr
backend: llama-cpp
parameters:
  model: Qwen3-ASR-0.6B-Q8_0.gguf
mmproj: mmproj-Qwen3-ASR-0.6B-Q8_0.gguf

Then call /v1/audio/transcriptions as usual:

curl http://localhost:8080/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@jfk.wav" \
  -F model="qwen3-asr"

LFM3-Audio (experimental)

The gallery entries lfm3-audio-2b and lfm3-audio-2b-q8 also transcribe through the llama-cpp backend, using the same model as speech-to-speech and TTS. When a language is given, the request asks for transcription in that language with the system prompt Perform ASR in <Language>.. Without one, the model is asked to infer the language. Transcription uses the second slot (parallel:2), so it can run next to a realtime session. A realtime session does not use it for the input transcript by default; see Realtime. This needs a llama.cpp build with LFM3-Audio support.

Using the parakeet-cpp backend

parakeet.cpp is a C++/ggml port of NVIDIA NeMo Parakeet that matches the upstream PyTorch models on CPU. GGUF weights for every model and quant are published in a single repo, mudler/parakeet-cpp-gguf. F16 is the recommended default, and Q4_K stays near-lossless on the small models. The easiest path is to import directly (the GGUFs auto-detect to this backend):

local-ai models import https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/tdt_ctc-110m-f16.gguf

Or write a model YAML:

name: parakeet-110m
backend: parakeet-cpp
parameters:
  model: tdt_ctc-110m-f16.gguf

Then call /v1/audio/transcriptions as usual. Pass timestamp_granularities[]=word for per-word timings:

curl http://localhost:8080/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@jfk.wav" \
  -F model="parakeet-110m" \
  -F "timestamp_granularities[]=word"

For real-time use, load a cache-aware streaming model (e.g. realtime_eou_120m-v1-*.gguf) and pass -F stream=true. Deltas are emitted as the audio is decoded, with end-of-utterance events closing each segment.

Diarization and sound classification

The same backend also serves the /v1/audio/diarization and /v1/audio/classification endpoints, and can attach a diarization or sound model to a live transcription session. options: accepts paths relative to the models directory, or absolute:

With a diarization_model companion, /v1/audio/transcriptions labels each segment with its speaker ("0", "1", … in order of first appearance) and splits segments where the speaker changes; with timestamp_granularities[]=word each word carries its speaker too. With stream=true the closing transcript.text.done event lists the segments with their speakers. Pass -F diarize=false to skip diarization for one request. The diarization GGUF can also be imported directly: local-ai models import https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/nemotron-3-diarization-f16.gguf.

speaker_model: needs libparakeet with C-API v10. A wrong setup fails at load time with one of these errors: parakeet-cpp: speaker_model needs libparakeet.so ABI 10 (parakeet_capi_speaker_registry_add_embedding); the loaded library is older, parakeet-cpp: speaker_model needs a diarization model (the primary or diarization_model:), parakeet-cpp: a speaker model cannot be the primary model; use it as speaker_model: next to a diarization model, parakeet-cpp: speaker_model "<path>" is a <kind> model, expected a speaker model (the file is not a speaker encoder GGUF), or parakeet-cpp: speaker_threshold "<value>" must be a distance in (0, 2) (1 minus cosine similarity) / parakeet-cpp: speaker_margin "<value>" must be a number in [0, 1) for a bad number.

The loader rejects a companion whose role duplicates the primary’s own (for example asr_model: on an already-ASR primary, or sound_model: on a CED primary), and rejects a companion GGUF that does not match the role its option names (for example sound_model: pointing at an ASR GGUF fails to load, naming the kind it expected). See Speaker Diarization for the Diarize RPC and Sound Classification for SoundDetection, and Realtime API for the live speaker/sound events emitted during a realtime session.

Segment timestamps

Transcriptions are split into segments the same way NVIDIA NeMo does: a new segment starts after sentence-ending punctuation (., ?, !), and each segment carries start/end times. This is the default (NeMo’s punctuation-only segmentation) and needs no configuration. While streaming, each end-of-utterance closes a segment, now with timestamps.

You can additionally split on silence by setting segment_gap_threshold (NeMo’s segment_gap_threshold, in encoder frames; off by default). When set, a gap between two words wider than the threshold also starts a new segment. The value is in frames to match NeMo exactly; the backend converts it to seconds using the model’s frame stride (frame_sec, reported by the engine):

name: parakeet-110m
backend: parakeet-cpp
parameters:
  model: tdt_ctc-110m-f16.gguf
options:
- segment_gap_threshold:12   # split on silence > 12 encoder frames (default 0 = off, punctuation-only)

Dynamic batching

The backend can coalesce concurrent transcription requests into a single batched engine call, which improves throughput on GPU when many requests arrive at once. Batching is off by default (batch_max_size:1, one request at a time); raise it to opt in. Two options: knobs control it:

name: parakeet-110m
backend: parakeet-cpp
parameters:
  model: tdt_ctc-110m-f16.gguf
options:
- batch_max_size:8      # max requests coalesced into one batch (default 1 = off)
- batch_max_wait_ms:15  # how long to wait to fill a batch, in ms (default 15)

By default each request runs on its own. Raise batch_max_size (for example 4 to 16) to enable batching; it pays off on GPU under concurrent load, where coalescing the per-step decode GEMMs across requests is a large throughput win. Leave it at 1 on CPU and for low-concurrency setups, where batching only adds latency. Batching only affects concurrent unary requests; streaming sessions always run on their own.

Moondream Ultra and Redux

Moondream publishes two derivatives of NVIDIA parakeet-tdt-0.6b-v3, Ultra and Redux. Both have a voice-activity-detection (VAD) head. The gallery has five entries, built from the GGUFs in mudler/parakeet-cpp-gguf:

The packed Redux file stores the encoder as ternary weights (213 MB). It cannot load on a GPU backend and cannot stream. If a GPU build fails to load it, check the backend log for the library message and use the redux-f16 or redux-q8_0 entry instead. The weights are CC-BY-4.0: credit Moondream and NVIDIA.

VAD-only slices

If you only need the VAD head, for the VAD endpoint or to cut audio before transcription, the same repository has two small files with the head cut out of the full model. The weights are not retrained, and the files cannot transcribe:

Measured by the parakeet.cpp author against loading a whole Redux or Ultra model: the files are 6 to 10 MB instead of 213 MB to 1.4 GB, load in a few milliseconds instead of 0.1 to 0.7 s, and use about 245 MiB peak memory for a 33 s clip instead of 0.6 to 1.6 GiB. The output is byte-identical to the full parent model, and the speed is the same as the parent’s head. A transcription request on a slice fails with an error. The slices load only with a parakeet.cpp build that includes VAD-only GGUF support, so an older backend build fails to load them. The weights are CC-BY-4.0: credit Moondream and NVIDIA.

With vad:true, long audio is cut at pauses found by the model’s VAD head into pieces of at most 30 seconds, and each piece is transcribed in turn. Word timestamps stay relative to the whole file. Audio of 30 seconds or less gives the same result as without the option. The gallery entries set it. Add it to your own model YAML like this:

name: moondream-ultra
backend: parakeet-cpp
parameters:
  model: ultra-q8_0.gguf
options:
- vad:true   # cut long audio at pauses (default false); needs a model with a VAD head

vad:true applies to offline transcription only and bypasses dynamic batching, because the batched entry point has no VAD variant. Streaming is not affected. A model without a VAD head fails each request with model has no VAD head, and a libparakeet.so that is too old to export the VAD entry point fails the load. Remove the option for models that have no VAD head.

Cutting long audio with Silero (vad_model)

A model without a VAD head, such as parakeet-cpp-tdt-0.6b-v3 or a Nemotron model, can cut long audio with Silero VAD instead. Name a Silero GGUF in the vad_model option. The path is resolved against the models directory, like the other companion files. vad_model implies vad:

name: parakeet-v3-silero
backend: parakeet-cpp
parameters:
  model: parakeet-cpp/tdt-0.6b-v3-f16.gguf
options:
- vad_model:parakeet-cpp/silero-vad-f16.gguf   # Silero GGUF that cuts long audio at pauses
- vad_min_pause:0.3                            # optional, seconds

The gallery entry parakeet-cpp-tdt-0.6b-v3-silero-vad installs both files with this configuration. Audio of 30 seconds or less is transcribed whole and the VAD does not run. vad:true alone keeps meaning “use the model’s own head”. With vad_model set, the Silero model is used even if the ASR model has a head.

The segmenter options below apply to both vad:true and vad_model. Each is optional; an unset value keeps the default of the detector in use, and a bad value fails the load:

vad_speech_pad (seconds) pads each region and only affects the VAD endpoint. vad_model needs a libparakeet.so that exports parakeet_capi_transcribe_path_json_vad_with; an older library fails the load with a message that names it.

Dropping noise words (guard_*)

A decode of noise or silence can contain words that no one said. An opt-in filter removes the words that stand alone or sit among low-confidence words, and the words that are only punctuation. It runs on the finished decode. It is off unless one of these options is set, and a bad value fails the load:

options:
- guard_min_local_conf:0.5
- guard_drop_punct_only:true

The filter applies to offline transcription, and with vad:true or vad_model to each piece on its own. It bypasses dynamic batching, like vad:true. Streaming is not affected. Speech with confident words comes out the same as without the filter. The number of dropped words is written to the backend log at debug level; the transcription response has no field for it. The options need a libparakeet.so that exports parakeet_capi_transcribe_path_json_with; an older library fails the load with a message that names it.

Bundle GGUF files (several models in one file)

A bundle is one GGUF file that holds several models, called components. Each component keeps its own licence. The backend opens the components it needs from the one file, so a single model YAML can serve transcription, VAD, diarization, speaker naming and sound events. A bundle needs a libparakeet.so from parakeet.cpp with bundle support (pin 781a973 or newer); the format is described in the parakeet.cpp bundle documentation. Single-model files and every existing option work as before.

The gallery has three bundles, built from mudler/parakeet-cpp-gguf:

The component that each role uses, and the option that picks another one:

speaker_component: also names registered speakers: LocalAI sends the voices from /v1/voice/register to the bundle’s speaker component, as it does for speaker_model:. This works in /v1/audio/diarization and in realtime live transcription, and it needs no speaker_model:. speaker_threshold, speaker_margin and speaker_strict apply too. Only voices that carry an encoder fingerprint (voices enrolled from speaker_profiles) and voices with no tag at all are used by default. A voice registered with a tag only matches through speaker_tag:. See Voice Recognition.

A *_component option without the matching companion option takes the component from the model file itself. The companion options (diarization_model:, sound_model:, speaker_model:, vad_model:, asr_model:) can also name a bundle file, even the same file as the model: the only component of the wanted kind is used, and the *_component option picks one when there are several. This YAML loads the same file for four roles:

name: parakeet-bundle
backend: parakeet-cpp
parameters:
  model: parakeet-cpp/parakeet-bundle-small.gguf
options:
- vad:true                  # cut long audio at pauses, with the Silero component
- diar_component:diar       # same as diarization_model:parakeet-cpp/parakeet-bundle-small.gguf
- sound_component:ced
- speaker_component:voice

A role that the bundle cannot fill fails with a message that lists the components, for example parakeet-cpp: diarization_model needs a "diar" component, but the bundle "<path>" has none (components: asr (asr), vad (vad)). A request for a role the loaded model does not have (diarization from a bundle without a diar component) returns parakeet-cpp: model is not a diarization model (the model file is a bundle without a "diar" component; ...). A *_component option on a file that is not a bundle, or on a library without bundle support, fails the load.

Licences: a bundle has no single licence, so the gallery entries use license: other. The licence, source and credit of every component are in the file header, and the NOTICE file next to each bundle in the repository (NOTICE-<bundle>.txt) has the credits and the full licence texts. Keep it with any copy of the file you pass on.

The licence of the CED weights is not consistent upstream: the model card says Apache-2.0, the upstream code repository is GPL-3.0 and the original checkpoint records say CC-BY-4.0. The file here is converted, not trained, and follows the model card. The weights of all components were converted to GGUF, and quantised where the table shows it; nothing was retrained.

See also

• Audio Transform - clean up the audio (echo cancellation, noise suppression, dereverberation) before passing it to a transcription model.

Text to Audio (TTS)

API Compatibility

The LocalAI TTS API is compatible with the OpenAI TTS API and the Elevenlabs API.

LocalAI API

The /tts endpoint can also be used to generate speech from text.

Usage

Input: input, model

For example, to generate an audio file, you can send a POST request to the /tts endpoint with the instruction as the request body:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
  "input": "Hello world",
  "model": "tts"
}'

Returns an audio/wav file.

List available voices

Use GET /v1/audio/voices to list named voices for installed TTS models:

curl http://localhost:8080/v1/audio/voices

Add the model query parameter to return one installed model:

curl 'http://localhost:8080/v1/audio/voices?model=pocket-tts'

Each voice can include language and gender metadata. LocalAI supplies the built-in Pocket TTS catalog. Other models can declare their catalog in YAML:

name: custom-tts
backend: custom
known_usecases: [tts]
tts:
  voices:
    - name: narrator
      language: en_GB
      gender: female

LocalAI returns 404 when the requested model is not installed. Models without voice metadata do not appear in the unfiltered response.

The Instructions field in the Text to Speech studio maps to the optional instructions property on /v1/audio/speech. Use it to describe delivery, such as tone or pace:

curl http://localhost:8080/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "tts",
    "input": "Welcome to LocalAI.",
    "instructions": "Speak slowly and warmly."
  }' --output speech.wav

Backend support for speech instructions varies. Backends that do not support this control may ignore it.

Voice Library

Every user who may use speech can keep reusable voice-cloning references of their own, and administrators manage all of them. Use the Library tab of the Voice add-on (Add-ons → Voice) in the LocalAI WebUI. (Recognition, the first tab of that page, is a separate feature: voiceprints for recognising who is speaking. See Voice recognition.) The library replaces per-model filesystem and YAML setup for supported cloning backends:

1. Select Add a speech voice and upload or record a clear reference clip. A speech voice can also be made while enrolling a speaker, from the enrol sheet.
2. Enter the exact words spoken in the clip. Add more audio/transcript pairs when the personality needs more examples.
3. Confirm that you have permission to clone the voice, then save the profile.
4. Open Text to Speech, choose a model marked Cloning ready, and select the saved voice.

The browser converts uploads and recordings to mono, 24 kHz, 16-bit PCM WAV so the same profile works across compatible backends. A profile can contain up to ten ordered references. Each clip must be between 1 and 120 seconds and no larger than 50 MiB; 6-30 seconds of clean audio is recommended. Profile audio is private biometric source material: LocalAI stores it below its configured data path, serves previews only to authenticated TTS users who may see that profile, and never returns its filesystem path.

Voice profile API

The WebUI uses the following endpoints. Every route needs access to the audio speech feature (audio_speech).

Who owns a profile

A profile has an owner: the user who created it. These rules apply to the endpoints above, to voice in /v1/audio/speech and /tts, and to the voice of a realtime session:

• You see, preview and speak with your own profiles and with the instance-wide ones.
• An instance-wide profile has no owner. Profiles an administrator creates are instance-wide, and so are profiles stored before ownership existed. Every user can use them; only an administrator can delete them (other users get 403).
• You can delete your own profiles. An administrator can list, use and delete all profiles.
• The profile of another user answers 404, never 403, in every endpoint and in speech requests, so nobody can learn that it exists.
• A user who is not an administrator may own at most 20 profiles. A request over the limit answers 409. The limit is a constant, not a setting. Each clip is limited to 50 MiB, as before. The reference audio of one such user is also limited to 100 MiB in total; a request over it answers 409. Administrators have neither limit. A profile that an administrator creates is visible to all users.
• A user who is not an administrator cannot set voice to a file path (a value with /, \ or ..): the request answers 400. They use a voice of the model or a localai://voice-profiles/<id> voice. Administrators, and servers without accounts, may still give a path.
• A server without accounts (authentication off) has no owners: every profile is instance-wide, as before.

The profiles are files below the data path of the instance (voice-profiles/, one folder per profile; owner_id in its profile.json). They follow the data path: frontends that share a data path share the profiles, and frontends with separate data paths each have their own library, as before. The profile limit is counted on the files under a lock of one process, so two frontends that create profiles for the same user at the same moment can exceed it by a few.

For example, you can create a profile without the WebUI:

curl http://localhost:8080/api/voice-profiles \
  -F 'name=Documentary narrator' \
  -F 'language=en-US' \
  -F 'transcript=The exact words spoken in this reference.' \
  -F 'consent_confirmed=true' \
  -F 'audio=@reference.wav;type=audio/wav'

Repeat transcript and audio in matching order to create a multi-reference personality:

curl http://localhost:8080/api/voice-profiles \
  -F 'name=Documentary personality' \
  -F 'transcript=The first exact transcript.' \
  -F 'audio=@reference-1.wav;type=audio/wav' \
  -F 'transcript=The second exact transcript.' \
  -F 'audio=@reference-2.wav;type=audio/wav' \
  -F 'consent_confirmed=true'

The response includes an opaque voice reference such as localai://voice-profiles/550e8400-e29b-41d4-a716-446655440000. Pass that value as voice to either TTS-compatible endpoint:

curl http://localhost:8080/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3-tts-base",
    "input": "This sentence will use the saved voice.",
    "voice": "localai://voice-profiles/550e8400-e29b-41d4-a716-446655440000"
  }' --output speech.wav

LocalAI resolves the opaque reference only for models that advertise voice-cloning support. Existing named speakers, backend-specific voice IDs, and explicit model YAML voice configuration remain available for models and advanced workflows that do not use the library.

The Voice Library uses the same server-side capability resolver for installed models and gallery recommendations. Administrators can inspect the currently configured galleries without maintaining a separate backend list:

curl 'http://localhost:8080/api/models?capability=voice_cloning&items=20'

Each returned model includes a non-null voice_cloning contract. Variant checks are applied before the model is returned, so TTS-only CustomVoice, VoiceDesign, or preset-prompt variants are not offered as reference-audio models. When the WebUI detects that no compatible model is installed, it uses this response to offer direct installation.

Model configuration

Voice Library support is automatic for known backends and model variants. A custom model can override that detection with tts.voice_cloning:

name: private-qwen-base
backend: qwen3-tts-cpp
parameters:
  model: private/qwen-talker-checkpoint.gguf
known_usecases:
  - tts
tts:
  # Optional: omit this for automatic backend and variant detection.
  voice_cloning: true
  # Optional model-wide fallback when a request does not select a saved profile.
  audio_path: voices/default-reference.wav
options:
  - tokenizer:private/qwen-tokenizer.gguf

tts.voice_cloning has three states:

The older options: ["voice_cloning:true"] and options: ["voice_cloning:false"] spellings remain accepted for compatibility. Prefer tts.voice_cloning; generic options may be forwarded to a backend, whereas the typed field is consumed only by LocalAI.

Reference selection follows this order:

1. A request voice, including a saved localai://voice-profiles/... URI.
2. The model’s tts.voice default.
3. The model’s tts.audio_path reference-audio fallback.
4. A backend-specific default voice or option.

When a saved profile is selected, LocalAI supplies its ordered private WAV and transcript pairs to Fish Speech and audio.cpp. Other cloning backends receive the first pair, which preserves their existing single-reference behavior. LocalAI does not rewrite the model YAML or copy recordings into the model directory.

Realtime pipeline default

Set tts.voice on a realtime pipeline model to use a saved Voice Library profile as the session default:

name: gpt-realtime
tts:
  voice: localai://voice-profiles/550e8400-e29b-41d4-a716-446655440000
pipeline:
  vad: silero-vad-ggml
  transcription: whisper-large-turbo
  llm: qwen3-4b
  tts: qwen3-tts-base

LocalAI resolves this profile when the realtime session starts. The selected TTS model must support Voice Library cloning.

You can also change the profile during a realtime session. Set audio.output.voice to a Voice Library URI in a session.update event:

{
  "type": "session.update",
  "session": {
    "audio": {
      "output": {
        "voice": "localai://voice-profiles/550e8400-e29b-41d4-a716-446655440000"
      }
    }
  }
}

If the same update changes model, the explicit audio.output.voice value takes precedence over the new model’s tts.voice default. The selected model must support Voice Library cloning.

Supported backend and model variants

This table describes the built-in resolver, not a frontend allowlist. Gallery entries and installed configs are evaluated by the server, and tts.voice_cloning can make a verified custom filename explicit.

Streaming TTS

LocalAI supports streaming TTS generation, allowing audio to be played as it’s generated. This is useful for real-time applications and reduces latency.

To enable streaming, add "stream": true to your request:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
  "input": "Hello world, this is a streaming test",
  "model": "voxcpm",
  "stream": true
}' | aplay

The audio will be streamed chunk-by-chunk as it’s generated, allowing playback to start before generation completes. This is particularly useful for long texts or when you want to minimize perceived latency.

You can also pipe the streamed audio directly to audio players like aplay (Linux) or save it to a file:

# Stream to aplay (Linux)
curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
  "input": "This is a longer text that will be streamed as it is generated",
  "model": "voxcpm",
  "stream": true
}' | aplay

# Stream to a file
curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
  "input": "Streaming audio to file",
  "model": "voxcpm",
  "stream": true
}' > output.wav

Note: Streaming TTS is implemented by the audio-cpp, crispasr, llama-cpp, magpie-tts-cpp, moss-tts-cpp, omnivoice-cpp, qwen3-tts-cpp, sherpa-onnx, supertonic, vibevoice-cpp and voxcpm backends. Other backends will fall back to non-streaming mode if streaming is not supported.

Backends

🐸 Coqui

Required: Don’t use LocalAI images ending with the -core tag,. Python dependencies are required in order to use this backend.

Coqui works without any configuration, to test it, you can run the following curl command:

    curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{         
        "backend": "coqui",
        "model": "tts_models/en/ljspeech/glow-tts",
        "input":"Hello, this is a test!"
        }'

You can use the env variable COQUI_LANGUAGE to set the language used by the coqui backend.

You can also use config files to configure tts models (see section below on how to use config files).

Fish Speech

Fish Speech models accept the compile backend option. Enabling it can improve inference performance on CUDA hardware, but the first request after loading the model includes the torch.compile warmup cost:

backend: fish-speech
options:
  - compile:true

When compilation is enabled, the backend uses the CUDA toolkit’s executable ptxas from $CUDA_HOME/bin (defaulting to /usr/local/cuda/bin) instead of the copy bundled with Triton. This allows newer GPU architectures supported by the installed CUDA toolkit to compile kernels. Set TRITON_PTXAS_PATH on the backend explicitly to select a different assembler.

Piper

To install the piper audio models manually:

• Download Voices from https://github.com/rhasspy/piper/releases/tag/v0.0.2
• Extract the .tar.tgz files (.onnx,.json) inside models
• Run the following command to test the model is working

To use the tts endpoint, run the following command. You can specify a backend with the backend parameter. For example, to use the piper backend:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
  "model":"it-riccardo_fasol-x-low.onnx",
  "backend": "piper",
  "input": "Ciao, sono Ettore"
}' | aplay

Note:

• aplay is a Linux command. You can use other tools to play the audio file.
• The model name is the filename with the extension.
• The model name is case sensitive.
• LocalAI must be compiled with the GO_TAGS=tts flag.

Transformers-musicgen

LocalAI also has experimental support for transformers-musicgen for the generation of short musical compositions. Currently, this is implemented via the same requests used for text to speech:

curl --request POST \
  --url http://localhost:8080/tts \
  --header 'Content-Type: application/json' \
  --data '{
    "backend": "transformers-musicgen",
    "model": "facebook/musicgen-medium",
    "input": "Cello Rave"
}' | aplay

Future versions of LocalAI will expose additional control over audio generation beyond the text prompt.

ACE-Step

ACE-Step 1.5 is a music generation model that can create music from text descriptions, lyrics, or audio samples. It supports both simple text-to-music and advanced music generation with metadata like BPM, key scale, and time signature.

Setup

Install the ace-step-turbo model from the Model gallery or run local-ai models install ace-step-turbo.

Usage

ACE-Step supports two modes: Simple mode (text description + vocal language) and Advanced mode (caption, lyrics, BPM, key, and more).

Simple mode:

curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
  "model": "ace-step-turbo",
  "input": "A soft Bengali love song for a quiet evening",
  "vocal_language": "bn"
}' --output music.flac

Advanced mode (using the /v1/sound-generation endpoint):

curl http://localhost:8080/v1/sound-generation -H "Content-Type: application/json" -d '{
  "model": "ace-step-turbo",
  "caption": "A funky Japanese disco track",
  "lyrics": "[Verse 1]\n...",
  "bpm": 120,
  "keyscale": "Ab major",
  "language": "ja",
  "duration_seconds": 225
}' --output music.flac

Music and sound generation API (/v1/sound-generation)

The /v1/sound-generation endpoint is compatible with the ElevenLabs sound generation API and can produce music, sound effects, and other audio content. It responds with a binary audio file and the appropriate Content-Type header (for example audio/wav, audio/mpeg, audio/flac, or audio/ogg). The request body is JSON and supports two usage modes.

Each generated sound has a unique filename. The filename remains unique when LocalAI changes its extension to match the audio format, so later generations do not overwrite earlier recordings in the WebUI history.

Simple mode:

Advanced mode:

Use params to pass backend-specific string options that do not have fixed request fields. Every value in params must be a string. Existing fixed fields remain authoritative when a fixed field and params contain the same option.

The audio-cpp backend uses a nonempty caption as the default style option. An explicit params.style takes precedence, including an empty string. In Studio → Sound, select Advanced mode and enter the musical style in Caption and the song text in Lyrics to use YuE2.

Install the YuE2 gallery model and generate a song:

local-ai models install audio-cpp-yue2-3b

curl http://localhost:8080/v1/sound-generation \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "audio-cpp-yue2-3b",
    "lyrics": "[Verse]\nMorning light across the bay\nCarry every cloud away",
    "params": {
      "style": "acoustic folk, warm vocals, gentle guitar",
      "cot": "off"
    }
  }' --output yue2.wav

Error responses: 400 for a missing or invalid model or request parameters, and 500 for a backend error during sound generation.

AudioLDM 2

AudioLDM 2 generates sound effects, music, and speech from a text description. Install the gallery model:

local-ai models install audioldm2

Generate a WAV file through the sound-generation endpoint:

curl http://localhost:8080/v1/sound-generation \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "audioldm2",
    "text": "Waves breaking on a rocky beach during a distant thunderstorm",
    "duration_seconds": 10
  }' --output storm.wav

AudioLDM 2 uses the AudioLDM2Pipeline from the diffusers backend. The duration_seconds field maps to the pipeline’s audio_length_in_s option, and prompt_influence maps to guidance_scale.

Configuration

You can configure ACE-Step models with various options:

name: ace-step-turbo
backend: ace-step
parameters:
  model: acestep-v15-turbo
known_usecases:
  - sound_generation
  - tts
options:
  - "device:auto"
  - "use_flash_attention:true"
  - "init_lm:true"  # Enable LLM for enhanced generation
  - "lm_model_path:acestep-5Hz-lm-0.6B"  # or acestep-5Hz-lm-4B
  - "lm_backend:pt"  # or vllm
  - "temperature:0.85"
  - "top_p:0.9"
  - "inference_steps:8"
  - "guidance_scale:7.0"

VibeVoice

VibeVoice-Realtime is a real-time text-to-speech model that generates natural-sounding speech from precomputed voice presets.

Setup

Install the vibevoice model in the Model gallery or run local-ai models install vibevoice.

Usage

Use the tts endpoint by specifying the vibevoice backend:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{         
     "model": "vibevoice",
     "input":"Hello!"
   }' | aplay

Voice presets

The Python vibevoice realtime 0.5B model uses .pt voice preset files. You can configure a model with a specific preset:

name: vibevoice
backend: vibevoice
parameters:
  model: microsoft/VibeVoice-Realtime-0.5B
tts:
  voice: "Frank"  # or use audio_path to specify a .pt file path
  # Available English voices: Carter, Davis, Emma, Frank, Grace, Mike

Then you can use the model:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
     "model": "vibevoice",
     "input":"Hello!"
   }' | aplay

OmniVoice

OmniVoice (omnivoice-cpp backend) is a native C++ / GGML text-to-speech engine. It supports voice cloning (from reference audio plus its transcript), voice design (steering the voice with attribute keywords such as gender, age, pitch, style, volume, and emotion), and streaming synthesis. Output is 24kHz mono audio and it covers 646 languages.

Setup

Install the omnivoice-cpp model in the Model gallery or run local-ai models install omnivoice-cpp. A higher-quality BF16 variant is available as omnivoice-cpp-hq (the default omnivoice-cpp ships Q8_0 GGUFs).

Usage

Use the speech endpoint by specifying the omnivoice-cpp backend:

curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
     "model": "omnivoice-cpp",
     "input": "Hello world, this is a test."
   }' | aplay

Voice cloning

Pass a reference audio file via the voice parameter and its transcript via the ref_text generation parameter:

curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
     "model": "omnivoice-cpp",
     "input": "Hello world, this is a test.",
     "voice": "path/to/reference_audio.wav",
     "params": { "ref_text": "This is the transcript of the reference audio." }
   }' | aplay

You can also pin a default cloned voice in the model config so callers do not have to pass it on every request. Both tts.voice and tts.audio_path are honored as the reference audio (a per-request voice overrides them); paths are resolved relative to the model directory:

name: omnivoice-cpp
backend: omnivoice-cpp
parameters:
  model: omnivoice-cpp/omnivoice-base-Q8_0.gguf
tts:
  voice_cloning: true                    # optional explicit declaration; gallery models are auto-detected
  audio_path: "voices/my_reference.wav"   # default cloning reference (or use tts.voice)
options:
  - "tokenizer:omnivoice-cpp/omnivoice-tokenizer-Q8_0.gguf"

Voice design

Steer the synthesized voice with attribute keywords (gender, age, pitch, style, volume, emotion) by passing an instructions string per request:

curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
     "model": "omnivoice-cpp",
     "input": "Hello world, this is a test.",
     "instructions": "female young high soft emotion:happy"
   }' | aplay

Configuration

The backend loads the base GGUF from parameters.model and its tokenizer from the tokenizer: option. A few optional generation knobs are available as options:

name: omnivoice-cpp
backend: omnivoice-cpp
parameters:
  model: omnivoice-cpp/omnivoice-base-Q8_0.gguf
options:
  - "tokenizer:omnivoice-cpp/omnivoice-tokenizer-Q8_0.gguf"
  - "use_fa:true"      # enable flash attention
  - "clamp_fp16:true"  # clamp activations for fp16 stability
  - "seed:42"          # deterministic generation
  - "denoise:true"     # denoise the generated audio

A per-request seed can also be supplied through the params map alongside ref_text.

Duration and postprocessing

For omnivoice-cpp, set these request-local controls in params, not as top-level fields:

{
  "model": "omnivoice-cpp",
  "input": "Hello world.",
  "params": {"duration":"2.5","postprocess":"false"}
}

• duration accepts a finite numeric string of seconds greater than zero and at most 30, inclusive. Whitespace, empty strings, and nonfinite values are invalid.
• The backend uses max(1, floor(seconds * 25)) generation frames. Each frame represents 40 milliseconds at the bundled codec rate. Thus, 2.5 selects 62 frames; tiny positive values select one frame.
• postprocess accepts only the exact strings "true" and "false". Buffered synthesis accepts either value, with or without duration.
• Omitted controls preserve engine defaults. Omitted duration leaves duration estimation and chunking unchanged.

A supplied duration forces single-shot generation and bypasses chunking. The 30-second cap is a conservative wrapper resource policy, aligned with the engine’s default chunk threshold. It is not an engine positional limit or a guarantee of available memory or successful synthesis.

Duration controls generation frames, not playback speed or guaranteed final WAV duration. Buffered postprocessing can trim or pad audio. With postprocess="false", buffered output preserves the raw decoded frame length; reference loudness matching remains active. Quantization does not guarantee perceptual accuracy.

Streaming accepts duration and explicit postprocess="true". It rejects postprocess="false" before reference reads, native synthesis, or audio output because the streaming engine always postprocesses audio.

Pocket TTS

Pocket TTS is a lightweight text-to-speech model designed to run efficiently on CPUs. It supports voice cloning through HuggingFace voice URLs or local audio files.

Setup

Install the pocket-tts model in the Model gallery or run local-ai models install pocket-tts.

Usage

Use the tts endpoint by specifying the pocket-tts backend:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{         
     "model": "pocket-tts",
     "input":"Hello world, this is a test."
   }' | aplay

Voice cloning

Pocket TTS supports voice cloning through built-in voice names, HuggingFace URLs, or local audio files. You can configure a model with a specific voice:

name: pocket-tts
backend: pocket-tts
tts:
  voice: "azelma"  # Built-in voice name
  # Or use HuggingFace URL: "hf://kyutai/tts-voices/alba-mackenna/casual.wav"
  # Or use local file path: "path/to/voice.wav"
  # Available built-in voices: alba, marius, javert, jean, fantine, cosette, eponine, azelma

To make a reference recording the model-wide fallback, use tts.audio_path. The gallery model is detected automatically; tts.voice_cloning is only needed when you want an explicit declaration:

name: pocket-tts-clone
backend: pocket-tts
tts:
  voice_cloning: true
  audio_path: "voices/reference.wav"

You can also pre-load a default voice for faster first generation:

name: pocket-tts
backend: pocket-tts
options:
  - "default_voice:azelma"  # Pre-load this voice when model loads

Then you can use the model:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{         
     "model": "pocket-tts",
     "input":"Hello world, this is a test."
   }' | aplay

Languages

Pocket TTS has a separate model for each language and loads the English one by default. Set the language option to load another:

name: pocket-tts-french
backend: pocket-tts
options:
  - "language:french"

Supported values depend on the installed pocket-tts version; current releases accept english, french, german, italian, spanish, portuguese and dutch, plus variants such as french_24l and english_2026-04. The built-in voice names are resolved against the selected language model.

Qwen3-TTS

Qwen3-TTS is a high-quality text-to-speech model that supports three modes: custom voice (predefined speakers), voice design (natural language instructions), and voice cloning (from reference audio).

Setup

Install the qwen-tts model in the Model gallery or run local-ai models install qwen-tts.

C++ / GGML gallery variants

For a native backend, install one of the Base variants qwen3-tts-cpp, qwen3-tts-cpp-0.6b-base-q4, qwen3-tts-cpp-1.7b-base, or qwen3-tts-cpp-1.7b-base-q4. These variants accept saved Voice Library profiles and are advertised automatically. Gallery entries containing customvoice or voicedesign provide their respective Qwen modes but are intentionally excluded from raw reference-audio cloning.

A private Qwen C++ Base conversion with an opaque filename can declare the capability explicitly. The tokenizer GGUF can sit beside the talker GGUF for automatic discovery:

name: company-narrator-engine
backend: qwen3-tts-cpp
parameters:
  model: qwen-private/talker.gguf
known_usecases:
  - tts
tts:
  voice_cloning: true
  audio_path: voices/default-reference.wav  # optional fallback

Concurrent requests

By default qwen3-tts-cpp synthesizes one request at a time. Set max_batch to let qwentts.cpp batch up to N concurrent /v1/audio/speech requests (streaming or not) into shared decode steps; further requests queue inside the engine:

options:
  - max_batch:4

The contributor measured four concurrent streaming requests through LocalAI on an RTX 5060 Ti (1.7B Base Q4_K_M): the first audio arrived after 0.31–0.33 seconds for all four with max_batch:4, compared with 0.30, 0.94, 1.48 and 2.02 seconds with max_batch:1. VRAM grows with max_batch (roughly 2.6 GB at 1, 6 GB at 4, 10.5 GB at 8 for that talker).

At load the backend attempts concurrent warmup to reduce graph initialization delays; this does not guarantee that every batch width is warmed. CustomVoice and VoiceDesign models reject these generic warmup requests because they require a speaker or instruction, so the backend logs a warning and continues loading. max_batch_warmup:false skips warmup.

Incremental synthesis (TTSLive, used by a realtime pipeline with streaming.tts: true) requires max_batch:1. On ROCm before 10.1 with RDNA4 (gfx1201), batched requests can hang the GPU in hipBLASLt (fixed in ROCm 10.1 by ROCm/rocm-libraries#8909); on those stacks add env: {ROCBLAS_USE_HIPBLASLT: "0"} to the model configuration.

llama.cpp gallery variants

llama.cpp gained native Qwen3-TTS support in ggml-org/llama.cpp#26254, so the llama-cpp backend can serve it on the same accelerator matrix it already uses for text generation: CUDA, ROCm, SYCL, Vulkan and Metal.

Install qwen3-tts-llamacpp (Q8_0 backbone) or qwen3-tts-llamacpp-q4 (Q4_K_M backbone) from the Model gallery, or run local-ai models install qwen3-tts-llamacpp-q4.

These models load two files: the backbone GGUF and a multimodal projector holding the speaker encoder and code predictor. A hand-written configuration must point at both:

name: qwen3-tts-llamacpp
backend: llama-cpp
known_usecases:
  - tts
mmproj: qwen3-tts-llamacpp/mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf
parameters:
  model: qwen3-tts-llamacpp/Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf

known_usecases: [tts] is not optional here. It is how LocalAI tells a Qwen3-TTS checkpoint apart from the text and vision GGUFs the same backend serves: without it the model is treated as a chat model, its projector is read as a vision tower, and Voice Library profiles are refused.

Importing such a repo through the Models page or local-ai models import writes that declaration for you. The importer reads the projector’s header and recognises the speech-synthesis pipeline, so ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF imports as a TTS model rather than as a chat model with a vision projector.

The upstream checkpoints are Base variants with no built-in speaker, so voice is required on every request. Pass either a path to a reference clip or a saved Voice Library profile. A request without one is rejected rather than served in an arbitrary voice:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
  "model": "qwen3-tts-llamacpp",
  "input": "Hello world, this is a test.",
  "voice": "voices/my-reference.wav"
}' > output.wav

Output is always 24 kHz mono 16-bit WAV. Streaming works on this backend, so "stream": true returns audio chunk by chunk as it is generated.

Set language to an ISO 639-1 code to pin the output language. The supported codes are zh, en, de, it, pt, es, ja, ko, fr and ru.

Three optional knobs travel in params: top_k and top_p adjust sampling, and max_frames caps how much audio a single request may generate. The model runs at 12.5 frames per second, so one frame is 0.08 seconds and the maximum duration in seconds is max_frames / 12.5. Leave it unset for the engine default of 512 frames, which is 40.96 seconds.

max_frames exists because generation occasionally fails to stop on its own. The model normally ends an utterance by emitting its end-of-speech token, but once in a while it does not, and the request then runs to the cap and returns far more audio than the text called for. It is uncommon, and it happens more on short inputs than on long ones. If you are synthesising predictable text and want a hard bound, allow roughly 8 frames per word: about 100 frames (8 seconds) for a short sentence, about 300 frames (24 seconds) for a paragraph.

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
  "model": "qwen3-tts-llamacpp",
  "input": "Hello world, this is a test.",
  "voice": "voices/my-reference.wav",
  "params": {"max_frames": "100"}
}' > output.wav

This backend accepts but ignores instructions, speed and sample_rate. The Base checkpoints have no expressive-style or rate control, and the output rate is fixed at 24 kHz.

Note that qwen3-tts-cpp (qwentts.cpp) remains available and is unaffected. It is a separate, independently maintained path to the same family of weights, not something this replaces.

Usage

Use the tts endpoint by specifying the qwen-tts backend:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{         
     "model": "qwen-tts",
     "input":"Hello world, this is a test."
   }' | aplay

Language

You can hint the synthesis language with the language request field:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
     "model": "qwen-tts",
     "input": "Bonjour le monde.",
     "language": "fr"
   }' | aplay

Supported languages: en (English), zh (Chinese), ru (Russian), ja (Japanese), ko (Korean), de (German), fr (French), es (Spanish), it (Italian), pt (Portuguese).

The value is matched case-insensitively and accepts a few forms for convenience:

• the two-letter code (fr, FR)
• a locale/region form, whose region is ignored (fr-FR, pt_BR, zh-Hans → fr/pt/zh)
• the English full name (french, Portuguese)

If the field is omitted or the value isn’t one of the supported languages, the backend defaults to English.

Custom Voice Mode

Qwen3-TTS supports predefined speakers. You can specify a speaker using the voice parameter:

name: qwen-tts
backend: qwen-tts
parameters:
  model: Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice
tts:
  voice: "Vivian"  # Available speakers: Vivian, Serena, Uncle_Fu, Dylan, Eric, Ryan, Aiden, Ono_Anna, Sohee

Available speakers:

• Chinese: Vivian, Serena, Uncle_Fu, Dylan, Eric
• English: Ryan, Aiden
• Japanese: Ono_Anna
• Korean: Sohee

Voice Design Mode

Voice Design allows you to create custom voices using natural language instructions. Configure the model with an instruct option:

name: qwen-tts-design
backend: qwen-tts
parameters:
  model: Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign
options:
  - "instruct:体现撒娇稚嫩的萝莉女声，音调偏高且起伏明显，营造出黏人、做作又刻意卖萌的听觉效果。"

Then use the model:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{         
     "model": "qwen-tts-design",
     "input":"Hello world, this is a test."
   }' | aplay

Per-request instructions

Instead of (or in addition to) the static YAML instruct option, you can pass an instructions string per request. It maps to the OpenAI instructions field and takes precedence over the YAML option when set, falling back to it when empty. This lets a single model config serve a different emotion (CustomVoice) or a different designed voice (VoiceDesign) on every request - useful for roleplay/narration clients that need many voices:

curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
     "model": "qwen-tts-design",
     "input": "Hello world, this is a test.",
     "instructions": "A calm, low-pitched elderly storyteller with a warm tone."
   }' | aplay

Backends that do not support style/voice instructions simply ignore the field.

You can also pass backend-specific generation parameters per request via the LocalAI params extension (a string-to-string map; values are coerced to the backend’s expected types). For example, with the Chatterbox backend:

curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -d '{
     "model": "chatterbox",
     "input": "Hello world, this is a test.",
     "params": { "exaggeration": "0.7", "cfg_weight": "0.3", "temperature": "0.8" }
   }' | aplay

Voice Clone Mode

Voice Clone allows you to clone a voice from reference audio. Configure the model with an AudioPath and optional ref_text:

name: qwen-tts-clone
backend: qwen-tts
parameters:
  model: Qwen/Qwen3-TTS-12Hz-1.7B-Base
tts:
  voice_cloning: true  # optional for this Base model; useful when a private checkpoint has an opaque name
  audio_path: "path/to/reference_audio.wav"  # Reference audio file
options:
  - "ref_text:This is the transcript of the reference audio."
  - "x_vector_only_mode:false"  # Set to true to use only speaker embedding (ref_text not required)

You can also use URLs or base64 strings for the reference audio. The backend automatically detects the mode based on available parameters (AudioPath → VoiceClone, instruct option → VoiceDesign, voice parameter → CustomVoice).

Then use the model:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{         
     "model": "qwen-tts-clone",
     "input":"Hello world, this is a test."
   }' | aplay

Multi-Voice Clone Mode

Qwen3-TTS also supports loading multiple voices for voice cloning, allowing you to select different voices at request time. Configure multiple voices using the voices option:

name: qwen-tts-multi-voice
backend: qwen-tts
parameters:
  model: Qwen/Qwen3-TTS-12Hz-1.7B-Base
options:
  - voices:[{"name":"jane","audio":"voices/jane.wav","ref_text":"voices/jane-ref.txt"},{"name":"john","audio":"voices/john.wav","ref_text":"voices/john-ref.txt"}]

The voices option accepts a JSON array where each voice entry must have:

• name: The voice identifier (used in API requests)
• audio: Path to the reference audio file (relative to model directory or absolute)
• ref_text: Path to the reference text file for the audio it is paired with

Then use the model with voice selection:

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{         
     "model": "qwen-tts-multi-voice",
     "input":"Hello world, this is Jane speaking.",
     "voice": "jane"
   }' | aplay

# Switch to a different voice
curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{         
     "model": "qwen-tts-multi-voice",
     "input":"Hello world, this is John speaking.",
     "voice": "john"
   }' | aplay

Voice Selection Priority:

1. voice parameter in the API request (highest priority)
2. voice option in the model configuration
3. Error if voice is not found among configured voices

Error Handling: If you request a voice that doesn’t exist in the voices list, the API will return an error with a list of available voices:

{"error": "Voice 'unknown' not found. Available voices: jane, john"}

Backward Compatibility: The multi-voice mode is backward compatible with existing single-voice configurations. Models using audio_path in the tts section will continue to work as before.

You can also use a config-file to specify TTS models and their parameters.

In the following example, a custom config loads xtts_v2 with a default cloning reference and language.

name: xtts_v2
backend: coqui
parameters:
  language: fr
  model: tts_models/multilingual/multi-dataset/xtts_v2

tts:
  voice_cloning: true
  audio_path: voices/reference.wav

For XTTS/YourTTS, tts.audio_path is the default cloning reference and a saved Voice Library profile overrides it per request. Other Coqui model families are not advertised as Voice Library-compatible unless they match the supported variant rules or are explicitly verified with tts.voice_cloning: true.

With this config, you can now use the following curl command to generate a text-to-speech audio file:

curl -L http://localhost:8080/tts \
    -H "Content-Type: application/json" \
    -d '{
"model": "xtts_v2",
"input": "Bonjour, je suis Ana Florence. Comment puis-je vous aider?"
}' | aplay

audio.cpp

audio.cpp is a multi-family GGML audio engine, so one installed backend covers TTS (supertonic, vibevoice, voxcpm2, fish_audio, pocket_tts, omnivoice, higgs_audio_tts), voice cloning (chatterbox, index_tts2, irodori_tts, moss_tts_local, moss_tts_nano) and voice design (qwen3_tts, irodori_tts), alongside ASR, VAD, diarization and separation.

Higgs Audio v3 is available directly from the model gallery as audio-cpp-higgs-audio-v3. It uses the Q8_0 GGUF validated by audio.cpp and supports expressive multilingual TTS and zero-shot voice cloning. For cloning, pass the path to a server-local WAV file in the OpenAI voice field. Use only reference audio for which you have the necessary rights and consent, and review the model’s research and non-commercial license before deployment.

local-ai models install audio-cpp-higgs-audio-v3

curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "audio-cpp-higgs-audio-v3",
    "input": "Welcome to LocalAI.",
    "voice": "/models/voices/reference.wav"
  }' \
  --output higgs.wav

name: supertonic
backend: audio-cpp
parameters:
  model: supertonic-3-orig.gguf
known_usecases:
  - FLAG_TTS
options:
  - backend:cuda
  - device:0

The family is read from the GGUF’s own metadata, so family: is only needed for a safetensors file or a package directory. Sending a speaker reference clip routes the request to the family’s voice-cloning task automatically, and supplying instructions routes it to voice design, where the family supports them. Cloning-only families such as chatterbox advertise no plain TTS task, so every request to them must carry a reference clip. See the audio.cpp backend page for the full option list, including the load. and session. namespaces and the supertonic packaging caveat.

NeMo-Speech.cpp (MagpieTTS)

NeMo-Speech.cpp is NVIDIA’s C++/ggml runtime for the Nemotron Speech models. Its magpietts family synthesizes speech through a NanoCodec decoder, and the same installed backend also covers transcription, diarization and translation.

name: magpie-tts
backend: nemo-speech-cpp
parameters:
  model: magpie-tts/magpietts.gguf
known_usecases:
  - FLAG_TTS
options:
  - codec_model:magpie-tts/nanocodec.gguf
  - tokenizer_dir:magpie-tts/extracted
  - gpu:0

The codec and the tokenizer directory are both required. Both are discovered from the model’s own directory when left unset: a sibling file whose name contains nanocodec and a sibling directory named extracted. MagpieTTS conditions on a speaker rather than on a prose style, so voice selects a speaker index or a voice name and instructions has no equivalent. See the NeMo-Speech.cpp backend page for the full option list and the conversion steps.

LFM3-Audio (llama.cpp, experimental)

LFM3-Audio-2B synthesises speech through the llama-cpp backend. Install the gallery entry lfm3-audio-2b (or lfm3-audio-2b-q8). Its voices are female (the default) and male. The language field takes a language code, and the default is en. The model uses the sampling defaults stored in its GGUF files. This needs a llama.cpp build with LFM3-Audio support.

Response format

To provide some compatibility with OpenAI API regarding response_format, ffmpeg must be installed (or a docker image including ffmpeg used) to leverage converting the generated wav file before the api provide its response.

Warning regarding a change in behaviour. Before this addition, the parameter was ignored and a wav file was always returned, with potential codec errors later in the integration (like trying to decode a mp3 file from a wav, which is the default format used by OpenAI)

Supported format thanks to ffmpeg are wav, mp3, aac, flac, opus, defaulting to wav if an unknown or no format is provided.

curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
  "input": "Hello world",
  "model": "tts",
  "response_format": "mp3"
}'

If a response_format is added in the query (other than wav) and ffmpeg is not available, the call will fail.

Sound prompt expansion

POST /v1/sound-generation accepts optional expansion_model to expand a sound description with an installed chat model before audio generation:

curl http://localhost:8080/v1/sound-generation \
  -H 'Content-Type: application/json' \
  -d '{
    "model_id": "music",
    "text": "A gentle piano instrumental with warm strings",
    "expansion_model": "chat",
    "instrumental": true
  }' --output music.wav

Replace music and chat with your installed audio and chat model names. The server requires non-empty text when you select expansion_model. Select an installed llama-cpp chat model with use_tokenizer_template: true and a tokenizer template that supports native tool calls. Expansion sends one strict, schema-backed expand_sound function tool with tool_choice: required. Other backends and configurations without tokenizer templates are rejected before inference. An explicit grammar is cleared for this request only; the installed model configuration is unchanged. The configuration flag alone does not guarantee that the selected model and template support native tools. The server validates the arguments and stops before audio generation if invalid.

If your account has a model allowlist, it must allow both the resolved audio model and the selected expansion model. Sound permission covers this internal expansion; separate Chat permission is not required. Omit expansion_model to preserve the backend’s existing prompt behavior. Python ACE-Step then keeps its native Simple-mode expansion. Advanced requests can continue to send caption, lyrics, and metadata directly.

The chat model must return exactly one expand_sound tool call with required caption and lyrics strings in its arguments. Assistant prose and reasoning are never used as fallback arguments. The server caps output at 2,048 tokens and accepts only these fields:

The keyscale tool schema enumerates the accepted spellings instead of using a regular expression. Use a root from A–G or a–g, optionally followed by # or b. The root can stand alone or have one space followed by major, minor, maj, min, dorian, phrygian, lydian, mixolydian, locrian, ionian, or aeolian. Mode names are lowercase; extra whitespace and other spellings are not accepted. This preserves the server’s existing vocabulary. Omit optional fields when unspecified; do not use an empty string or null for an unknown key. Empty lyrics remains valid for instrumental sound. Invalid-key errors quote and escape the rejected key, showing at most 64 characters and the total character count if truncated. They do not include other generated arguments.

Explicit instrumental: true clears generated lyrics. Your selected language (or vocal_language) takes precedence over generated language. An explicit duration takes precedence over generated duration. Conflicting caption, style, lyrics, and music metadata in params cannot replace expanded fields. The audio backend receives structured fields and an expansion marker, with the original text cleared. Backend support for expanded requests varies.

An unavailable model, a model without native tool support, a missing or wrong tool call, multiple calls, or invalid arguments returns a request error. Expansion failures stop the request before audio generation. Cancellation propagates to model loading, text inference, and audio generation.

In Studio → Sound → Simple, choose a Prompt expansion model before submitting. Sound uses the same model picker as Home, with search for longer model lists and keyboard navigation. The list contains installed, enabled chat models. You must explicitly choose a model or Backend default; opening or dismissing the picker does not select a model. Backend default omits expansion_model and uses the audio backend’s existing prompt behavior. Sound remembers either choice in this browser, separately from your chat model. If a remembered model disappears or loses chat support, choose again. Sound refreshes the list when you return to the browser tab. Without an installed chat model, choose Backend default to continue. Re-run with edits restores the expansion choice from that result. Older results without a saved expansion choice use Backend default. Advanced keeps its caption, lyrics, metadata, and thinking controls; it never sends expansion_model.

Expanded requests support these audio families:

Other audio-cpp families reject expanded requests before audio inference. The chat expansion may already have run when the backend reports this error. Choose Backend default to use that backend’s existing behavior. The UI does not detect backend families or silently retry without expansion.

Sound Classification

Sound-event classification (audio tagging) answers the question “what am I hearing?” - given an audio clip, it returns a list of scored AudioSet labels (e.g. Baby cry, infant cry, Glass breaking, Dog bark, Alarm).

LocalAI exposes this through the /v1/audio/classification endpoint, modelled after /v1/audio/transcriptions. The reference backend is ced.cpp (CED, a 527-class AudioSet tagger), a small ViT over a log-mel spectrogram ported to ggml with full PyTorch parity. Apache-2.0 weights are redistributable as GGUF.

parakeet.cpp can also load a CED model (through third_party/ced.cpp) and serve /v1/audio/classification from the same backend used for ASR and diarization. It scores the clip in 10 s windows and averages each class’s score across the windows before sorting and applying top_k/threshold - CED’s own method for clips longer than one window. Install parakeet-cpp-ced-tiny or parakeet-cpp-ced-base from the gallery, or point parameters.model at a CED GGUF under backend: parakeet-cpp. A parakeet-cpp ASR model can also point sound_model at a CED GGUF to add live sound events during realtime transcription - see Realtime API - and to add timed sound events to an offline diarization call with include_sounds=true - see Speaker Diarization. Such a model also answers /v1/audio/classification itself when it declares the sound_classification usecase, as the gallery entry parakeet-cpp-multilingual-diarization-speakers-sounds does.

Because classification is exposed as a regular OpenAI-style endpoint, any HTTP client works - there is no Python dependency on the consumer side.

In distributed mode, LocalAI stages uploaded audio and realtime sound-detection windows on the selected worker before classification. The API server and worker do not need a shared temporary directory.

Endpoint

POST /v1/audio/classification
Content-Type: multipart/form-data

Response

{
  "model": "ced-base-f16",
  "detections": [
    {"index": 23, "label": "Baby cry, infant cry", "score": 0.87},
    {"index": 22, "label": "Crying, sobbing", "score": 0.41}
  ]
}

Detections are returned in score-descending order. Scores are per-class probabilities (multi-label, independent), so they do not sum to 1.

Example

First install a classification model from the gallery (the example below uses ced-base-f16):

local-ai run ced-base-f16

curl http://localhost:8080/v1/audio/classification \
  -H "Content-Type: multipart/form-data" \
  -F file="@/path/to/clip.wav" \
  -F model="ced-base-f16" \
  -F top_k=10

The same request works unchanged against a parakeet-cpp CED model:

name: parakeet-ced-tiny
backend: parakeet-cpp
parameters:
  model: ced-tiny-q8_0.gguf
known_usecases:
  - sound_classification

curl http://localhost:8080/v1/audio/classification \
  -H "Content-Type: multipart/form-data" \
  -F file="@/path/to/clip.wav" \
  -F model="parakeet-ced-tiny" \
  -F top_k=10

The bundle entries parakeet-cpp-bundle-small and parakeet-cpp-bundle-standard also serve this endpoint: they hold CED-Small next to the transcription, VAD and diarization models (sound_component:ced). See Bundle GGUF files.

See also

• Audio to Text - speech transcription
• Speaker Diarization - who spoke when

Speaker Diarization

Speaker diarization answers the question “who spoke when?” - given an audio clip with multiple speakers, it returns time-stamped segments labelled with a stable speaker ID (SPEAKER_00, SPEAKER_01, …).

LocalAI exposes this through the /v1/audio/diarization endpoint, modelled after /v1/audio/transcriptions. Five backends are supported today:

• sherpa-onnx - pyannote-3.0 segmentation + a speaker-embedding extractor (3D-Speaker, NeMo, WeSpeaker) + fast clustering. Pure diarization - no transcription cost. Recommended when you only need speaker turns.
• vibevoice.cpp - produces speaker-labelled segments as a by-product of its long-form ASR pass, so you can optionally get a transcript per segment for free.
• NeMo-Speech.cpp - NVIDIA Sortformer, served standalone by the NeMo-Speech.cpp backend. It is end to end, so the speaker capacity is fixed by the checkpoint and the count hints are ignored. The same backend can instead put speaker tags on a transcript, by attaching a Sortformer model to an ASR one.
• audio.cpp - the sortformer_diar family, served by the multi-modality audio.cpp backend.
• parakeet.cpp - NVIDIA Nemotron-3-Diarization (Sortformer, up to 8 speakers), served standalone or paired with a Parakeet ASR model for per-segment text. See the Audio to Text page for the parakeet-cpp option reference.

Because diarization is exposed as a regular OpenAI-compatible endpoint, any HTTP client works. There is no Python dependency on pyannote or NeMo on the consumer side.

In distributed mode, LocalAI stages uploaded audio on the remote worker before running dedicated diarization and releases the staged input after the request. The worker does not need access to the frontend’s temporary upload directory.

Endpoint

POST /v1/audio/diarization
Content-Type: multipart/form-data

Response - json (default)

Compact payload, no transcription, no per-speaker summary:

{
  "task": "diarize",
  "duration": 12.34,
  "num_speakers": 2,
  "segments": [
    {"id": 0, "speaker": "SPEAKER_00", "label": "0", "start": 0.00, "end": 2.34},
    {"id": 1, "speaker": "SPEAKER_01", "label": "1", "start": 2.34, "end": 4.10}
  ]
}

speaker is the normalized, zero-padded label clients should display. label preserves the raw backend-emitted ID for clients that maintain their own speaker dictionary.

Response - verbose_json

Adds per-speaker totals and (when the backend supports it and include_text=true) the per-segment transcript:

{
  "task": "diarize",
  "duration": 12.34,
  "language": "en",
  "num_speakers": 2,
  "segments": [
    {"id": 0, "speaker": "SPEAKER_00", "label": "0", "start": 0.00, "end": 2.34, "text": "Hello, world."},
    {"id": 1, "speaker": "SPEAKER_01", "label": "1", "start": 2.34, "end": 4.10, "text": "How are you?"}
  ],
  "speakers": [
    {"id": "SPEAKER_00", "label": "0", "total_speech_duration": 5.6, "segment_count": 3},
    {"id": "SPEAKER_01", "label": "1", "total_speech_duration": 1.76, "segment_count": 1}
  ]
}

Speaker names

With a parakeet-cpp model that has a speaker_model: (or a speaker_component:, for a bundle file such as parakeet-cpp-bundle-standard) and voices registered through /v1/voice/register, segments whose speaker matches a registered voice gain name and name_score (the cosine similarity of the match), and the matching speakers entry gains name. Both fields are omitted for a speaker that was not identified, so an unnamed response looks exactly as before. speaker stays SPEAKER_NN, and RTTM output still uses SPEAKER_NN. See Voice Recognition for the setup and the limits.

{
  "task": "diarize",
  "duration": 12.34,
  "num_speakers": 2,
  "segments": [
    {"id": 0, "speaker": "SPEAKER_00", "label": "0", "start": 0.00, "end": 2.34, "text": "Hello, world.", "name": "Alice", "name_score": 0.82},
    {"id": 1, "speaker": "SPEAKER_01", "label": "1", "start": 2.34, "end": 4.10, "text": "How are you?"}
  ],
  "speakers": [
    {"id": "SPEAKER_00", "label": "0", "name": "Alice", "total_speech_duration": 5.6, "segment_count": 3},
    {"id": "SPEAKER_01", "label": "1", "total_speech_duration": 1.76, "segment_count": 1}
  ]
}

Sound events

With include_sounds=true the response gains a sounds array: the sound events (AudioSet labels such as Dog, Applause, Cough) found anywhere in the clip, in the same call that returns the speakers, the text and the voice prints. Each item is {start, end, label, confidence}: start and end are in seconds from the start of the audio, and confidence is the peak score the tagger reached while the event lasted (0 to 1). Items are sorted by start. Both json and verbose_json carry the array; rttm has no place for it and returns 400.

curl http://localhost:8080/v1/audio/diarization \
  -F model=parakeet-cpp-multilingual-diarization-speakers-sounds \
  -F file=@meeting.wav \
  -F include_text=true -F include_speaker_profiles=true -F include_sounds=true \
  -F response_format=verbose_json

{
  "task": "diarize",
  "duration": 31.2,
  "language": "en",
  "num_speakers": 2,
  "segments": [
    {"id": 0, "speaker": "SPEAKER_00", "label": "0", "start": 0.0, "end": 6.4, "text": "Good morning, everyone."},
    {"id": 1, "speaker": "SPEAKER_01", "label": "1", "start": 6.8, "end": 11.2, "text": "Morning."}
  ],
  "speakers": [
    {"id": "SPEAKER_00", "label": "0", "total_speech_duration": 6.4, "segment_count": 1},
    {"id": "SPEAKER_01", "label": "1", "total_speech_duration": 4.4, "segment_count": 1}
  ],
  "speaker_profiles": {"version": 1, "encoder": {"identity": "sha256:...", "dimension": 256}, "speakers": ["..."]},
  "sounds": [
    {"start": 5.5, "end": 6.75, "label": "Cough", "confidence": 0.91},
    {"start": 12.0, "end": 14.5, "label": "Applause", "confidence": 0.68}
  ]
}

An empty sounds array means the sound model ran and found no event. The field is absent when include_sounds is not set. The events come from the same sound stream, with the same on and off thresholds, that a realtime session runs, so a clip gives the same events offline and live. The thresholds are not request fields.

The model needs a sound companion, a sound_model: option pointing at a CED GGUF (the gallery entry parakeet-cpp-multilingual-diarization-speakers-sounds has one). Without it the request fails with HTTP 501 and a message that starts with the stable code include_sounds_unsupported; the same happens for a backend that cannot report sound events. LocalAI never answers with an empty list in place of that error. The speaker segments, text and voice prints are independent of the sound model and cost nothing extra when include_sounds is off.

Response - rttm

NIST RTTM, the standard interchange format used by pyannote.metrics / dscore:

SPEAKER audio 1 0.000 2.340 <NA> <NA> SPEAKER_00 <NA> <NA>
SPEAKER audio 1 2.340 1.760 <NA> <NA> SPEAKER_01 <NA> <NA>

Returned as Content-Type: text/plain; charset=utf-8.

Quick start

First install a diarization-capable model from the gallery. The example below uses vibevoice-cpp-asr, which serves the vibevoice.cpp backend and returns speaker-labelled segments (and, optionally, a transcript):

local-ai run vibevoice-cpp-asr

curl http://localhost:8080/v1/audio/diarization \
  -H "Content-Type: multipart/form-data" \
  -F file="@meeting.wav" \
  -F model="vibevoice-cpp-asr" \
  -F num_speakers=3

The sections below show how to configure the two supported backends by hand when you want full control over the segmentation and embedding models.

Backend setup - sherpa-onnx (pure diarization)

Sherpa-onnx needs two ONNX models: pyannote segmentation and a speaker-embedding extractor. Place them under your LocalAI models directory and reference them from the YAML:

name: pyannote-diarization
backend: sherpa-onnx
type: diarization
parameters:
  model: sherpa-onnx-pyannote-segmentation-3-0/model.onnx
options:
  - diarize.embedding_model=3dspeaker_speech_campplus_sv_zh-cn_16k-common.onnx
  # Optional clustering knobs (per-call DiarizeRequest fields override these):
  - diarize.threshold=0.5
  - diarize.min_duration_on=0.3
  - diarize.min_duration_off=0.5
known_usecases:
  - FLAG_DIARIZATION

Both model: and diarize.embedding_model= are resolved relative to the LocalAI models directory.

Backend setup - vibevoice.cpp (diarization + ASR)

vibevoice.cpp’s ASR mode emits [{Start, End, Speaker, Content}] natively, so a single pass gives both diarization and transcription:

name: vibevoice-diarize
backend: vibevoice-cpp
parameters:
  model: vibevoice-asr.gguf
options:
  - type=asr
  - tokenizer=vibevoice-tokenizer.gguf
known_usecases:
  - FLAG_DIARIZATION
  - FLAG_TRANSCRIPT

Pass include_text=true on the request to populate the text field on each diarization segment.

curl http://localhost:8080/v1/audio/diarization \
  -H "Content-Type: multipart/form-data" \
  -F file="@interview.wav" \
  -F model="vibevoice-diarize" \
  -F include_text=true \
  -F response_format=verbose_json

Backend setup - parakeet-cpp (Nemotron-3-Diarization)

Choose an existing gallery entry for the output you need:

The complete -asr-speakers entry downloads Nemotron-3-Diarization, Parakeet TDT+CTC 110M ASR, and the WeSpeaker ResNet34 speaker encoder. It configures both asr_model and speaker_model; no custom gallery configuration is needed. See Remember speakers in the Web UI for installation and enrollment.

The parakeet-cpp-multilingual-diarization-speakers-sounds entry downloads Parakeet TDT 0.6B v3 (25 European languages), Nemotron-3-Diarization, the WeSpeaker ResNet34 speaker encoder and CED-Tiny, and sets diarization_model, speaker_model and sound_model. It answers the whole request above from one model name. See Sound events.

The entries parakeet-cpp-bundle-small and parakeet-cpp-bundle-standard hold Nemotron-3-Diarization, an ASR model and the WeSpeaker speaker encoder in one file (diar_component:diar and speaker_component:voice), so one install serves the transcript and the identification options above. See Bundle GGUF files.

For manual configuration, this example pairs Sortformer with ASR:

name: parakeet-diarize
backend: parakeet-cpp
parameters:
  model: nemotron-3-diarization-q8_0.gguf
options:
  - asr_model:tdt_ctc-110m-f16.gguf
known_usecases:
  - diarization

Getting text on each segment needs both: an asr_model companion loaded on the model, and include_text=true on the request. With only one of the two, segments carry no text and no error is raised. Sortformer has a fixed speaker capacity and no clustering stage, so num_speakers, min_speakers, max_speakers and clustering_threshold are ignored (logged at debug); min_duration_on and min_duration_off are honored. Speaker labels are the decimal index the model assigned ("0", "1", …), or "unknown" when a segment has no diarized speaker.

curl http://localhost:8080/v1/audio/diarization \
  -H "Content-Type: multipart/form-data" \
  -F file="@meeting.wav" \
  -F model="parakeet-diarize" \
  -F include_text=true \
  -F response_format=verbose_json

Sortformer clusters on voice-like characteristics, not on “is this a human”. A loud non-speech sound with voice-like pitch and rhythm (a rooster crow, in one test clip) can come back as its own speaker segment alongside the real speakers. This is model behavior, not a bug in the LocalAI integration: treat an unexpected extra speaker as a hint the clip may contain a non-speech sound, and use Sound Classification to confirm what it is.

Notes

• Speaker identity across files: speaker IDs (SPEAKER_00, SPEAKER_01, …) are local to each request. To track the same person across multiple recordings, combine /v1/audio/diarization with /v1/voice/embed (speaker embedding) and maintain your own embedding store.
• Hints vs. forces: num_speakers overrides clustering when set; min_speakers / max_speakers are advisory and only honored by backends that expose a range hint. vibevoice.cpp and parakeet-cpp (Sortformer) ignore them - the model picks the count itself.
• Sample rate: input is automatically converted to 16 kHz mono via ffmpeg before the backend sees it; sherpa-onnx pyannote-3.0 requires 16 kHz.

See also

• Sound Classification - tag non-speech sound events (alarms, glass breaking, baby cry) in a clip.

Backend profile transport

The parakeet backend supports opt-in speaker profile export through the internal DiarizeRequest.include_speaker_profiles field. This native transport underpins HTTP profile export and explicit enrollment through POST /v1/voice/register, as described in Portable speaker enrollment below. It requires a configured speaker_model and a library with parakeet_capi_diarize_profiles_pcm_json; an empty recognition registry is supported. Export does not register anyone. With include_text and a loaded ASR companion, one profile-capable diarization supplies all speaker slots, profiles, names, and intervals. Timestamped ASR words are assigned to those same slots; the backend does not run a second diarization. Either inference failure fails the request. If no ASR companion is loaded, the existing fallback applies: the response includes profiles and diarization segments without text. A loaded ASR companion without the timestamped PCM API returns an explicit error.

DiarizeResponse.speaker_profiles_json carries the native version-1 speaker_profiles object, including original clean preview intervals and one embedding per usable speaker. Normal requests retain their existing output. Profile speaker values are raw native slot IDs. Match their decimal string to segment label or speaker-summary label, not to normalized SPEAKER_NN, array position, or display name. Slots can be sparse, and profile order can differ from transcript order. Profiles retain their original clean intervals even when transcript segments use word boundaries or duration filters. These vectors are sensitive biometric data: callers must authorize export and explicit enrollment separately.

The internal backend Status response supplies speaker_encoder, derived from the loaded encoder’s SHA-256 identity and dimension. Enrollment code must use backend.ModelSpeakerEncoder with server-selected model configuration and validate profiles against that result, never against caller-provided metadata. Unavailable metadata or unsupported export fails closed. Renaming a GGUF does not change its identity; modifying or quantizing its bytes does.

Recognition replay carries registration IDs separately from display names. Distinct IDs with the same display name remain independent native entries, and both offline and realtime matches are translated back to display names. Legacy transport clients without IDs retain name-keyed behavior. The native registry’s aggregation defaults are unchanged. LocalAI’s recognition registry remains global and in-memory; this adds neither persistence nor automatic registration and is unrelated to persistent TTS voice cloning.

Portable speaker enrollment

Profile-capable parakeet models can export one biometric embedding per discovered speaker, including when the recognition registry is empty. Export is opt-in:

curl http://localhost:8080/v1/audio/diarization \
  -F model=parakeet-diarization -F file=@conversation.wav \
  -F include_speaker_profiles=true -F include_text=true \
  -F response_format=verbose_json

The /audio/diarization alias has the same protection. With user authentication, export additionally requires the voice-recognition permission. Existing model access controls still apply. Without opt-in, speaker_profiles is omitted. Both json and verbose_json support profiles; rttm with profiles returns 400. include_text=true retains supported transcripts in either JSON format. Unsupported profile backends return 501 rather than silently omitting profiles.

Alternatively send Content-Type: application/json:

{
  "model": "parakeet-diarization",
  "file": "<raw base64 audio bytes>",
  "include_speaker_profiles": true,
  "include_text": true,
  "response_format": "verbose_json"
}

The speaker_profiles response object contains version: 1, encoder: {"identity": "sha256:<64 lowercase hex digits>", "dimension": N}, and speakers. Each speaker contains:

• speaker: the raw numeric speaker slot;
• clean_duration: retained clean speech in seconds;
• intervals: {start, end} ranges in seconds in the original recording;
• unavailable_reason: null for usable profiles, otherwise a reason string;
• embedding: one vector for a usable speaker, omitted when unavailable.

UI association: convert each profile’s numeric speaker to a decimal string and match segment/summary label. Do not use SPEAKER_NN, array position, or human name. Slots may be sparse and out of order; display names may repeat. Preview intervals against the original audio, not separated audio. Disable saving unavailable profiles. Enrollment is explicit, never automatic; only relabel after a successful registration response. See portable voice registration.

Profiles are sensitive, unsigned biometric data, not proof of identity or consent. Do not log their vectors. Obtain the speaker’s consent before enrollment.

API tracing excludes the entire exchange for /v1/audio/diarization, its /audio/diarization alias, and /v1/voice/register before capturing bodies. This also protects JSON base64 audio when profile export is off. These routes produce no in-memory or persisted API trace; other routes keep their existing tracing behavior. External proxies and client logs must apply the same privacy policy. Existing trace files from older versions are not retroactively scrubbed.

Remember speakers in the Web UI

Use a LocalAI build with portable enrollment support and a profile-capable parakeet-cpp backend. The backend needs the profile APIs from merged upstream commit bee7c14 or a compatible later build. Installing the model weights alone does not update an older backend.

1. Open Models → Explore and search for parakeet-cpp-nemotron-3-diarization-asr-speakers.
2. Select Install and wait for installation to complete. Check Operate → Activity for progress or errors.
3. Open Studio → Diarization (or /app/diarization). Select that model and upload your recording.

Obtain the speaker’s consent before enrollment. To remember a speaker from that recording:

1. Select Prepare speakers to remember, then select Diarize. This requests profiles, transcript text, and speaker summaries. Use a profile-capable parakeet-cpp model configured with a speaker encoder.
2. In Speakers, select Preview 1, Preview 2, or another available interval to listen to clean speech from the original recording. Playback stops at the end of that interval. Stop preview stops it earlier. Your browser must support the recording’s audio format.
3. For an unknown speaker, select Name and remember. Enter a name and select Remember. No second recording or audio upload is needed.
4. After the server confirms registration, the name appears on all turns for that speaker. A failed save keeps the entered name so you can retry.

Upload another recording and select Diarize to match remembered voices. You can turn off Prepare speakers to remember; recognition does not require another profile export. Matches show their names; unmatched speakers keep their speaker labels. With preparation off, the UI requests speaker turns without transcript text. Use the API example below to request text without exporting profiles.

Speakers without a usable profile cannot be remembered; try longer speech without overlapping speakers. Duplicate names are allowed: each save creates a separate registration, not a merged voice. Changing the model or recording clears the current results and save dialog. A save already sent to the server can still complete, but cannot rename turns in a different recording.

The page requires the Audio Diarization permission and access to the selected model. Preparing profiles and remembering speakers additionally require Voice Recognition. Users without that permission can still run normal diarization. If the backend does not support profiles, the page reports an error: choose a compatible model or turn off Prepare speakers to remember. It does not silently retry without profiles.

Use Manage remembered voices, then the Enrollment tab, to see or remove registrations saved in this browser. Clean-clip voice enrollment stays available there and does not require diarization.

API example: install, export, and remember

This example uses the same complete gallery entry and requires curl and jq. The commands assume a local server without authentication. If authentication is enabled, add -H "Authorization: Bearer <key>" to every request using your authorized key. Keep keys out of shared scripts, logs, and shell history; see Authentication. Installation requires model-management access; inference and enrollment require the permissions described above.

Install the model if it is not already installed:

LOCALAI=http://localhost:8080
MODEL=parakeet-cpp-nemotron-3-diarization-asr-speakers
curl --fail-with-body "$LOCALAI/models/apply" \
  -H 'Content-Type: application/json' \
  -d '{"id":"localai@parakeet-cpp-nemotron-3-diarization-asr-speakers"}'

Installation is asynchronous. Wait for successful completion in Operate → Activity before continuing. API clients can query the returned job status URL; see the model gallery API.

Export profiles and transcript text from your recording, keeping the complete JSON response:

umask 077
WORK=$(mktemp -d)
curl --fail-with-body "$LOCALAI/v1/audio/diarization" \
  -F "model=$MODEL" -F file=@conversation.wav \
  -F include_text=true -F include_speaker_profiles=true \
  -F response_format=verbose_json > "$WORK/diarization.json"

# Inspect raw slots, clean intervals, and transcript labels without printing vectors.
jq '.speaker_profiles.speakers[] | {speaker, clean_duration, intervals, unavailable_reason}' \
  "$WORK/diarization.json"
jq '.segments[] | {label, start, end, text}' "$WORK/diarization.json"

Choose a usable raw speaker slot whose decimal string matches the intended segment label. Listen to its intervals in the original recording before assigning a name. Do not select by array position, normalized SPEAKER_NN, or display name. If unavailable_reason indicates insufficient speech, try a longer recording without overlapping speakers.

Replace 0 below with your chosen raw slot. Zero is valid, but does not mean “the first array element.” Keep the complete speaker_profiles object unchanged:

SLOT=0
NAME=Ada
jq --arg model "$MODEL" --arg name "$NAME" --argjson slot "$SLOT" \
  '{model: $model, name: $name, speaker_slot: $slot, speaker_profiles: .speaker_profiles}' \
  "$WORK/diarization.json" > "$WORK/register.json"
curl --fail-with-body "$LOCALAI/v1/voice/register" \
  -H 'Content-Type: application/json' \
  --data-binary @"$WORK/register.json"

After successful registration, submit another recording with the same model:

curl --fail-with-body "$LOCALAI/v1/audio/diarization" \
  -F "model=$MODEL" -F file=@next-conversation.wav \
  -F include_text=true -F response_format=verbose_json > "$WORK/next.json"
jq '.segments[] | {label, name, start, end, text}' "$WORK/next.json"

# Remove private example outputs when no longer needed.
rm -f "$WORK/diarization.json" "$WORK/register.json" "$WORK/next.json"
rmdir "$WORK"

Matching speakers can now carry name, even though this request omits include_speaker_profiles. Keep include_text=true and verbose_json when you want transcript text. Recognition is not proof of identity. Registrations remain global and disappear on server restart. See portable profile registration for encoder compatibility and validation rules.

Audio Transform

The audio-transform endpoints take audio in and emit audio out, optionally conditioned on a second reference audio signal. The category is generic by design - concrete operations include joint acoustic echo cancellation + noise suppression + dereverberation (LocalVQE), voice conversion (reference = target speaker), pitch shifting, audio super-resolution, and so on.

The first shipping backend is LocalVQE, a 1.3 M-parameter GGML-based model that performs joint AEC + noise suppression

• dereverberation on 16 kHz mono speech, ~9.6× realtime on a desktop CPU. It is a derivative of the Microsoft DeepVQE paper.

Source separation and voice conversion are served by the audio.cpp backend: its htdemucs and mel_band_roformer families produce the named stems described below, and seed_vc, vevo2 and miocodec do voice conversion against a reference speaker.

The mental model

Every audio-transform request carries:

• audio - the primary input file (required).
• reference - an auxiliary signal whose meaning is backend-specific (optional).
  • For echo cancellation: the loopback / far-end signal played through the speakers.
  • For voice conversion: the target speaker’s reference clip.
  • For pitch / style transfer: a tonal or style reference.
  • When omitted, the backend treats it as silence and degrades gracefully (LocalVQE, for example, does denoise + dereverb only when ref is empty).
• params - a generic key=value map forwarded to the backend.
  • LocalVQE keys: noise_gate=true|false, noise_gate_threshold_dbfs=<float>.

This shape mirrors WebRTC’s ProcessStream(near) / ProcessReverseStream(far) APM API, NVIDIA Maxine’s NvAFX_Run paired-stream signature, and the ICASSP AEC challenge 2-channel WAV convention.

Batch endpoint

POST /audio/transformations (alias POST /audio/transform) - multipart form-data, returns audio bytes.

First install an audio-transform model from the gallery (the examples below use localvqe-v1.3-4.8m):

local-ai run localvqe-v1.3-4.8m

Example (LocalVQE: cancel echo, suppress noise, gate residual):

curl -X POST http://localhost:8080/audio/transformations \
  -F model=localvqe-v1.3-4.8m \
  -F audio=@mic.wav \
  -F reference=@loopback.wav \
  -F 'params[noise_gate]=true' \
  -F 'params[noise_gate_threshold_dbfs]=-50' \
  -o enhanced.wav

When reference is omitted, LocalVQE zero-fills the reference channel and the operation reduces to noise suppression + dereverberation.

What LocalAI does to your upload before the backend sees it

By default, nothing: the file reaches the backend at its own sample rate and its own channel count. A WAV already carrying plain 16-bit PCM is passed through byte for byte; any other container or encoding is transcoded to 16-bit PCM WAV with the rate and the channel layout kept.

The exception is a backend that declares it needs a fixed input shape. LocalVQE does: its echo cancellation is trained on 16 kHz mono and needs the primary input and the reference in the same shape, so uploads for it are folded to 16 kHz mono s16 with ffmpeg. The declaration is BackendCapability.AudioTransformInputMono16k in core/config, and localvqe is currently the only backend that sets it.

This matters for anything that is not speech enhancement. Source separation models refuse any sample rate but their checkpoint’s own (44.1 kHz for every published htdemucs and mel_band_roformer checkpoint) and rely on the stereo image to tell a centred vocal from a wide mix, so a 16 kHz mono downmix would remove both the format they accept and the cue they work from.

Multi-output transforms (source separation stems)

Some transforms produce several named outputs from one run: htdemucs yields drums, bass, other and vocals in a single pass. The response body can carry only one file, so:

• The backend runs once and writes every stem beside the main output.
• params[stem]=<name> chooses which one the body carries. Without it the default is vocals when the model has one, and the model’s first output otherwise. An unknown stem name is refused with an error listing the real ones, never silently substituted.
• Every stem, including the one in the body, is named in the X-Audio-Stems response header, a compact JSON array:

X-Audio-Stems: [{"name":"drums","url":"/generated-audio/transform.drums.wav"},
                {"name":"bass","url":"/generated-audio/transform.bass.wav"},
                {"name":"other","url":"/generated-audio/transform.other.wav"},
                {"name":"vocals","url":"/generated-audio/transform.vocals.wav"}]

Fetch any of those URLs to get the other stems without paying for a second separation. sample_rate and response_format are applied to the stems as well as to the body, so the whole set stays in the shape you asked for. The header is listed in Access-Control-Expose-Headers, so browser clients can read it.

Single-output transforms (echo cancellation, voice conversion) do not set the header at all, and params[stem] against such a model is refused rather than ignored.

# isolate the vocals (the default), then see where the other stems went
curl -sS -D headers.txt -X POST http://localhost:8080/audio/transformations \
  -F model=htdemucs -F audio=@song.wav -o vocals.wav
grep -i '^x-audio-stems' headers.txt

# or ask for a specific stem in the body
curl -sS -X POST http://localhost:8080/audio/transformations \
  -F model=htdemucs -F audio=@song.wav -F 'params[stem]=drums' -o drums.wav

The stems live in the generated-content directory beside the main output and are served from /generated-audio/. Like every other generated artifact, they are not swept automatically.

Text-conditioned transforms (speech to speech)

Most transforms are audio in, audio out. A few are not: a speech-to-speech model resynthesises a line in another voice and needs to know what the line says. /audio/transform has no text field, so the text travels as params[text].

curl -sS -X POST http://localhost:8080/audio/transformations \
  -F model=audio-cpp-vevo2-speech-to-speech \
  -F audio=@source.wav \
  -F reference=@target-speaker.wav \
  -F 'params[text]=The quick brown fox jumps over the lazy dog.' \
  -o converted.wav