Voice Recognition

LocalAI supports voice (speaker) recognition: speaker verification (1:1), speaker identification (1:N) against a built-in vector store, speaker embedding, and demographic analysis (age / gender / emotion from voice).

The audio analog to Face Recognition, served over the same /v1/voice/* HTTP API by two backends:

• voice-detect (recommended, default). A standalone C++/ggml engine (voice-detect.cpp): no Python, no onnxruntime, no torch runtime. Each gallery entry is a single self-describing GGUF. This is the recommended option for new deployments.
• speaker-recognition (Python). The original SpeechBrain / ONNX backend. Still supported; see the Python backend below.

Both backends expose the identical wire format, so the API examples on this page work with either - only the gallery entry name (the model field) changes.

voice-detect (ggml) backend

The voice-detect backend reads the embedding (or analysis) architecture (voicedetect.arch) directly from the GGUF metadata, so installing a gallery entry is all that is needed to select an engine. It drives the VoiceEmbed / VoiceVerify / VoiceAnalyze gRPC rpcs behind the /v1/voice/{embed,verify,analyze,register,identify,forget} endpoints.

Gallery entries

The four speaker-recognition entries drive verify / embed / identify. voice-detect-emotion-wav2vec2 is the analysis head behind /v1/voice/analyze (continuous age estimate plus gender and emotion class scores) and is non-commercial / research use only.

Quickstart

Install the default entry (recommended for copy-paste):

local-ai models install voice-detect-ecapa-tdnn

Verify that two audio clips were spoken by the same person:

curl -sX POST http://localhost:8080/v1/voice/verify \
  -H "Content-Type: application/json" \
  -d '{
    "model": "voice-detect-ecapa-tdnn",
    "audio1": "https://example.com/alice_1.wav",
    "audio2": "https://example.com/alice_2.wav"
  }'

Analyze age / gender / emotion (install the analyze entry first):

local-ai models install voice-detect-emotion-wav2vec2

curl -sX POST http://localhost:8080/v1/voice/analyze \
  -H "Content-Type: application/json" \
  -d '{"model": "voice-detect-emotion-wav2vec2", "audio": "https://example.com/alice.wav"}'

The 1:N register / identify / forget workflow and the rest of the API are identical to the API reference below - just pass a voice-detect-* model name. The default verify threshold is ~0.25 for the ECAPA-TDNN / ERes2Net / CAM++ recognizers and ~0.30 for WeSpeaker ResNet34.

speaker-recognition (Python) backend

The speaker-recognition backend follows the same two-engine pattern under one image.

Engines

Both entries are commercial-safe Apache-2.0. SpeechBrain is the default - it’s a lightweight pure-PyTorch checkpoint that auto- downloads on first use. The wespeaker-resnet34 entry wires the direct-ONNX path for CPU-only deployments that don’t want the torch runtime.

Quickstart

Install the default backend and model:

local-ai models install speechbrain-ecapa-tdnn

Verify that two audio clips were spoken by the same person:

curl -sX POST http://localhost:8080/v1/voice/verify \
  -H "Content-Type: application/json" \
  -d '{
    "model": "speechbrain-ecapa-tdnn",
    "audio1": "https://example.com/alice_1.wav",
    "audio2": "https://example.com/alice_2.wav"
  }'

Response:

{
  "verified": true,
  "distance": 0.18,
  "threshold": 0.25,
  "confidence": 28.0,
  "model": "speechbrain-ecapa-tdnn",
  "processing_time_ms": 340.0
}

1:N identification workflow (register → identify → forget)

Same flow as face recognition, same in-memory vector store under the hood.

1. Register known speakers:

   curl -sX POST http://localhost:8080/v1/voice/register \
     -H "Content-Type: application/json" \
     -d '{
       "model": "speechbrain-ecapa-tdnn",
       "name": "Alice",
       "audio": "https://example.com/alice.wav"
     }'
   # → {"id": "b2f...", "name": "Alice", "registered_at": "2026-04-22T..."}

2. Identify an unknown probe:

   curl -sX POST http://localhost:8080/v1/voice/identify \
     -H "Content-Type: application/json" \
     -d '{
       "model": "speechbrain-ecapa-tdnn",
       "audio": "https://example.com/unknown.wav",
       "top_k": 5
     }'
   # → {"matches": [{"id":"b2f...","name":"Alice","distance":0.19,"match":true,...}]}

3. Remove a speaker by ID:

   curl -sX POST http://localhost:8080/v1/voice/forget \
     -d '{"id": "b2f..."}'
   # → 204 No Content

API reference

POST /v1/voice/verify (1:1)

Returns verified, distance, threshold, confidence, model, and processing_time_ms.

POST /v1/voice/analyze

Returns demographic attributes (age, gender, emotion) inferred from speech:

Emotion is inferred from the SUPERB emotion-recognition checkpoint (superb/wav2vec2-base-superb-er, Apache 2.0) - 4-way categorical neutral / happy / angry / sad. The model auto-downloads on the first analyze call.

Age and gender are opt-in: no standard-transformers checkpoint with a clean classifier head is shipped as the default. The high-accuracy Audeering age/gender model uses a custom multi-task head that AutoModelForAudioClassification doesn’t load safely (the age weights are silently dropped and the classifier is re-initialised with random values). To enable age/gender, set age_gender_model:<repo> in the model YAML’s options: pointing at a checkpoint with a vanilla Wav2Vec2ForSequenceClassification head. Override the emotion default similarly via emotion_model:. Set either to an empty string to disable that head.

If a head fails to load (offline, disk full, transformers missing), the engine degrades gracefully: it still returns the attributes it could compute. When nothing can be computed the backend returns 501 Unimplemented.

Analyze is supported by both speechbrain-ecapa-tdnn and wespeaker-resnet34 - the speaker recognizer and the analysis head are independent.

POST /v1/voice/register (1:N enrollment)

Returns {id, name, registered_at}. The id is an opaque UUID used by /v1/voice/identify and /v1/voice/forget.

POST /v1/voice/identify (1:N recognition)

Returns a list of matches sorted by ascending distance, each with id, name, labels, distance, confidence, and match (distance ≤ threshold).

POST /v1/voice/forget

Returns 204 No Content on success, 404 Not Found if the ID is unknown.

POST /v1/voice/embed

Returns the L2-normalized speaker embedding vector.

Returns {embedding: float[], dim: int, model: string}. Dimension depends on the recognizer: 192 for ECAPA-TDNN, 256 for WeSpeaker ResNet34.

Note: the OpenAI-compatible /v1/embeddings endpoint is intentionally text-only - it does nothing useful with audio input. Use /v1/voice/embed for audio.

Audio input

Audio is materialised by the HTTP layer to a temporary WAV file before the gRPC call. All audio fields accept:

• http:// / https:// URLs (downloaded server-side, subject to ValidateExternalURL safety checks).
• Raw base64 (no prefix).
• Data URIs (data:audio/wav;base64,...).

The backend itself always receives a filesystem path - the same convention the Whisper / Voxtral transcription backends use.

Threshold reference

Pass threshold explicitly when switching recognizers - the per-model default only applies when omitted.

Related features

• Face Recognition - the image analog; the two share a registry design.
• Audio to Text - transcription (Whisper, Voxtral, faster-whisper). Runs in addition to, not instead of, voice recognition.
• Stores - the generic vector store powering both the face and voice 1:N recognition pipelines.
• Embeddings - text-only OpenAI-compatible embedding endpoint; for audio embeddings use /v1/voice/embed.