Voice Recognition

LocalAI supports voice (speaker) recognition through the speaker-recognition backend: speaker verification (1:1), speaker identification (1:N) against a built-in vector store, speaker embedding, and demographic analysis (age / gender / emotion from voice).

The audio analog to Face Recognition, following the same two-engine pattern under one image.

Engines

Both entries are commercial-safe Apache-2.0. SpeechBrain is the default — it’s a lightweight pure-PyTorch checkpoint that auto- downloads on first use. The wespeaker-resnet34 entry wires the direct-ONNX path for CPU-only deployments that don’t want the torch runtime.

Quickstart

Install the default backend and model:

local-ai models install speechbrain-ecapa-tdnn

Verify that two audio clips were spoken by the same person:

curl -sX POST http://localhost:8080/v1/voice/verify \
  -H "Content-Type: application/json" \
  -d '{
    "model": "speechbrain-ecapa-tdnn",
    "audio1": "https://example.com/alice_1.wav",
    "audio2": "https://example.com/alice_2.wav"
  }'

Response:

{
  "verified": true,
  "distance": 0.18,
  "threshold": 0.25,
  "confidence": 28.0,
  "model": "speechbrain-ecapa-tdnn",
  "processing_time_ms": 340.0
}

1:N identification workflow (register → identify → forget)

Same flow as face recognition, same in-memory vector store under the hood.

1. Register known speakers:

   curl -sX POST http://localhost:8080/v1/voice/register \
     -H "Content-Type: application/json" \
     -d '{
       "model": "speechbrain-ecapa-tdnn",
       "name": "Alice",
       "audio": "https://example.com/alice.wav"
     }'
   # → {"id": "b2f...", "name": "Alice", "registered_at": "2026-04-22T..."}

2. Identify an unknown probe:

   curl -sX POST http://localhost:8080/v1/voice/identify \
     -H "Content-Type: application/json" \
     -d '{
       "model": "speechbrain-ecapa-tdnn",
       "audio": "https://example.com/unknown.wav",
       "top_k": 5
     }'
   # → {"matches": [{"id":"b2f...","name":"Alice","distance":0.19,"match":true,...}]}

3. Remove a speaker by ID:

   curl -sX POST http://localhost:8080/v1/voice/forget \
     -d '{"id": "b2f..."}'
   # → 204 No Content

API reference

POST /v1/voice/verify (1:1)

Returns verified, distance, threshold, confidence, model, and processing_time_ms.

POST /v1/voice/analyze

Returns demographic attributes (age, gender, emotion) inferred from speech:

Emotion is inferred from the SUPERB emotion-recognition checkpoint (superb/wav2vec2-base-superb-er, Apache 2.0) — 4-way categorical neutral / happy / angry / sad. The model auto-downloads on the first analyze call.

Age and gender are opt-in: no standard-transformers checkpoint with a clean classifier head is shipped as the default. The high-accuracy Audeering age/gender model uses a custom multi-task head that AutoModelForAudioClassification doesn’t load safely (the age weights are silently dropped and the classifier is re-initialised with random values). To enable age/gender, set age_gender_model:<repo> in the model YAML’s options: pointing at a checkpoint with a vanilla Wav2Vec2ForSequenceClassification head. Override the emotion default similarly via emotion_model:. Set either to an empty string to disable that head.

If a head fails to load (offline, disk full, transformers missing), the engine degrades gracefully: it still returns the attributes it could compute. When nothing can be computed the backend returns 501 Unimplemented.

Analyze is supported by both speechbrain-ecapa-tdnn and wespeaker-resnet34 — the speaker recognizer and the analysis head are independent.

POST /v1/voice/register (1:N enrollment)

Returns {id, name, registered_at}. The id is an opaque UUID used by /v1/voice/identify and /v1/voice/forget.

POST /v1/voice/identify (1:N recognition)

Returns a list of matches sorted by ascending distance, each with id, name, labels, distance, confidence, and match (distance ≤ threshold).

POST /v1/voice/forget

Returns 204 No Content on success, 404 Not Found if the ID is unknown.

POST /v1/voice/embed

Returns the L2-normalized speaker embedding vector.

Returns {embedding: float[], dim: int, model: string}. Dimension depends on the recognizer: 192 for ECAPA-TDNN, 256 for WeSpeaker ResNet34.

Note: the OpenAI-compatible /v1/embeddings endpoint is intentionally text-only — it does nothing useful with audio input. Use /v1/voice/embed for audio.

Audio input

Audio is materialised by the HTTP layer to a temporary WAV file before the gRPC call. All audio fields accept:

• http:// / https:// URLs (downloaded server-side, subject to ValidateExternalURL safety checks).
• Raw base64 (no prefix).
• Data URIs (data:audio/wav;base64,...).

The backend itself always receives a filesystem path — the same convention the Whisper / Voxtral transcription backends use.

Threshold reference

Pass threshold explicitly when switching recognizers — the per-model default only applies when omitted.

Related features

• Face Recognition — the image analog; the two share a registry design.
• Audio to Text — transcription (Whisper, Voxtral, faster-whisper). Runs in addition to, not instead of, voice recognition.
• Stores — the generic vector store powering both the face and voice 1:N recognition pipelines.
• Embeddings — text-only OpenAI-compatible embedding endpoint; for audio embeddings use /v1/voice/embed.