Engines we build
Eighteen engines,written from scratch.
Most backends wrap somebody else's engine. These do not. Each one exists because the thing we needed was a multi-gigabyte Python install, or closed, or nobody had built it yet. What you get instead is a binary and a GGUF file, checked against the reference implementation in CI, running on the machine you already own.
Hearing
Turning sound into something a model can act on, words first and then everything else in the room.
C++17
parakeet.cpp
Transcribe a meeting on a laptop CPU and be finished before whisper.cpp has cleared the first minute.
- About 27x faster than whisper.cpp turbo on CPU, and about 12x on GPU
- WER 0 against NVIDIA NeMo on every published checkpoint, so the transcript is identical
- Cache-aware streaming with end-of-utterance detection, for live audio
- Ten checkpoints, from 110M to 1.1B, and 40 or more locales on the streaming multilingual model
C++17
moss-transcribe.cpp
Get the transcript, the speaker labels and the timestamps out of a single pass, then export straight to srt or json.
- 1.58x to 1.78x faster than PyTorch on CPU, on about 1.5x less memory
- Byte-identical transcript against the reference, cosine 1.0 component by component
- 3.4 GB at f32 down to 511 MB at q4_k, still byte-identical through q5_k
C++17
ced.cpp
Let the model hear a smoke alarm, a dog or breaking glass, not only the words somebody typed.
- 527 AudioSet sound classes, multi-label, tagged in about 55 ms
- 6 MB on disk at ced-tiny q8_0, 111 MB at ced-base
- About 1.25x faster than PyTorch at f32 on half the memory
- Works over REST and live over the realtime websocket
C++
LocalVQE
Keep a voice loop usable in a real room, with the echo, the noise and the reverb removed before the model ever hears it.
- Echo cancellation, noise suppression and dereverberation in one pass
- Streaming and causal, 16 ms latency, 5x realtime on a desktop CPU
- From a 17 KB linear filter to a 19 MB joint model, pick what your CPU can afford
- About 21x realtime on a single Raspberry Pi 5 core
Voice
Speech coming back out, in a voice you chose or one you cloned from a few seconds of audio.
C++17
moss-tts.cpp
Clone a voice from a short reference clip and have it read anything back at 48 kHz stereo.
- About 1.9x faster per frame than PyTorch on CPU, both at fp32
- Codec decode matches the reference at 114.9 dB SNR
- 48 kHz stereo out of MOSS-TTS-Local v1.5, 12 codebooks
- Flat C API as well as a CLI, so it embeds anywhere
C++17
magpie-tts.cpp
Ship multilingual speech from one GGUF that already carries the codec, the tokenizer and the pronunciation dictionaries.
- 63x faster than the NeMo reference at f32, 73x at q8_0
- 5 named voices, 9 languages plus 3 Arabic variants
- 541 MB at q4_k, everything bundled in the single file
- Deterministic from a seed, and parity holds to 3.6e-5 on the full decode
C++
vibevoice.cpp
Read a multi-speaker script in cloned voices, and transcribe long recordings back with speaker labels, from the same binary.
- Voice cloning from roughly 5 seconds of reference audio
- Long-form ASR with diarization on the same engine as the synthesis
- 11 GB down to 6.8 GB at Q8_0 with no measurable recall loss
- Realtime 0.5B, 1.5B and a 7B ASR model
CExperimental
voxtral-tts.c
Run a 4B speech model with nothing but a C compiler and libm, reading the weights straight off the safetensors file.
- Pure C, no dependency beyond the C standard library and math
- BF16 weights read from mmap, no conversion step
- 20 preset voices across 9 languages, 24 kHz output
- Optional BLAS, Apple Accelerate, NEON and CUDA paths
Identity
Working out who is in front of the microphone or the camera, and whether they are really there.
C++17
voice-detect.cpp
Tell who is speaking, and read their age, gender and mood, without an onnxruntime install anywhere near it.
- Embedding cosine 0.9999 or better against the reference, often exactly 1.0
- 5.4x lower peak memory than the Python path, 62 MB against 334 MB
- Six model families, ECAPA-TDNN and WeSpeaker through ERes2Net and CAM++
- Verification, identification against a registry, plus age, gender and emotion
C++17
face-detect.cpp
Detect, recognise and verify a face, and catch a photo held up to the camera, all from one shared library.
- Boxes and landmarks land within 1 pixel of insightface
- Recognition embedding cosine 1.000000 against the reference
- Detect, align, recognise, demographics and anti-spoofing in one pipeline
- The yunet-sface pack is Apache-2.0, so it is usable commercially
Vision
Finding things in an image and naming them, including things nobody trained a class for.
C++17
locate-anything.cpp
Ask for the red mug on the left in plain words and get coordinates back, not a caption.
- 1.66x to 3.09x faster than the official PyTorch on CPU
- Identical detections, IoU 1.000 against the reference
- At q8_0 it is about 4.8x faster than PyTorch f32 and still box-identical
- 9.2 GB at f16, 4.7 GB at q4_k, and an annotated PNG out of the box
C++17
rf-detr.cpp
Get boxes and instance masks at COCO quality out of any image, with no PyTorch anywhere in the process.
- 11 variants, 5 detection and 6 segmentation, from Nano to 2XLarge
- About 9% faster than PyTorch on CPU at F16, and 1.86x smaller
- Mean mask IoU 0.99 against PyTorch on the small segmentation variants
- 44 published GGUFs, F32 through Q4_K
Space
Reading distance, camera pose and shape out of ordinary photos, with no rig and no capture setup.
C++17
depth-anything.cpp
Turn one ordinary photo into distance in metres, a camera pose and a point cloud you can open in a 3D viewer.
- 1.31x faster than PyTorch on CPU at q8_0, in half the memory
- Loads about 6.7x faster, 112 ms against 749 ms
- 99 MB at q4_k, and correlation 1.0 with the reference component by component
- Exports to glb, COLMAP and PLY, plus confidence and a sky mask
C++
free-splatter.cpp
Turn a handful of snapshots into a 3D Gaussian scene with no camera poses, no rig and no GPU.
- 0.22 s per forward pass on Vulkan, against 1.37 s for the PyTorch reference on CUDA
- 14 s on 12 CPU threads, roughly 4x the reference, with no GPU at all
- Pose-free, so ordinary photos are enough
- One 3D Gaussian per pixel, ready for any splat viewer
C++
trellis2.cpp
Drop in one image and get back a watertight textured mesh you can hand straight to a 3D tool.
- Single image to GLB with PBR materials, all inference in C++
- Prebuilt f16 GGUFs, so no safetensors download and no conversion
- Flat C ABI plus a Go demo server with a browser mesh viewer
- Metal on by default on Apple, CUDA and CPU elsewhere
Text
Serving language models, and cleaning what goes into them before it leaves the machine.
C++20In development
vllm.cpp
Serve a language model with vLLM's throughput from a 66 MiB binary instead of a 9 GB virtualenv.
- 66 MiB to install, against 9.1 GiB for a vLLM environment
- Continuous batching, paged KV cache, prefix caching and speculative decoding
- 25 or more architectures, gated token for token against a pinned vLLM oracle
- CPU, CUDA, Metal and Vulkan from the same source
C++
privacy-filter.cpp
Catch names, addresses and card numbers on the machine, before any of it reaches a model or a log.
- 7.7x faster than HF Transformers on an 8k token document, on CPU
- Runs flat to 131k tokens on GPU where HF runs out of memory at about 16k
- Exact UTF-8 byte offsets for every span it finds
- 360 tokens a second on a Raspberry Pi 5, on-device
Data
The storage and quantization work that decides what actually fits on your hardware.
Go
local-store
Get vector search inside LocalAI with nothing to deploy, nothing to configure and no second service to run.
- Ships in-tree and is the default, so embeddings work on a fresh install
- Exact cosine similarity, zero configuration
- Backs RAG, the face and voice registries and the semantic router cache
- Swap in valkey-store per request when you need durability
Shell
apex-quant
Fit a 35B mixture-of-experts model on a card you already own, and watch it run faster than the full-size build.
- 64.6 GB down to 12.2 GB, at 74.4 tokens a second against 30.4
- APEX Quality beats F16 perplexity at a third of the size
- Ordinary GGUF files, so stock llama.cpp opens them unpatched
- 201 builds already sitting in the LocalAI gallery
Nothing in that modality yet.
The rule we hold them to
A port only ships once it matches the original.
Every engine here is gated against the framework it replaces, on the same input, on the same machine. That means a transcript that comes out word for word identical, boxes that land on the same pixels, or a waveform inside a stated tolerance. Speed is the part we then go and win, and the numbers on this page come out of each engine's own benchmark suite, not a marketing run.