Features / Every modality
Text, images, video, music and voice.
Send text, image, audio and video requests to one API in the OpenAI and Anthropic formats.
- Text and visionchat, tools
- Images and videogenerate
- Music and soundgenerate
- Speechsay, transcribe
- Searchembed, rerank
What it does
- Text and vision: chat, tool calls, image understanding and object detection.
- Images and video: image generation and video generation, also 3D.
- Animated video (development build): With the Sandbox add-on and its media profile, the assistant can build short videos with ffmpeg from a sentence: a title card with a fade, or a slideshow with a slow zoom and crossfade, saved as MP4 in your Files. The Sandbox has no internet. Results need a look before you use them. A landscape video can be cropped to vertical. Simple animated charts work with mixed results.
- Music and sound: make music and sound effects from a text prompt. See text to audio.
- Speech: text to speech, transcription, voice activity detection and speaker recognition. The Realtime API takes voice in and gives voice back.
- Search: embeddings and a reranker.
- Use the OpenAI API with the clients you already have. LocalAI also answers in the Anthropic format, and text to speech and sound generation accept the ElevenLabs format.
- The engine runs on a CPU, so it does not need a GPU. NVIDIA, AMD, Intel and Apple GPUs make it faster, and large image and video models need one to be practical.
- Distributed mode spreads models over several machines.
Good to know
- Animated video needs the Sandbox add-on, which an admin installs from the development catalog today, and a model that can call tools. A chat uses the media profile unless an admin limits the choice. The first use pulls the media image.
- Each kind of model needs its own backend. LocalAI downloads it the first time a model needs it.
- The compatibility table lists the model families that run.