Features / Every modality

Text, images, video, music and voice.

Send text, image, audio and video requests to one API in the OpenAI and Anthropic formats.

What it does

  • Text and vision: chat, tool calls, image understanding and object detection.
  • Images and video: image generation and video generation, also 3D.
  • Animated video (development build): With the Sandbox add-on and its media profile, the assistant can build short videos with ffmpeg from a sentence: a title card with a fade, or a slideshow with a slow zoom and crossfade, saved as MP4 in your Files. The Sandbox has no internet. Results need a look before you use them. A landscape video can be cropped to vertical. Simple animated charts work with mixed results.
  • Music and sound: make music and sound effects from a text prompt. See text to audio.
  • Speech: text to speech, transcription, voice activity detection and speaker recognition. The Realtime API takes voice in and gives voice back.
  • Search: embeddings and a reranker.
  • Use the OpenAI API with the clients you already have. LocalAI also answers in the Anthropic format, and text to speech and sound generation accept the ElevenLabs format.
  • The engine runs on a CPU, so it does not need a GPU. NVIDIA, AMD, Intel and Apple GPUs make it faster, and large image and video models need one to be practical.
  • Distributed mode spreads models over several machines.
Good to know
  • Animated video needs the Sandbox add-on, which an admin installs from the development catalog today, and a model that can call tools. A chat uses the media profile unless an admin limits the choice. The first use pulls the media image.
  • Each kind of model needs its own backend. LocalAI downloads it the first time a model needs it.
  • The compatibility table lists the model families that run.