Images, video and 3D
Image understanding, detection, faces, image and video generation, 3D and motion. The Studio add-on makes and keeps images, video and 3D in one page, and the Face add-on is a page for face recognition.
LocalAI supports understanding images by using LLaVA, and implements the GPT Vision API from OpenAI. Usage OpenAI docs: https://platform.openai.com/docs/guides/vision First install a vision-capable model from the gallery (the examples below use moondream2-20250414, a small vision model): local-ai run moondream2-20250414 To let LocalAI understand and reply with what sees in the image, use the /v1/chat/completions endpoint, for example with curl:
LocalAI supports object detection and image segmentation through various backends. This feature allows you to identify and locate objects within images with high accuracy and real-time performance. Available backends include RF-DETR (Python) and rf-detr.cpp (native C++/ggml) for object detection and segmentation, and sam3.cpp for image segmentation (SAM 3/2/EdgeTAM). For detecting faces specifically, see the dedicated Face Recognition feature - its /v1/detection support is tuned for face bounding boxes and ships with commercially-safe model options.
LocalAI supports face recognition: face verification (1:1), face identification (1:N) against a built-in vector store, face embedding, face detection, demographic analysis (age / gender), and antispoofing / liveness detection. The same /v1/face/* HTTP API is served by two backends:
Generate and edit images with Stable Diffusion and diffusers models through the images API.
LocalAI can generate videos from text prompts and optional image or audio conditioning via the /video endpoint. Supported backends include diffusers, stablediffusion (including MiniMax-H3, which generates video and audio together), vllm-omni, mlx-video on Apple Silicon, and the dedicated longcat-video backend. MLX-Video on Apple Silicon Install the native Metal backend, then configure an LTX-2 repository:
LocalAI can generate textured 3D meshes from a single conditioning image via the /3d/generations endpoint, powered by the trellis2cpp backend — a C++/GGML port of Microsoft TRELLIS.2 (trellis2.cpp). The output is a binary glTF (.glb) asset with PBR materials. Generation is image-conditioned only — there is no text-prompt path. Provide a photo or rendering of a single object (ideally on a plain background) and TRELLIS.2 reconstructs a full 3D mesh from it.
LocalAI supports text-to-motion with kimodo.cpp on CPU and Vulkan. In Studio → 3D, selecting an animation model replaces the image input with a motion prompt and the selected model’s animation controls. The Studio preview plays at real-time speed using the GLB’s timestamps. Slow or delayed display frames skip ahead rather than slowing the animation. Pause and the timeline let you inspect individual poses; playback resumes from that point. Kimodo exports an animated skeleton GLB: joint rotations and root movement, without a mesh or skin. SOMA models produce the compact 30-joint skeleton; G1 models produce 34 joints. These outputs are animation assets, not textured characters, and cannot use TRELLIS print remeshing.
LocalAI’s gemxcpp backend converts timestamped RGB frames into human poses using gem-x.cpp. It runs resident GEM-X, ViTPose and YOLOX models on CPU or Vulkan. Live inference does not need SAM3D Body. This initial integration is API-only: no Motion UI, offline clip export, body mesh or SONIC controller is included. Install and configure Import https://huggingface.co/LocalAI-io/GEM-X-GGUF through the model importer. It selects gemxcpp and downloads three checksummed, revision-pinned assets. These are custom F32 GGUF architectures, not language models. Weight licenses are described in the source model card. Allow memory for all three models and execution buffers, beyond their roughly 4 GB combined download size.