LocalAI provides a variety of images to support different environments. These images are available on quay.io and Docker Hub.

All-in-One images comes with a pre-configured set of models and backends, standard images instead do not have any model pre-configured and installed.

For GPU Acceleration support for Nvidia video graphic cards, use the Nvidia/CUDA images, if you don’t have a GPU, use the CPU images. If you have AMD or Mac Silicon, see the build section.

Prerequisites

Before you begin, ensure you have a container engine installed if you are not using the binaries. Suitable options include Docker or Podman. For installation instructions, refer to the following guides:

All-in-one images

All-In-One images are images that come pre-configured with a set of models and backends to fully leverage almost all the LocalAI featureset. These images are available for both CPU and GPU environments. The AIO images are designed to be easy to use and requires no configuration. Models configuration can be found here separated by size.

In the AIO images there are models configured with the names of OpenAI models, however, they are really backed by Open Source models. You can find the table below

CategoryModel nameReal model (CPU)Real model (GPU)
Text Generationgpt-4phi-2hermes-2-pro-mistral
Multimodal Visiongpt-4-vision-previewbakllavallava-1.6-mistral
Image Generationstablediffusionstablediffusiondreamshaper-8
Speech to Textwhisper-1whisper with whisper-base model<= same
Text to Speechtts-1en-us-amy-low.onnx from rhasspy/piper<= same
Embeddingstext-embedding-ada-002all-MiniLM-L6-v2 in Q4all-MiniLM-L6-v2

Usage

Select the image (CPU or GPU) and start the container with Docker:

  # CPU example
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest-aio-cpu
# For Nvidia GPUs:
# docker run -p 8080:8080 --gpus all --name local-ai -ti localai/localai:latest-aio-gpu-nvidia-cuda-11
# docker run -p 8080:8080 --gpus all --name local-ai -ti localai/localai:latest-aio-gpu-nvidia-cuda-12
  

LocalAI will automatically download all the required models, and the API will be available at localhost:8080.

Or with a docker-compose file:

  version: "3.9"
services:
  api:
    image: localai/localai:latest-aio-cpu
    # For a specific version:
    # image: localai/localai:v2.21.1-aio-cpu
    # For Nvidia GPUs decomment one of the following (cuda11 or cuda12):
    # image: localai/localai:v2.21.1-aio-gpu-nvidia-cuda-11
    # image: localai/localai:v2.21.1-aio-gpu-nvidia-cuda-12
    # image: localai/localai:latest-aio-gpu-nvidia-cuda-11
    # image: localai/localai:latest-aio-gpu-nvidia-cuda-12
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      interval: 1m
      timeout: 20m
      retries: 5
    ports:
      - 8080:8080
    environment:
      - DEBUG=true
      # ...
    volumes:
      - ./models:/build/models:cached
    # decomment the following piece if running with Nvidia GPUs
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: 1
    #           capabilities: [gpu]
  

Available AIO images

DescriptionQuayDocker Hub
Latest images for CPUquay.io/go-skynet/local-ai:latest-aio-cpulocalai/localai:latest-aio-cpu
Versioned image (e.g. for CPU)quay.io/go-skynet/local-ai:v2.21.1-aio-cpulocalai/localai:v2.21.1-aio-cpu
Latest images for Nvidia GPU (CUDA11)quay.io/go-skynet/local-ai:latest-aio-gpu-nvidia-cuda-11localai/localai:latest-aio-gpu-nvidia-cuda-11
Latest images for Nvidia GPU (CUDA12)quay.io/go-skynet/local-ai:latest-aio-gpu-nvidia-cuda-12localai/localai:latest-aio-gpu-nvidia-cuda-12
Latest images for AMD GPUquay.io/go-skynet/local-ai:latest-aio-gpu-hipblaslocalai/localai:latest-aio-gpu-hipblas
Latest images for Intel GPU (sycl f16)quay.io/go-skynet/local-ai:latest-aio-gpu-intel-f16localai/localai:latest-aio-gpu-intel-f16
Latest images for Intel GPU (sycl f32)quay.io/go-skynet/local-ai:latest-aio-gpu-intel-f32localai/localai:latest-aio-gpu-intel-f32

Available environment variables

The AIO Images are inheriting the same environment variables as the base images and the environment of LocalAI (that you can inspect by calling --help). However, it supports additional environment variables available only from the container image

VariableDefaultDescription
PROFILEAuto-detectedThe size of the model to use. Available: cpu, gpu-8g
MODELSAuto-detectedA list of models YAML Configuration file URI/URL (see also running models)

Standard container images

Standard container images do not have pre-installed models.

Images are available with and without python dependencies. Note that images with python dependencies are bigger (in order of 17GB).

Images with core in the tag are smaller and do not contain any python dependencies.

See Also

Last updated 20 Sep 2024, 18:16 +0200 . history