Containers

LocalAI supports Docker, Podman, and other OCI-compatible container engines. This guide covers the common aspects of running LocalAI in containers.

Prerequisites

Before you begin, ensure you have a container engine installed:

• Install Docker (Mac, Windows, Linux)
• Install Podman (Linux, macOS, Windows WSL2)

Podman might not resolve short image names such as localai/localai:latest. The Podman and Compose examples use the docker.io/ prefix to identify the container registry explicitly.

Quick Start

The fastest way to get started is with the CPU image:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest
# Or with Podman:
podman run -p 8080:8080 --name local-ai -ti docker.io/localai/localai:latest

This will:

• Start LocalAI (you’ll need to install models separately)
• Make the API available at http://localhost:8080

Image Types

LocalAI provides several image types to suit different needs. These images work with both Docker and Podman.

Standard Images

Standard images don’t include pre-configured models. Use these if you want to configure models manually.

CPU Image

docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 docker.io/localai/localai:latest

GPU Images

Choose the image that matches your hardware and installed drivers:

• NVIDIA CUDA 12 is the compatibility choice for systems with CUDA 12 drivers. Use CUDA 13 when your NVIDIA driver and toolkit support CUDA 13.
• AMD ROCm is for supported AMD GPUs, while Intel is for Intel GPUs with the required device runtime.
• Jetson uses the L4T ARM64 image. Choose its CUDA 12 image for Jetson AGX Orin-class devices or CUDA 13 for DGX Spark.
• Vulkan works across vendors and is the fallback when no matching CUDA, ROCm, or Intel image is available.

NVIDIA CUDA 13:

docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 --device nvidia.com/gpu=all docker.io/localai/localai:latest-gpu-nvidia-cuda-13

NVIDIA CUDA 12:

docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 --device nvidia.com/gpu=all docker.io/localai/localai:latest-gpu-nvidia-cuda-12

AMD GPU (ROCm):

docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblas
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 --device rocm.com/gpu=all docker.io/localai/localai:latest-gpu-hipblas

Intel GPU:

docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-intel
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 --device gpu.intel.com/all docker.io/localai/localai:latest-gpu-intel

Vulkan:

docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 docker.io/localai/localai:latest-gpu-vulkan

NVIDIA Jetson (L4T ARM64):

CUDA 12 (for Nvidia AGX Orin and similar platforms):

docker run -ti --name local-ai -p 8080:8080 --runtime nvidia --gpus all localai/localai:latest-nvidia-l4t-arm64

CUDA 13 (for Nvidia DGX Spark):

docker run -ti --name local-ai -p 8080:8080 --runtime nvidia --gpus all localai/localai:latest-nvidia-l4t-arm64-cuda-13

Using Compose

The repository’s docker-compose.yaml installs phi-2-chat from the model gallery by default. Change its command list to select a different gallery model.

For a more manageable setup, especially with persistent volumes, use Docker Compose or Podman Compose:

Using CDI (Container Device Interface) - Recommended for NVIDIA Container Toolkit 1.14+

The CDI approach is recommended for newer versions of the NVIDIA Container Toolkit (1.14 and later). It provides better compatibility and is the future-proof method:

version: "3.9"
services:
  api:
    image: docker.io/localai/localai:latest-gpu-nvidia-cuda-12
    # For CUDA 13, use: docker.io/localai/localai:latest-gpu-nvidia-cuda-13
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      # start_period, not timeout, is the knob for a slow first boot: startup
      # preload can download tens of GB before the API binds, and failures
      # inside the start period leave the container `starting` rather than
      # marking it unhealthy. timeout is a per-probe deadline.
      start_period: 60m
      interval: 1m
      timeout: 10s
      retries: 3
    ports:
      - 8080:8080
    environment:
      - DEBUG=false
    volumes:
      - ./models:/models:cached
    # CDI driver configuration (recommended for NVIDIA Container Toolkit 1.14+)
    # This uses the nvidia.com/gpu resource API
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia.com/gpu
              count: all
              capabilities: [gpu]

Save this as compose.yaml and run:

docker compose up -d
# Or with Podman:
podman-compose up -d

Using Legacy NVIDIA Driver - For Older NVIDIA Container Toolkit

If you are using an older version of the NVIDIA Container Toolkit (before 1.14), or need backward compatibility, use the legacy approach:

version: "3.9"
services:
  api:
    image: docker.io/localai/localai:latest-gpu-nvidia-cuda-12
    # For CUDA 13, use: docker.io/localai/localai:latest-gpu-nvidia-cuda-13
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      # start_period, not timeout, is the knob for a slow first boot: startup
      # preload can download tens of GB before the API binds, and failures
      # inside the start period leave the container `starting` rather than
      # marking it unhealthy. timeout is a per-probe deadline.
      start_period: 60m
      interval: 1m
      timeout: 10s
      retries: 3
    ports:
      - 8080:8080
    environment:
      - DEBUG=false
    volumes:
      - ./models:/models:cached
    # Legacy NVIDIA driver configuration (for older NVIDIA Container Toolkit)
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Persistent Storage

The container exposes the following volumes:

The host paths can be anywhere on persistent storage, but the container paths must be exactly /models, /backends, /configuration, and /data. In UnRAID and other container-template UIs, create one path mapping for each row in the table above.

Backend OCI images contain symbolic links. When /backends is stored on a filesystem that cannot create links, such as some CIFS/SMB mounts, LocalAI materializes each link as a regular file so installation can complete. This can use more disk space than a local filesystem. Prefer a Docker or Podman named volume for /backends when possible.

To use bind mounts:

docker run -ti --name local-ai -p 8080:8080 \
  -v $PWD/models:/models \
  -v $PWD/backends:/backends \
  -v $PWD/configuration:/configuration \
  -v $PWD/data:/data \
  localai/localai:latest
# Or with Podman:
podman run -ti --name local-ai -p 8080:8080 \
  -v $PWD/models:/models \
  -v $PWD/backends:/backends \
  -v $PWD/configuration:/configuration \
  -v $PWD/data:/data \
  docker.io/localai/localai:latest

Or use named volumes:

docker volume create localai-models
docker volume create localai-backends
docker volume create localai-configuration
docker volume create localai-data
docker run -ti --name local-ai -p 8080:8080 \
  -v localai-models:/models \
  -v localai-backends:/backends \
  -v localai-configuration:/configuration \
  -v localai-data:/data \
  localai/localai:latest
# Or with Podman:
podman volume create localai-models
podman volume create localai-backends
podman volume create localai-configuration
podman volume create localai-data
podman run -ti --name local-ai -p 8080:8080 \
  -v localai-models:/models \
  -v localai-backends:/backends \
  -v localai-configuration:/configuration \
  -v localai-data:/data \
  docker.io/localai/localai:latest

Container user and existing volumes

The image starts as root only to run entrypoint.sh, which gives the data folders (/models, /backends, /configuration, /data, /tmp/generated) to the user localai (uid and gid 10001) and then runs LocalAI as that user. An add-on that runs commands in an isolated sandbox needs this, because it refuses to run as root. See the sandbox in the LocalAI container for the devices and permissions it needs.

• Volumes from an older image are owned by root. The first start with this image changes their owner to 10001 (it logs this, and it can take a while on a large models folder). No manual step is needed.
• LOCALAI_UID and LOCALAI_GID change the user. docker run --user or a Kubernetes runAsUser is respected, and the data folders must then be writable by that user.
• LOCALAI_RUN_AS_ROOT=true keeps the old behaviour. Add-ons that run commands in a sandbox are then unavailable.
• A bind mount whose owner cannot be changed (a read-only or network mount) must be made writable for uid 10001 on the storage side.

Next Steps

After installation:

1. Access the WebUI at http://localhost:8080
2. Check available models: curl http://localhost:8080/v1/models
3. Install additional models
4. Try out examples

Troubleshooting

Container won’t start

• Check container engine is running: docker ps or podman ps
• Check port 8080 is available: netstat -an | grep 8080 (Linux/Mac)
• View logs: docker logs local-ai or podman logs local-ai

GPU not detected

• Ensure Docker has GPU access: docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi
• For Podman, pass the GPU with the --device flags shown in the GPU sections above (for example --device nvidia.com/gpu=all)
• For NVIDIA: Install NVIDIA Container Toolkit
• For AMD: Ensure devices are accessible: ls -la /dev/kfd /dev/dri

NVIDIA Container fails to start with “Auto-detected mode as ’legacy’” error

If you encounter this error:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: invalid expression

This indicates a Docker/NVIDIA Container Toolkit configuration issue. The container runtime’s prestart hook fails before LocalAI starts. This is not a LocalAI code bug.

Solutions:

1. Use CDI mode (recommended): Update your docker-compose.yaml to use the CDI driver configuration:

   deploy:
     resources:
       reservations:
         devices:
           - driver: nvidia.com/gpu
             count: all
             capabilities: [gpu]

2. Upgrade NVIDIA Container Toolkit: Ensure you have version 1.14 or later, which has better CDI support.

3. Check NVIDIA Container Toolkit configuration: Run nvidia-container-cli --query-gpu to verify your installation is working correctly outside of containers.

4. Verify Docker GPU access: Test with docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi

Models not downloading

• Check internet connection
• Verify disk space: df -h
• Check container logs for errors: docker logs local-ai or podman logs local-ai
• If the logs show status 401 or 403, the source needs authentication. See Private Registries and Galleries

Full image reference

The quick-start examples above use the Docker Hub image names. Every image is published to both Docker Hub and Quay. The tables below map the Docker Hub tag to its Quay equivalent for each variant. Replace v4.12.1 with a released version to pin a specific build.

See Also

• Full image reference - Complete Quay and Docker Hub image matrix
• Install Models - Install and configure models
• GPU Acceleration - GPU setup and optimization
• Kubernetes Installation - Deploy on Kubernetes

Linux Installation

Desktop launcher AppImage (x86_64)

Download LocalAI-Launcher-x86_64.AppImage and its .sha256 file from GitHub Releases. In that directory:

sha256sum --check LocalAI-Launcher-x86_64.AppImage.sha256
chmod +x LocalAI-Launcher-x86_64.AppImage
./LocalAI-Launcher-x86_64.AppImage

You can also double-click the executable image in a file manager. Normal AppImage mounting requires FUSE support (on Ubuntu 22.04, libfuse2). If FUSE is unavailable, extract it into a writable directory and run the launcher:

./LocalAI-Launcher-x86_64.AppImage --appimage-extract
./squashfs-root/AppRun

This is the existing desktop/tray launcher, not the server bundled into an image. First use needs internet access to download LocalAI. Models and backends are downloaded separately. The launcher stores configuration in ~/.localai/launcher.json, runtime downloads in ~/.localai/bin, and default models/backends under ~/.localai/. Fyne preferences use the user configuration directory. These remain writable outside the read-only image. Keep your home writable; replacing or deleting the AppImage does not delete your data.

The supported release target is x86_64 Ubuntu 22.04 or newer, with glibc 2.35+, OpenGL and X11 desktop libraries (XWayland on Wayland). A tray-enabled desktop is recommended. This is not a static executable: other distributions must provide compatible graphics libraries. ARM64 AppImages are not currently published; use the server binaries below instead. Server/backend requirements are separate from launcher requirements. Replace the AppImage manually to update the launcher; there is no AppImage self-updater.

For build dependencies, pinned tools and smoke checks, see the AppImage build guide.

Manual Installation

Download Binary

You can manually download the appropriate binary for your system from the releases page:

1. Go to GitHub Releases
2. Download the binary for your architecture (amd64, arm64, etc.)
3. Make it executable:

chmod +x local-ai-*

4. Run LocalAI:

./local-ai-*

Run your first model

Starting the binary on its own gives you an empty server. To get a working chat right away, run LocalAI with a model name and it will download and serve it from the gallery:

./local-ai-* run qwen3-4b

Once it is ready, open the WebUI at http://localhost:8080 or send a request to the API:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "qwen3-4b",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

System Requirements

Hardware requirements vary based on:

• Model size
• Quantization method
• Backend used

For performance benchmarks with different backends like llama.cpp, visit this link.

Configuration

After installation, you can:

• Access the WebUI at http://localhost:8080
• Configure models in the models directory
• Customize settings via environment variables or config files

Start LocalAI on demand with systemd

LocalAI accepts a single TCP listener passed through the systemd socket activation protocol. This lets systemd listen on the public port and start LocalAI only when the first client connects.

Create /etc/systemd/system/local-ai.socket:

[Unit]
Description=LocalAI API socket

[Socket]
ListenStream=8080
NoDelay=true

[Install]
WantedBy=sockets.target

Create the matching /etc/systemd/system/local-ai.service:

[Unit]
Description=LocalAI

[Service]
Type=simple
User=localai
Group=localai
ExecStart=/usr/local/bin/local-ai run
WorkingDirectory=/var/lib/local-ai

Adjust the user, binary path, working directory, and model configuration for your installation. Then enable the socket, not the service:

sudo systemctl daemon-reload
sudo systemctl enable --now local-ai.socket

The first connection to port 8080 starts local-ai.service; systemd holds that connection until LocalAI is ready to accept it. LOCALAI_ADDRESS and --address are ignored while an inherited listener is present. LocalAI rejects activation with multiple stream listeners so it cannot silently choose the wrong endpoint.

For a Podman-managed container, configure Podman to preserve and pass the systemd socket file descriptor into the container. The LocalAI process inside the container consumes the same activation protocol.

Activation needs both LISTEN_PID and LISTEN_FDS. If only one of them is set, LocalAI ignores them and binds --address as usual. A container engine started from a socket-activated system unit can leak a bare LISTEN_PID into every container it spawns, and that is not an activation attempt.

Next Steps

• Try it out with examples
• Learn about available models
• Configure GPU acceleration
• Customize your configuration

macOS Installation

The easiest way to install LocalAI on macOS is using the DMG application.

Download

Download the latest DMG from GitHub releases:

Download LocalAI.dmg

Installation Steps

1. Download the LocalAI.dmg file from the link above
2. Open the downloaded DMG file
3. Drag the LocalAI application to your Applications folder
4. Launch LocalAI from your Applications folder

First Launch

The app you installed is a small launcher that manages the LocalAI server for you. On the first launch it offers to download and install the latest server release; once that finishes, the server starts automatically and on every following launch of the app.

The launcher lives in the menu bar (look for the LocalAI icon in the top right of your screen) and does not open a window of its own. From the menu bar icon you can start and stop the server, open the WebUI, check for updates, and change settings, including turning off the automatic server start (“Start LocalAI when the launcher opens” under Settings).

Once the server is running, the WebUI is available at http://localhost:8080.

Verification

The LocalAI.dmg (and the app inside it) and the local-ai server binary are signed with an Apple Developer ID and notarized by Apple, so they launch with no quarantine prompt or workaround. To inspect the signature yourself:

spctl --assess --type open --context context:primary-signature -v /Applications/LocalAI.app
codesign --verify --deep --strict --verbose=2 /Applications/LocalAI.app

Next Steps

After installing LocalAI, you can:

• Access the WebUI at http://localhost:8080
• Try it out with examples
• Learn about available models
• Customize your configuration

Run with Kubernetes

Use the Helm chart to run LocalAI on Kubernetes. Add the chart repository first, then install the chart:

helm repo add go-skynet https://go-skynet.github.io/helm-charts/
helm repo update
helm show values go-skynet/local-ai > values.yaml
helm install local-ai go-skynet/local-ai -f values.yaml

Edit values.yaml before you install. The chart is maintained in the helm-charts repository. The chart repository is separate from the LocalAI release. Check its values.yaml for the image tag it uses.

Example manifests

You can also apply a plain deployment file from the examples repository:

kubectl apply -f https://raw.githubusercontent.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment.yaml

For NVIDIA GPUs:

kubectl apply -f https://raw.githubusercontent.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment-nvidia.yaml

Next steps

• For several nodes with a shared database, see Distributed mode.
• For GPU images, see GPU acceleration.
• Before you expose the service, read Authentication.

Running on Nvidia ARM64

LocalAI can be run on Nvidia ARM64 devices, such as the Jetson Nano, Jetson Xavier NX, Jetson AGX Orin, and Nvidia DGX Spark. The following instructions will guide you through building and using the LocalAI container for Nvidia ARM64 devices.

Platform Compatibility

• CUDA 12 L4T images: Compatible with Nvidia AGX Orin and similar platforms (Jetson Nano, Jetson Xavier NX, Jetson AGX Xavier)
• CUDA 13 L4T images: Compatible with Nvidia DGX Spark

Prerequisites

• Docker engine installed (https://docs.docker.com/engine/install/ubuntu/)
• Nvidia container toolkit installed (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-ap)

Pre-built Images

Pre-built images are available on quay.io and dockerhub:

CUDA 12 (for AGX Orin and similar platforms)

docker pull quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64
# or
docker pull localai/localai:latest-nvidia-l4t-arm64

CUDA 13 (for DGX Spark)

docker pull quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64-cuda-13
# or
docker pull localai/localai:latest-nvidia-l4t-arm64-cuda-13

Build the container

If you need to build the container yourself, use the following commands:

CUDA 12 (for AGX Orin and similar platforms)

git clone https://github.com/mudler/LocalAI

cd LocalAI

docker build --build-arg SKIP_DRIVERS=true --build-arg BUILD_TYPE=cublas --build-arg BASE_IMAGE=nvcr.io/nvidia/l4t-jetpack:r36.4.0 --build-arg IMAGE_TYPE=core -t quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-core .

CUDA 13 (for DGX Spark)

git clone https://github.com/mudler/LocalAI

cd LocalAI

docker build --build-arg SKIP_DRIVERS=false --build-arg BUILD_TYPE=cublas --build-arg CUDA_MAJOR_VERSION=13 --build-arg CUDA_MINOR_VERSION=0 --build-arg BASE_IMAGE=ubuntu:24.04 --build-arg IMAGE_TYPE=core -t quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-cuda-13-core .

Usage

Run the LocalAI container on Nvidia ARM64 devices using the following commands, where /data/models is the directory containing the models:

CUDA 12 (for AGX Orin and similar platforms)

docker run -e DEBUG=true -p 8080:8080 -v /data/models:/models -ti --restart=always --name local-ai --runtime nvidia --gpus all quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64

CUDA 13 (for DGX Spark)

docker run -e DEBUG=true -p 8080:8080 -v /data/models:/models -ti --restart=always --name local-ai --runtime nvidia --gpus all quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64-cuda-13

Note: /data/models is the directory containing the models. You can replace it with the directory containing your models.

GPU reporting in distributed mode

If you run a worker on a Jetson, DGX Spark (GB10), or Thor and the Nodes page in the frontend shows the node as fully used, check two things:

1. NVIDIA_DRIVER_CAPABILITIES must include utility so nvidia-smi / NVML work inside the container. With --gpus all alone (or --runtime nvidia without extra flags) only compute is wired in on some driver versions. Add -e NVIDIA_DRIVER_CAPABILITIES=compute,utility to your docker run. For Docker Compose with driver: nvidia, use capabilities: [gpu, compute, utility] on the device reservation. Include compute for CUDA libraries such as libcuda.so.1; utility alone only provides monitoring libraries and tools.
2. Pass --init to docker run (or init: true in compose) so the container has a proper PID 1 reaper - otherwise short-lived child processes like nvidia-smi can intermittently fail with waitid: no child processes.

On unified-memory devices LocalAI auto-detects the SoC via /sys/devices/soc0/{family,soc_id} and reports system RAM as VRAM, so nvidia-smi is not strictly required for VRAM metrics. See Distributed Mode → NVIDIA GPU support for full context.