Getting started :: LocalAI

Install LocalAI

Mon, 01 Jan 0001 00:00:00 +0000

How to install LocalAI

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

LocalAI is a free, open-source alternative to OpenAI (Anthropic, etc.), functioning as a drop-in replacement REST API for local inferencing. It allows you to run LLMs, generate images, and produce audio, all locally or on-premises with consumer-grade hardware, supporting multiple model families and architectures.

Setting Up Models

Mon, 01 Jan 0001 00:00:00 +0000

Learn how to install, configure, and manage models in LocalAI

Try it out

Mon, 01 Jan 0001 00:00:00 +0000

Once LocalAI is installed, you can start it (either by using docker, or the cli, or the systemd service). By default the LocalAI WebUI should be accessible from http://localhost:8080. You can also use 3rd party projects to interact with LocalAI as you would use OpenAI (see also Integrations ). After installation, install new models by navigating the model gallery, or by using the local-ai CLI. Tip To install models with the WebUI, see the Models section. With the CLI you can list the models with local-ai models list and install them with local-ai models install .

Customizing the Model

Mon, 01 Jan 0001 00:00:00 +0000

To customize the prompt template or the default settings of the model, a configuration file is utilized. This file must adhere to the LocalAI YAML configuration standards. For comprehensive syntax details, refer to the advanced documentation. The configuration file can be located either remotely (such as in a Github Gist) or within the local filesystem or a remote URL. LocalAI can be initiated using either its container image or binary, with a command that includes URLs of model config files or utilizes a shorthand format (like huggingface:// or github://), which is then expanded into complete URLs.

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers common issues you may encounter when using LocalAI, organized by category. For each issue, diagnostic steps and solutions are provided. Quick Diagnostics Before diving into specific issues, run these commands to gather diagnostic information: # Check LocalAI is running and responsive curl http://localhost:8080/readyz # List loaded models curl http://localhost:8080/v1/models # Check LocalAI version local-ai --version # Enable debug logging for detailed output DEBUG=true local-ai run # or local-ai run --log-level=debug For Docker deployments:

Build your first agent

Mon, 01 Jan 0001 00:00:00 +0000

LocalAGI is embedded in LocalAI. There is nothing separate to install or run. The agent platform ships inside the LocalAI binary and container image, and it is enabled by default. If you already have LocalAI running and a model installed, you have everything you need to build an agent. This page walks you from an empty Agents page to an agent that answers a message and uses one tool. Before you start: install a tool-calling model An agent is a loop around a chat model, so it needs a model that supports tool (function) calling. This guide uses qwen3-4b, a small CPU-friendly Qwen3 model that supports tool calling. It is the same model used in the Quickstart, so if you followed that page you already have it.

Containers

Mon, 01 Jan 0001 00:00:00 +0000

Install and use LocalAI with container engines (Docker, Podman)

Linux Installation

Mon, 01 Jan 0001 00:00:00 +0000

Install LocalAI on Linux using binaries

macOS Installation

Mon, 01 Jan 0001 00:00:00 +0000

Install LocalAI on macOS using the DMG application

Run with Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

For installing LocalAI in Kubernetes, the deployment file from the examples can be used and customized as preferred: kubectl apply -f https://raw.githubusercontent.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment.yaml For Nvidia GPUs: kubectl apply -f https://raw.githubusercontent.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment-nvidia.yaml Alternatively, the helm chart can be used as well: helm repo add go-skynet https://go-skynet.github.io/helm-charts/ helm repo update helm show values go-skynet/local-ai > values.yaml helm install local-ai go-skynet/local-ai -f values.yaml

Build LocalAI

Mon, 01 Jan 0001 00:00:00 +0000

Build LocalAI can be built as a container image or as a single, portable binary. Note that some model architectures might require Python libraries, which are not included in the binary. LocalAI’s extensible architecture allows you to add your own backends, which can be written in any language, and as such the container images contains also the Python dependencies to run all the available backends (for example, in order to run backends like Diffusers that allows to generate images and videos from text).

Docker Installation

Mon, 01 Jan 0001 00:00:00 +0000

Install LocalAI using Docker containers - the recommended installation method