Setting Up Models

This section covers everything you need to know about installing and configuring models in LocalAI. You’ll learn multiple methods to get models running.

Prerequisites

• LocalAI installed and running (see Quickstart if you haven’t set it up yet)
• Basic understanding of command line usage

Method 1: Using the Model Gallery (Easiest)

The Model Gallery is the simplest way to install models. It provides pre-configured models ready to use.

GPU recommendations require a memory estimate within 95% of the detected model memory budget at a 4096-token context. If none of the sampled candidates fit, the recommendation section is hidden. You can still browse the gallery and check individual models at your intended context size. On the Models page the recommendations appear as a “Best for this machine” list in the pane beside the table while no model is selected. Once you have installed a model, the list shows only the best fit and offers the others behind “more that fit”. The Home page also omits static GPU suggestions when no fitting recommendation is available.

Via WebUI

1. Open the LocalAI WebUI at http://localhost:8080
2. Navigate to Models → Explore
3. Browse or search the available models
4. Click “Install” on any model you want
5. Wait for installation to complete. Progress appears in the strip at the top of the app, and Operate → Activity shows every install in flight, plus what failed and what finished (see Activity)

For more details, refer to the Gallery Documentation.

The same Models page owns the complete lifecycle. Switch to Installed to search local configurations, filter them by running, idle, disabled, pinned, or distributed state, and open a model’s runtime controls. Load and stop are on each row; edit, pin, disable, backend logs, and remove are in the row menu and in the details of the selected model. The current view, search, filter, and selection are stored in the URL so links and browser history preserve your place. The disk strip in the page header shows the free space on the models disk and opens a review of what can be removed to free more (see Model gallery).

Via CLI

# List available models
local-ai models list

# Install a specific model
local-ai models install llama-3.2-1b-instruct:q4_k_m

# Start LocalAI with a model from the gallery
local-ai run llama-3.2-1b-instruct:q4_k_m

To run models available in the LocalAI gallery, you can use the model name as the URI. For example, to run LocalAI with the Hermes model, execute:

local-ai run hermes-2-theta-llama-3-8b

To install only the model, use:

local-ai models install hermes-2-theta-llama-3-8b

Note: The galleries available in LocalAI can be customized to point to a different URL or a local directory. For more information on how to setup your own gallery, see the Gallery Documentation.

Browse Online

Visit models.localai.io to browse all available models in your browser.

On a phone, the model page wraps build descriptions in the Variants table to keep the fit status and install buttons visible.

Method 1.5: Import Models via WebUI

The WebUI import page (Build → Import) takes either a source to resolve or a configuration to write. Both live on the same page, behind the two tabs in its header. From a source, it is a short guided flow: source, review, import, done.

From a source

1. Open the LocalAI WebUI at http://localhost:8080
2. Open Build, then Import
3. Paste the source into the Source field (e.g. https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-GGUF), or start from one of the examples
4. Review what the page says, then press Enter or click Import

While you type, the page reads the spelling of the source (Hugging Face repository, direct URL, configuration file, OCI image, Ollama model, or a path on the host) and shows the request the form will send. It also runs the checks that can be made before the import starts: whether the name is already in use, whether the chosen backend is installed, and how much disk and memory are free. The server returns no preview of the model configuration, the size or the memory a model needs before the import starts, so the page does not show them. They appear once the import starts, next to the free memory and disk.

The What you can paste panel beside the field lists every accepted scheme: huggingface://, hf://, a full Hugging Face URL, any direct https:// URL, file:// and absolute paths on the host, oci://, ocifile://, and ollama://.

Expand Advanced options directly below the request preview to override settings before importing. The controls include backend, name, description, quantizations, MMProj quantizations, model type, embeddings support, the diffusers-specific fields, and arbitrary custom key-value preferences. The backend list can be narrowed by modality first. Fields that the selected backend cannot use are hidden, and anything you typed into them is kept in case you switch back.

The source format check does not contact the source or validate it as a model. An HTTPS URL can point to a web page rather than model weights.

Leaving the backend on auto-detect lets LocalAI choose from the source. Selecting a backend sends an explicit backend preference in the previewed request. If more than one installed backend can serve the detected modality, the page says so and offers the candidates inline — picking one resubmits the import.

Repositories under mlx-community are imported with the native MLX backend. LocalAI uses Hugging Face’s pipeline metadata to select mlx-vlm for vision-language models and mlx-audio for text-to-speech models; other MLX repositories use mlx. An explicit backend selection in the import form always overrides this automatic routing.

Once the import starts, the page reports the download size and the memory the model needs against what is free, then the current phase, the bytes transferred and a progress bar. When the model is ready, the page names it and links to a chat with it and to Models.

Writing YAML

For full control over model configuration, switch to the Write YAML tab and edit the configuration directly, then click Create. The editor provides syntax highlighting and a copy button, and accepts the same configuration keys documented under Advanced.

This is especially useful for:

• Custom model configurations
• Fine-tuning model parameters
• Setting up complex model setups
• Editing existing model configurations

When the configuration references remote assets, Create does not download them inside the request: the import is queued like a gallery installation, and the page reports the job’s progress until the model is ready. These configurations are queued:

• a URL in parameters.model or mmproj, or download_files entries;
• an hf:// reference or a bare owner/repo in parameters.model, on a backend that consumes a model directory;
• an explicit artifacts: block. The configuration file is written once every download has finished, so a failed download never leaves a config pointing at files that do not exist. A configuration that only references files already on disk is written immediately, as before.

A queued import runs through the same installer as a gallery installation, so it differs from an immediate write in two ways:

• Recommended sampling parameters for the model family (for example temperature and top_p) are added under parameters: when the configuration does not set them. The immediate write stores your configuration exactly as posted.
• When automatic backend installation is enabled, the backend named in the configuration is installed as part of the job.

Set the parameters you care about explicitly if you need the same stored file in both cases.

The same split applies to the API behind the page: POST /models/import answers a queued import with a job id and a status URL to poll (/models/jobs/{id} — the same shape gallery installs use), and a download-free import with a plain success response.

If you cancel a queued import, the import page and model editor report the cancellation and stay on the current page. They report success only after the job succeeds.

Method 2: Installing from Hugging Face

LocalAI can directly install models from Hugging Face:

# Install and run a model from Hugging Face
local-ai run huggingface://TheBloke/phi-2-GGUF

The format is: huggingface://<repository>/<model-file> ( is optional)

Examples

local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf

Method 3: Installing from OCI Registries

Ollama Registry

local-ai run ollama://gemma:2b

Standard OCI Registry

local-ai run oci://localai/phi-2:latest

Run Models via URI

To run models via URI, specify a URI to a model file or a configuration file when starting LocalAI. Valid syntax includes:

• file://path/to/model (absolute path to a file within your models directory)
• huggingface://repository_id/model_file (e.g., huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf)
• From OCIs: oci://container_image:tag, ollama://model_id:tag
• From configuration files: https://gist.githubusercontent.com/.../phi-2.yaml

Configuration files can be used to customize the model defaults and settings. For advanced configurations, refer to the Customize Models section.

Examples

local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
local-ai run ollama://gemma:2b
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest

Method 4: Manual Installation

For full control, you can manually download and configure models.

Step 1: Download a Model

Download a GGUF model file. Popular sources:

• Hugging Face

Example:

mkdir -p models

wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q4_K_M.gguf \
  -O models/phi-2.Q4_K_M.gguf

Step 2: Create a Configuration File (Optional)

Create a YAML file to configure the model:

# models/phi-2.yaml
name: phi-2
parameters:
  model: phi-2.Q4_K_M.gguf
  temperature: 0.7
context_size: 2048
threads: 4
backend: llama-cpp

Customize model defaults and settings with a configuration file. For advanced configurations, refer to the Advanced Documentation.

Step 3: Run LocalAI

Choose one of the following methods to run LocalAI:

mkdir models

cp your-model.gguf models/

docker run -p 8080:8080 -v $PWD/models:/models -ti --rm localai/localai:latest --models-path /models --context-size 700 --threads 4

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "your-model.gguf",
     "prompt": "A long time ago in a galaxy far, far away",
     "temperature": 0.7
   }'

mkdir models

wget https://huggingface.co/TheBloke/Luna-AI-Llama2-Uncensored-GGUF/resolve/main/luna-ai-llama2-uncensored.Q4_0.gguf -O models/luna-ai-llama2

cp -rf prompt-templates/getting_started.tmpl models/luna-ai-llama2.tmpl

docker run -p 8080:8080 -v $PWD/models:/models -ti --rm localai/localai:latest --models-path /models --context-size 700 --threads 4

curl http://localhost:8080/v1/models

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "luna-ai-llama2",
     "messages": [{"role": "user", "content": "How are you?"}],
     "temperature": 0.9
   }'

git clone https://github.com/go-skynet/LocalAI

cd LocalAI

cp your-model.gguf models/

docker compose up -d --pull always

curl http://localhost:8080/v1/models

curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
     "model": "your-model.gguf",
     "prompt": "A long time ago in a galaxy far, far away",
     "temperature": 0.7
   }'

# With binary
local-ai --models-path ./models

GPU Acceleration

For instructions on GPU acceleration, visit the GPU Acceleration page.

For more model configurations, visit the Examples Section.

Understanding Model Files

File Formats

• GGUF: Modern format, recommended for most use cases
• GGML: Older format, still supported but deprecated

Quantization Levels

Models come in different quantization levels (quality vs. size trade-off):

Choosing the Right Model

Consider:

• RAM available: Larger models need more RAM
• Use case: Different models excel at different tasks
• Speed: Smaller quantizations are faster
• Quality: Higher quantizations produce better output

Model Configuration

Basic Configuration

Create a YAML file in your models directory:

name: my-model
parameters:
  model: model.gguf
  temperature: 0.7
  top_p: 0.9
context_size: 2048
threads: 4
backend: llama-cpp

Advanced Configuration

See the Model Configuration guide for all available options.

Managing Models

List Installed Models

Ollama clients can list configured models with GET /api/tags and loaded models with GET /api/ps. Each entry includes size in bytes when LocalAI can resolve a non-empty primary weights file on disk. This is the size of that file, not the total size of a multi-file model or its memory use. Unknown sizes are omitted; /api/ps also omits size_vram because per-model VRAM use is not available.

# Via API
curl http://localhost:8080/v1/models

# Via CLI
local-ai models list

Disk Usage

In the WebUI, the Installed tab of Models shows how much disk each installed configuration uses. The Size column holds the size on disk, and for a model that shares files with others it adds the shared part (“1.2 GB shared”). A file referenced by more than one configuration is counted once in the total under the table, which also gives the shared total. The inspector beside the table names the models a model shares files with, and a model page lists its files under Usage and history, with each file’s size, the other models that use it, and whether it is missing. Removing a model frees only its exclusive files; the files it shares stay while another configuration uses them. A reference whose file is not on disk (a download that never finished, or a file removed by hand) is marked on the model and in the cleanup review.

Only an admin can read the report. For anyone else, or if the read fails, the Size column shows the size of the files the gallery lists, and “size unknown” for a model the gallery does not list.

The report behind these views is available directly (admin only):

curl http://localhost:8080/api/models/storage

It returns per-model sizes with the shared portion split out, the file-to-models relation, and missing references. Files in the models directory that no configuration references are not included.

Remove Models

Simply delete the model file and configuration from your models directory:

rm models/model-name.gguf
rm models/model-name.yaml  # if exists

Troubleshooting

Model Not Loading

1. Check backend: Ensure the required backend is installed

   local-ai backends list
   local-ai backends install llama-cpp  # if needed

2. Check logs: Enable debug mode

   DEBUG=true local-ai

3. Verify file: Ensure the model file is not corrupted

Out of Memory

• Use a smaller quantization (Q4_K_S or Q2_K)
• Reduce context_size in configuration
• Close other applications to free RAM

Wrong Backend

Check the Compatibility Table to ensure you’re using the correct backend for your model.

Best Practices

1. Start small: Begin with smaller models to test your setup
2. Use quantized models: Q4_K_M is a good balance for most use cases
3. Organize models: Keep your models directory organized
4. Backup configurations: Save your YAML configurations
5. Monitor resources: Watch RAM and disk usage