Distributed Mode

Distributed mode enables horizontal scaling of LocalAI across multiple machines using PostgreSQL for state and node registry, and NATS for real-time coordination. Unlike the P2P/federation approach, distributed mode is designed for production deployments and Kubernetes environments where you need centralized management, health monitoring, and deterministic routing.

Architecture Overview

                    ┌─────────────────┐
                    │   Load Balancer  │
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
      ┌───────▼──────┐ ┌────▼─────┐ ┌─────▼──────┐
      │  Frontend #1 │ │ Frontend │ │ Frontend #N│
      │  (LocalAI)   │ │  #2      │ │  (LocalAI) │
      └──────┬───────┘ └────┬─────┘ └─────┬──────┘
             │              │              │
     ┌───────▼──────────────▼──────────────▼───────┐
     │              PostgreSQL + NATS               │
     │  (node registry, jobs, coordination)         │
     └───────┬──────────────┬──────────────┬───────┘
             │              │              │
      ┌──────▼──────┐ ┌────▼─────┐ ┌─────▼──────┐
      │  Worker #1  │ │ Worker   │ │ Worker #N  │
      │  (generic)  │ │ #2       │ │  (generic) │
      └─────────────┘ └──────────┘ └────────────┘

Frontends are stateless LocalAI instances that receive API requests and route them to worker nodes via the SmartRouter. All frontends share state through PostgreSQL and coordinate via NATS.

Workers are generic processes that self-register with a frontend. They don’t have a fixed backend type — the SmartRouter dynamically installs the required backend via NATS backend.install events when a model request arrives.

Scheduling Algorithm

The SmartRouter uses idle-first scheduling with preemptive eviction:

1. If the model is already loaded on a node → use it (per-model gRPC address)
2. If no node has the model → prefer nodes with enough free VRAM
3. Fall back to idle nodes (zero models), then least-loaded nodes
4. If no node has capacity → evict the least-recently-used model with zero in-flight requests to free a node
5. If all models are busy → wait (with timeout) for a model to become idle, then evict
6. Send backend.install NATS event with backend name + model ID → worker starts a new gRPC process on a dynamic port
7. SmartRouter calls gRPC LoadModel on the model-specific port, records in DB

Each model gets its own gRPC backend process, so a single worker can serve multiple models simultaneously (e.g., a chat model and an embedding model).

Prerequisites

• PostgreSQL (with pgvector extension recommended for RAG) — used for node registry, job store, auth, and shared state
• NATS server — used for real-time backend lifecycle events and file staging
• All services must be on the same network (or reachable via configured URLs)

Quick Start with Docker Compose

The easiest way to try distributed mode locally is with the provided Docker Compose file:

docker compose -f docker-compose.distributed.yaml up

This starts PostgreSQL, NATS, a LocalAI frontend, and one worker node. When you send an inference request, the SmartRouter automatically installs the needed backend on the worker and loads the model. See the file for details on adding GPU support, shared volumes, and additional workers.

Frontend Configuration

The frontend is a standard LocalAI instance with distributed mode enabled. These flags are added to the local-ai run command:

Optional: S3 Object Storage

For multi-host deployments where workers don’t share a filesystem, S3-compatible storage enables distributed file transfer (model files, configs):

When S3 is not configured, model files are transferred directly from the frontend to workers via HTTP — no shared filesystem needed. Each worker runs a small HTTP file transfer server alongside the gRPC backend process. This is the default and works out of the box.

For high-throughput or very large model files, S3 can be more efficient since it avoids streaming through the frontend.

Worker Configuration

Workers are started with the worker subcommand. Each worker is generic — it doesn’t need a backend type at startup:

local-ai worker \
  --register-to http://frontend:8080 \
  --registration-token changeme \
  --nats-url nats://nats:4222

Worker Address Configuration

The simplest way to configure a worker’s network address is with a single variable:

Example:

environment:
  LOCALAI_ADDR: "192.168.1.100:50051"
  LOCALAI_NATS_URL: "nats://frontend:4222"
  LOCALAI_REGISTER_TO: "http://frontend:8080"
  LOCALAI_REGISTRATION_TOKEN: "my-secret"

For advanced networking scenarios (NAT, load balancers, separate gRPC/HTTP ports), the following override variables are available:

Node Labels

Workers can declare labels at startup for scheduling constraints:

Labels can also be managed via the admin API (see Label Management API below).

The system automatically applies hardware-detected labels on registration:

• gpu.vendor – GPU vendor (nvidia, amd, intel, vulkan)
• gpu.vram – GPU VRAM bucket (8GB, 16GB, 24GB, 48GB, 80GB+)
• node.name – The node’s registered name

How Workers Operate

Workers start as generic processes with no backend installed. When the SmartRouter needs to load a model on a worker, it sends a NATS backend.install event with the backend name and model ID. The worker:

1. Installs the backend from the gallery (if not already installed)
2. Starts a new gRPC backend process on a dynamic port (each model gets its own process)
3. Replies with the allocated gRPC address
4. The SmartRouter calls LoadModel via direct gRPC to that address

Workers can run multiple models concurrently — each model gets its own gRPC process on a separate port. For example, an embedding model on port 50051 and a chat model on port 50052 can run simultaneously on the same worker.

When the SmartRouter needs to free capacity, it can unload models with zero in-flight requests without affecting other models on the same worker.

Node Management API

The API is split into two prefixes with distinct auth:

/api/node/ — Node self-service

Used by workers themselves (registration, heartbeat, etc.). Authenticated via the registration token, exempt from global auth.

/api/nodes/ — Admin management

Used by the WebUI and admin API consumers. Requires admin authentication.

The Nodes page in the React WebUI provides a visual overview of all registered workers, their statuses, and loaded models.

Node Approval

By default, new worker nodes start in pending status and must be approved by an admin before they can receive traffic. This prevents unknown machines from joining the cluster.

To approve a pending node via the API:

curl -X POST http://frontend:8080/api/nodes/<node-id>/approve \
  -H "Authorization: Bearer <admin-token>"

The Nodes page in the WebUI also shows pending nodes with an Approve button.

To skip manual approval and let nodes join immediately, set --auto-approve-nodes (or LOCALAI_AUTO_APPROVE_NODES=true) on the frontend. This is convenient for development and trusted environments.

Node Statuses

Agent Workers

Agent workers are dedicated processes for executing agent chats and MCP CI jobs. Unlike backend workers (which run gRPC model inference), agent workers use cogito to orchestrate multi-step conversations with tool calls.

local-ai agent-worker \
  --register-to http://frontend:8080 \
  --nats-url nats://nats:4222 \
  --registration-token changeme

Agent workers:

• Execute agent chat messages dispatched via NATS
• Run MCP CI jobs (with access to MCP servers via docker)
• Handle MCP tool discovery and execution requests from the frontend
• Get auto-provisioned API keys during registration for calling the inference API

In the docker-compose setup, the agent worker mounts the Docker socket so it can run MCP stdio servers (e.g., docker run commands):

agent-worker-1:
  command: agent-worker
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

MCP in Distributed Mode

MCP servers configured in model configs work in distributed mode. The frontend routes MCP operations through NATS to agent workers:

• MCP discovery (GET /v1/mcp/servers/:model): routed to agent workers which create sessions and return server info
• MCP tool execution (during /v1/chat/completions): tool calls are routed to agent workers via NATS request-reply
• MCP CI jobs: executed entirely on agent workers with access to docker for stdio-based MCP servers

Scaling

Adding worker capacity: Start additional worker instances pointing to the same frontend. They self-register automatically:

# Additional workers — no backend type needed
local-ai worker \
  --register-to http://frontend:8080 \
  --node-name worker-2 \
  --nats-url nats://nats:4222 \
  --registration-token changeme

local-ai worker \
  --register-to http://frontend:8080 \
  --node-name worker-3 \
  --nats-url nats://nats:4222 \
  --registration-token changeme

Multiple frontend replicas: Run multiple LocalAI frontends behind a load balancer. Since all state is in PostgreSQL and coordination is via NATS, frontends are fully stateless and interchangeable.

Model Scheduling

Model scheduling controls where models are placed and how many replicas are maintained. It combines two optional features:

Node Selectors

Pin models to nodes with specific labels. Only nodes matching all selector labels are eligible:

# Only schedule on NVIDIA nodes in the us-east zone
curl -X POST http://frontend:8080/api/nodes/scheduling \
  -H "Content-Type: application/json" \
  -d '{"model_name": "llama3", "node_selector": {"gpu.vendor": "nvidia", "zone": "us-east"}}'

Without a node selector, models can schedule on any healthy node (default behavior).

Replica Auto-Scaling

Control the number of model replicas across the cluster:

Auto-scaling is only active when min_replicas > 0 or max_replicas > 0.

# Scale llama3 between 2 and 4 replicas on NVIDIA nodes
curl -X POST http://frontend:8080/api/nodes/scheduling \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "llama3",
    "node_selector": {"gpu.vendor": "nvidia"},
    "min_replicas": 2,
    "max_replicas": 4
  }'

The Replica Reconciler runs as a background process on the frontend:

• Scale up: Adds replicas when all existing replicas are busy (have in-flight requests)
• Scale down: Removes idle replicas after 5 minutes of inactivity
• Maintain minimum: Ensures min_replicas are always loaded (recovers from node failures)
• Eviction protection: Models with auto-scaling enabled are never evicted below min_replicas

All fields are optional and composable:

• Node selector only: pin model to matching nodes, single replica
• Replicas only: auto-scale across all nodes
• Both: auto-scale on matching nodes only

Label Management API

Scheduling API

Comparison with P2P

Troubleshooting

Worker not registering:

• Verify the frontend URL is reachable from the worker (curl http://frontend:8080/api/node/register)
• Check that --registration-token matches on both frontend and worker
• Ensure auth is enabled on the frontend (LOCALAI_AUTH=true)

NATS connection errors:

• Confirm NATS is running and reachable (nats-server --signal ldm or check port 4222)
• Check that --nats-url uses the correct hostname/IP from the worker’s network perspective

PostgreSQL connection errors:

• Verify the connection URL format: postgresql://user:password@host:5432/dbname?sslmode=disable
• Ensure the database exists and the user has CREATE TABLE permissions (for auto-migration)
• Check that pgvector extension is installed if using RAG features

Node shows as unhealthy or offline:

• The HealthMonitor marks nodes offline when heartbeats are missed. Check network connectivity between worker and frontend.
• Verify --heartbeat-interval is not set too high
• Offline nodes automatically restore to healthy when they re-register (no re-approval needed)

Backend not installing:

• Check the worker logs for backend.install events

Port conflicts on workers:

• Each model gets its own gRPC process on an incrementing port (50051, 50052, …)
• The HTTP file transfer server runs on the base port - 1 (default: 50050)
• Ensure the port range is not blocked by firewalls or used by other services
• Verify the backend gallery configuration is correct
• The worker needs network access to download backends from the gallery