Distributed Mode

Distributed mode enables horizontal scaling of LocalAI across multiple machines using PostgreSQL for state and node registry, and NATS for real-time coordination. Unlike the P2P/federation approach, distributed mode is designed for production deployments and Kubernetes environments where you need centralized management, health monitoring, and deterministic routing.

Architecture Overview

Frontends are stateless LocalAI instances that receive API requests and route them to worker nodes via the SmartRouter. All frontends share state through PostgreSQL and coordinate via NATS.

Workers are generic processes that self-register with a frontend. They don’t have a fixed backend type - the SmartRouter dynamically installs the required backend via NATS backend.install events when a model request arrives.

Scheduling Algorithm

The SmartRouter uses idle-first scheduling with preemptive eviction:

1. If the model is already loaded on a node → use it (per-model gRPC address)
2. Drop any node without room to store the model on its models filesystem (see Disk headroom)
3. If no node has the model → prefer nodes with enough free VRAM
4. Fall back to idle nodes (zero models), then least-loaded nodes
5. If no node has capacity → evict the least-recently-used model with zero in-flight requests to free a node
6. If all models are busy → wait (with timeout) for a model to become idle, then evict
7. Send backend.install NATS event with backend name + model ID → worker starts a new gRPC process on a dynamic port
8. SmartRouter calls gRPC LoadModel on the model-specific port, records in DB

Each model gets its own gRPC backend process, so a single worker can serve multiple models simultaneously (e.g., a chat model and an embedding model).

Prerequisites

• PostgreSQL (with pgvector extension recommended for RAG) - used for node registry, job store, auth, and shared state
• NATS server - used for real-time backend lifecycle events and file staging
• All services must be on the same network (or reachable via configured URLs)

Quick Start with Docker Compose

The easiest way to try distributed mode locally is with the provided Docker Compose file:

docker compose -f docker-compose.distributed.yaml up

This starts PostgreSQL, NATS, a LocalAI frontend, and one worker node. When you send an inference request, the SmartRouter automatically installs the needed backend on the worker and loads the model. See the file for details on adding GPU support, shared volumes, and additional workers.

Frontend Configuration

The frontend is a standard LocalAI instance with distributed mode enabled. These flags are added to the local-ai run command:

The model load deadline scales with the checkpoint

The LoadModel deadline starts after the backend is installed and the model files are staged, so it covers only the worker backend’s own checkpoint read and pipeline init. That work is proportional to the bytes on disk, which makes any fixed deadline a model-size cliff rather than a timeout: a 70 GB video checkpoint on a Jetson Thor worker failed reproducibly against the old fixed 5m default (rpc error: code = DeadlineExceeded after 953.5s of wall clock, roughly 11m of which was backend install and staging), and simply raising the constant would only move the cliff to the next larger model while making a genuinely wedged small model hang for the whole inflated duration.

So the deadline is derived per model:

budget = 5m + 20s per GiB of checkpoint,  capped at 6h

The per-GiB rate is deliberately pessimistic — it corresponds to reading weights at about 54 MB/s, below what any supported storage sustains — because the two errors are not symmetric: a budget that is too long costs only failure latency on a load that was going to fail anyway, while a budget that is too short causes a guaranteed false failure on a load that was perfectly healthy.

The size is measured from the model files on the frontend’s disk, over the same set of paths that get staged to the worker. If those files are not present locally — a backend handed a bare HuggingFace repo id fetches its own weights on the worker — there is nothing to measure and the budget stays at the plain 5m default. Pin LOCALAI_NATS_MODEL_LOAD_TIMEOUT for those models if their load is slow.

When the budget is exceeded, the error names the budget, the checkpoint size it was derived from, and the knob that overrides it, instead of surfacing a bare context deadline exceeded.

The cold-load lock ceiling

The router also bounds how long a single cold load may hold the per-model advisory lock, so a worker that dies mid-install cannot pin every other replica’s request for that model. That bound is derived, not configured, and it is based on progress rather than on wall-clock time.

The load starts with a base budget of max(backend-install-timeout + model-load-timeout + 5m, 25m) — with the defaults, 15m + 5m + 5m = 25m. That budget covers the steps that report no progress: node selection, backend install, and the remote LoadModel call. Raising either timeout widens it in step, so a longer load deadline is never clipped.

That base is the hold’s starting budget, not its maximum. Because the derived load budget above can exceed it — a 70 GB checkpoint’s 28m20s against a 25m base — the hold is widened again as the router enters the load phase, by the derived budget plus the same 5m of slack. Without that step the ceiling would cancel a load that was still comfortably inside its own deadline.

While model files are staging, however, the deadline extends every time staging does real work, and expires only once staging has been silent for a 5-minute stall window. Real work means uploaded bytes, and also the resumable-upload verify phase: when a shard is already present on the worker from an earlier attempt, the frontend HEADs it and hashes the local copy to confirm it matches, then skips the transfer. That phase uploads nothing at all — on a 70 GB model resuming with 56 GB already staged it ran for six-plus consecutive minutes at ~45s per shard — so hashing counts as progress too. Otherwise a resumed transfer would be mistaken for a wedged one. Staging time is a function of checkpoint size and available bandwidth, not a constant: a 70 GB model at 26 MB/s needs about 45 minutes, and a 600 GB checkpoint needs hours. A fixed ceiling would therefore be a model-size cliff — every increase just moves the cliff to the next larger model. Extending on progress means a large model transfers for as long as it legitimately needs, while a worker that dies mid-transfer still releases the lock within the stall window.

An absolute cap of 24h ends the hold even if progress keeps arriving, so a degenerate peer trickling a few bytes at a time cannot pin the lock forever. No configuration is needed for either value; both are sized well above any legitimate transfer.

NATS JWT authentication (recommended for production)

By default, NATS connections are anonymous: any client that can reach port 4222 may publish control-plane subjects such as nodes.<id>.backend.install. Enable JWT auth to scope workers to their own node subjects and give the frontend a dedicated service credential.

NATS TLS / mTLS (optional)

Use tls:// in --nats-url / LOCALAI_NATS_URL for encrypted transport. When the server uses a private CA or requires client certificates, set:

The same env vars apply to backend workers and local-ai agent-worker. If the server cert is already trusted by the OS, tls:// alone is enough.

Worker register response (when minting is enabled and the node is approved):

{
  "id": "…",
  "nats_jwt": "eyJ…",
  "nats_user_seed": "SU…"
}

Workers connect with that JWT and seed automatically (shown once; store securely). Override with LOCALAI_NATS_JWT / LOCALAI_NATS_USER_SEED if needed. Set LOCALAI_NATS_REQUIRE_AUTH=true on workers when the bus requires credentials.

When LOCALAI_NATS_REQUIRE_AUTH=true and no static credentials are provided, a worker that registers while still pending admin approval keeps re-registering (with backoff) until an admin approves it and the frontend mints its JWT - it does not start unauthenticated. This retry is bounded: if the node is never approved (or no credentials are minted) after a large number of attempts, the worker exits non-zero so the failure is visible (a crash-looping or failed worker) rather than hanging silently. Minted worker JWTs are also refreshed automatically before they expire (the worker re-registers at ~75% of the JWT lifetime), so long-running workers survive past LOCALAI_NATS_WORKER_JWT_TTL; the NATS connection picks up the new JWT on its next reconnect. If refresh fails persistently, the worker exits (to restart and re-acquire) rather than drifting toward an expired, unrenewable JWT. Statically configured (LOCALAI_NATS_JWT) and service (LOCALAI_NATS_SERVICE_JWT) credentials are used as-is and not refreshed.

Generate operator/account material with scripts/nats-auth-setup.sh (requires nsc). Configure the NATS server with account resolver JWTs before enabling LOCALAI_NATS_REQUIRE_AUTH.

Optional: S3 Object Storage

For multi-host deployments where workers don’t share a filesystem, S3-compatible storage enables distributed file transfer (model files, configs):

When S3 is not configured, model files are transferred directly from the frontend to workers via HTTP - no shared filesystem needed. Each worker runs a small HTTP file transfer server alongside the gRPC backend process. This is the default and works out of the box.

For high-throughput or very large model files, S3 can be more efficient since it avoids streaming through the frontend.

Shared models directory

If every node (frontend and workers) mounts the same models directory at the same path - for example a shared volume or network filesystem, as shown in the “Shared Volume Mode” section of docker-compose.distributed.yaml - the model files are already present on each worker at their canonical path. In that case staging is wasted work: it copies files that already exist into a per-model subdirectory the worker then loads from, which shows up as a re-download of a model you already have.

Set LOCALAI_DISTRIBUTED_SHARED_MODELS=true (or --distributed-shared-models) on the frontend to skip staging entirely. The router then leaves the model’s absolute paths untouched and the worker loads them directly from the shared volume.

This flag is a contract you assert: all nodes must mount identical paths. Leave it off (the default) when workers have independent models directories - the frontend stages files to them over HTTP (or S3) as described above.

Model artifact staging

For managed Hugging Face artifacts, the controller resolves the repository and downloads every selected file. Workers receive the committed snapshot through the existing directory stager. They never receive HF_TOKEN and do not contact Hugging Face for managed artifacts.

With LOCALAI_DISTRIBUTED_SHARED_MODELS enabled, workers use the shared absolute snapshot path and skip transfer. Otherwise, the controller stages the complete snapshot tree to each worker before loading the backend.

Watching Backend Installs

While a worker downloads a backend, the admin operations strip at the top of the UI shows real-time progress: a percentage, and, when the install targets several workers, a roll-up of how far the fan-out has got, 2 of 5 nodes done. Per-file byte counts are not on the strip; they are in the per-node detail below.

The per-node detail is on the Operate → Activity page (Activity). When an install targets more than one worker, an N nodes tag appears on the operation card, with one row per worker showing:

• A status pill: Queued (gray), Downloading (blue), Worker busy (yellow), Done (green), or Failed (red).
• The file currently being downloaded with current/total bytes and percentage.
• A thin per-node progress bar.
• Any error returned by the worker.

The yellow Worker busy pill means the worker took longer than --backend-install-timeout to acknowledge but is most likely still working in the background. The admin UI clears it as soon as the worker finishes; no action is required from the operator.

If a worker is running an older LocalAI release that does not report progress, its row in the breakdown will still show terminal status (queued / done / failed / worker busy) but no per-file progress.

The Record on that page - what model and backend installs and removals have finished - is read from PostgreSQL rather than from the replica’s memory. Every replica reports the same record, it survives restarts, a replica added by a scale-out or a rolling deploy reports it in full, and Clear history clears it for every replica.

Worker Configuration

Workers are started with the worker subcommand. Each worker is generic - it doesn’t need a backend type at startup:

local-ai worker \
  --register-to http://frontend:8080 \
  --registration-token changeme \
  --nats-url nats://nats:4222

Worker Health Probes

The worker’s HTTP server (base port - 1, default 50050) exposes two unauthenticated probes:

/readyz reports something the frontend cannot see on its own. The node registry’s status and last_heartbeat are driven by an HTTP heartbeat to the frontend, which is a different network path from NATS — a worker can keep heartbeating while its NATS link is dead, and so appear healthy in the registry while being unable to receive any work. The local probe closes that gap.

The container image’s HEALTHCHECK detects worker mode and probes this endpoint automatically; no HEALTHCHECK_ENDPOINT override is needed. Set HEALTHCHECK_ENDPOINT only to pin an explicit URL.

Worker Address Configuration

The simplest way to configure a worker’s network address is with a single variable:

Example:

environment:
  LOCALAI_ADDR: "192.168.1.100:50051"
  LOCALAI_NATS_URL: "nats://frontend:4222"
  LOCALAI_REGISTER_TO: "http://frontend:8080"
  LOCALAI_REGISTRATION_TOKEN: "my-secret"

For advanced networking scenarios (NAT, load balancers, separate gRPC/HTTP ports), the following override variables are available:

Backend gRPC port range

Every backend process a worker starts listens on its own gRPC port, allocated upward from the worker’s base port (LOCALAI_SERVE_ADDR, default 50051). LOCALAI_GRPC_MAX_PORT sets the top of that range. The width of [base port, LOCALAI_GRPC_MAX_PORT] is therefore a hard cap on how many backend processes one worker can run concurrently.

Set it when the worker shares a host with other services and you need to keep the rest of the ephemeral range clear, or when you want a worker’s backend count bounded explicitly rather than by whatever the host happens to allow:

# Confine this worker's backends to 50051-50150 (100 concurrent backends).
LOCALAI_SERVE_ADDR=0.0.0.0:50051
LOCALAI_GRPC_MAX_PORT=50150

Leave it unset (the default) and the worker may use anything up to 65535.

Budget headroom above your real concurrency. When a backend stops, its port is held briefly before it can be reused, so a worker with heavy start/stop churn has more ports tied up than it has running backends at any instant. If the range does fill, backend starts fail with:

no free gRPC port in range: 50051-50150 is fully consumed by 100 running
backend(s) and 12 port(s) still in quarantine; raise LOCALAI_GRPC_MAX_PORT to
widen the range

Raise LOCALAI_GRPC_MAX_PORT (or reduce how many models you schedule onto that worker). A value above 65535 is clamped, and a value below the base port is ignored in favour of the full range, so a typo degrades the setting rather than wedging every backend start on the node.

NVIDIA GPU support

When running workers in a container, two runtime settings affect how VRAM usage is reported back to the frontend:

• NVIDIA_DRIVER_CAPABILITIES must include utility. Without it, the NVML library (and therefore nvidia-smi) is not available inside the container. CUDA compute still works, but the worker cannot query free VRAM and the Nodes page will show the node as fully used. Set NVIDIA_DRIVER_CAPABILITIES=compute,utility (or, with the NVIDIA CDI runtime, list capabilities: [gpu, utility] on the device reservation).

• Run the container with init: true (or docker run --init). The worker process becomes PID 1 in the container and cannot reap zombies on its own. Without an init, nvidia-smi calls can fail intermittently with waitid: no child processes, which briefly clears free-VRAM metrics.

Unified memory devices (Jetson, DGX Spark / GB10, Thor): these SoCs share one physical RAM between CPU and GPU. LocalAI detects them via /sys/devices/soc0/family and /sys/devices/soc0/soc_id (no nvidia-smi required) and reports system-RAM figures as VRAM. Free VRAM therefore tracks MemAvailable in /proc/meminfo.

Node Labels

Workers can declare labels at startup for scheduling constraints:

Labels can also be managed via the admin API (see Label Management API below).

The system automatically applies hardware-detected labels on registration:

• gpu.vendor – GPU vendor (nvidia, amd, intel, vulkan)
• gpu.vram – GPU VRAM bucket (8GB, 16GB, 24GB, 48GB, 80GB+)
• node.name – The node’s registered name

How Workers Operate

Workers start as generic processes with no backend installed. When the SmartRouter needs to load a model on a worker, it sends a NATS backend.install event with the backend name and model ID. The worker:

1. Installs the backend from the gallery (if not already installed)
2. Starts a new gRPC backend process on a dynamic port (each model gets its own process)
3. Replies with the allocated gRPC address
4. The SmartRouter calls LoadModel via direct gRPC to that address

Workers can run multiple models concurrently - each model gets its own gRPC process on a separate port. For example, an embedding model on port 50051 and a chat model on port 50052 can run simultaneously on the same worker.

When the SmartRouter needs to free capacity, it can unload models with zero in-flight requests without affecting other models on the same worker.

Node Management API

The API is split into two prefixes with distinct auth:

/api/node/ - Node self-service

Used by workers themselves (registration, heartbeat, etc.). Authenticated via the registration token, exempt from global auth.

/api/nodes/ - Admin management

Used by the WebUI and admin API consumers. Requires admin authentication.

The Nodes page in the React WebUI provides a visual overview of all registered workers, their statuses, and loaded models. The page opens with a one-line cluster pulse summarising node health and an attention callout that surfaces nodes needing action (for example pending approvals). Below that, a roster of node panels lists each worker with its inline model chips (no expand click needed), filtered by an All / Backend / Agent segmented control. Selecting a panel opens a dedicated node detail page at /app/nodes/:id with per-node metrics, models, and backend actions. Model scheduling lives on its own Scheduling page (separate nav item), not as a tab on the Nodes page.

Per-node VRAM budget

Each worker advertises its detected VRAM, and the SmartRouter uses that number when picking a node with enough free memory. You can cap the VRAM a node offers for placement so it never gets scheduled beyond a chosen limit, leaving headroom for other workloads on that machine.

There are two ways to set the cap:

• At the worker: start it with --vram-budget / LOCALAI_VRAM_BUDGET (see Worker Configuration).
• From the frontend, live: set it per node in the node capacity editor on the node detail page, or via the admin API:

# Cap node placement at 80% of its detected VRAM
curl -X PUT http://frontend:8080/api/nodes/<node-id>/vram-budget \
  -H "Authorization: Bearer <admin-token>" \
  -H "Content-Type: application/json" \
  -d '{"value":"80%"}'

# Or an absolute amount
curl -X PUT http://frontend:8080/api/nodes/<node-id>/vram-budget \
  -H "Authorization: Bearer <admin-token>" \
  -d '{"value":"12GB"}'

# Clear the budget (revert to all detected VRAM)
curl -X DELETE http://frontend:8080/api/nodes/<node-id>/vram-budget \
  -H "Authorization: Bearer <admin-token>"

The value accepts the same formats as the standalone budget: a percentage (80%) or an absolute amount (12GB, 12GiB, 12000MB, or raw bytes). It is a hard ceiling: the node’s advertised VRAM becomes min(detected, budget), so a budget can only lower the number, never raise it above the hardware. An admin-set node budget is sticky across worker restarts: it is stored in the node registry and reapplied when the worker re-registers, so it wins over whatever the worker reports on reconnect. For the underlying semantics and the standalone equivalent, see VRAM Budget.

Disk headroom

Model weights are staged onto the worker’s disk before the backend loads them, so a node needs free space as well as free VRAM. Each worker reports the capacity of the filesystem backing its models directory (--models-path), not the root filesystem, on registration and on every heartbeat. Those figures appear as total_disk and available_disk in the nodes API and as Models disk free on the node detail page.

Before placing a model, the SmartRouter removes any node whose models filesystem cannot hold it. The requirement is derived from the model’s actual on-disk size plus a small margin (5%, at least 1 GiB), rather than a fixed percentage of the node’s disk — a fixed threshold would take a small-but-usable node out of rotation for models it could comfortably store. When the model’s size cannot be determined locally (a bare HuggingFace repo id that the worker fetches itself), the node only has to clear a 2 GiB floor.

If no node has enough space, the request fails immediately with a capacity error naming the requirement and each node’s free space, for example:

scheduling longcat-video-avatar-1.5: no node has enough free disk for the model:
need 73.5 GB free on the models filesystem, but nvidia-thor has 0 B free of 937.0 GB

This is deliberately a scheduling-time verdict. Without it, a worker with a full disk still reported status: healthy, accepted the staging request, transferred tens of gigabytes and only then failed with no space left on device — minutes after a decision that could never have succeeded.

Workers that predate this feature (or whose disk reading fails) report total_disk as 0. Such nodes are treated as unknown, not full, and stay in rotation, so a rolling upgrade never empties the candidate pool. A full disk is distinguishable because it reports a non-zero total_disk with available_disk at 0.

Low disk does not mark a node unhealthy. Disk is compared per model rather than against a global threshold, so a node that is too small for one model remains a valid target for smaller ones. The check is also skipped entirely in shared-models mode, where nothing is staged to the worker at all.

Turning the check off

The check is on by default. To disable it, start the frontend with --distributed-disk-headroom-check=false / LOCALAI_DISTRIBUTED_DISK_HEADROOM_CHECK=false, or toggle Settings → Distributed → Disk headroom check in the WebUI (distributed_disk_headroom_check via POST /api/settings). The runtime setting takes effect on the next placement, with no restart; the env/CLI flag only sets the value LocalAI boots with, and both write the same underlying value, so the last change wins.

Disabling means warn, do not block. Node selection goes back to ignoring free disk (the pre-check behaviour), but the check still runs, and when it would have rejected every node it logs a warning naming the shortfall:

WARN No node has room to store this model, but the disk-headroom check is DISABLED;
     scheduling anyway — staging will most likely fail with ENOSPC
     model=longcat-video-avatar-1.5 knob=distributed-disk-headroom-check

The alternative — skipping the check outright — was rejected because it reproduces the condition that made the original bug expensive: the cluster was doing something that could not work and said nothing about it. The escape hatch exists for setups where the size estimate is wrong (deduplicating or compressing filesystems, a backend that fetches its own weights rather than using the staged copy), and in exactly those cases the operator needs to see what LocalAI thought was wrong. Disabling is logged once at startup as well.

The LocalAI Assistant can also set a node budget conversationally through the set_node_vram_budget MCP tool.

Node Approval

By default, new worker nodes start in pending status and must be approved by an admin before they can receive traffic. This prevents unknown machines from joining the cluster.

To approve a pending node via the API:

curl -X POST http://frontend:8080/api/nodes/<node-id>/approve \
  -H "Authorization: Bearer <admin-token>"

The Nodes page in the WebUI also shows pending nodes with an Approve button.

To skip manual approval and let nodes join immediately, set --auto-approve-nodes (or LOCALAI_AUTO_APPROVE_NODES=true) on the frontend. This is convenient for development and trusted environments.

Node Statuses

Agent Workers

Agent workers are dedicated processes for executing agent chats and MCP CI jobs. Unlike backend workers (which run gRPC model inference), agent workers use cogito to orchestrate multi-step conversations with tool calls.

local-ai agent-worker \
  --register-to http://frontend:8080 \
  --nats-url nats://nats:4222 \
  --registration-token changeme

Agent workers:

• Execute agent chat messages dispatched via NATS
• Run MCP CI jobs (with access to MCP servers via docker)
• Handle MCP tool discovery and execution requests from the frontend
• Get auto-provisioned API keys during registration for calling the inference API

In the docker-compose setup, the agent worker mounts the Docker socket so it can run MCP stdio servers (e.g., docker run commands):

agent-worker-1:
  command: agent-worker
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

MCP in Distributed Mode

MCP servers configured in model configs work in distributed mode. The frontend routes MCP operations through NATS to agent workers:

• MCP discovery (GET /v1/mcp/servers/:model): routed to agent workers which create sessions and return server info
• MCP tool execution (during /v1/chat/completions): tool calls are routed to agent workers via NATS request-reply
• MCP CI jobs: executed entirely on agent workers with access to docker for stdio-based MCP servers

vLLM Multi-Node (Data-Parallel)

A single vLLM model can span multiple GPU nodes via data parallelism: the head node serves the OpenAI API and runs the local DP ranks, follower nodes run vanilla vllm serve --headless and speak ZMQ directly to the head. LocalAI’s role is starting the follower processes and surfacing them in the admin UI; the cross-rank tensor traffic is vLLM’s own.

This mode is operator-launched - the head config and each follower’s invocation must agree on the topology (data_parallel_size, data_parallel_size_local, data_parallel_address, data_parallel_rpc_port). The SmartRouter does not place follower ranks automatically.

Head node configuration

The head runs the existing single-node vLLM gRPC backend. Set engine_args to publish the DP topology vLLM expects:

backend: vllm
parameters:
  model: moonshotai/Kimi-K2.6-Instruct
engine_args:
  data_parallel_size: 4              # total ranks across all nodes
  data_parallel_size_local: 2        # ranks on the head node
  data_parallel_address: 10.0.0.1    # head's reachable IP
  data_parallel_rpc_port: 32100      # any free port; followers connect here
  enable_expert_parallel: true       # for MoE models

The head will start its 2 local ranks, listen on 10.0.0.1:32100, and wait for the remaining 2 ranks to handshake.

Follower nodes

Each follower runs local-ai p2p-worker vllm with matching topology, an explicit start rank, and the head’s address:

local-ai p2p-worker vllm \
  moonshotai/Kimi-K2.6-Instruct \
  --data-parallel-size 4 \
  --data-parallel-size-local 2 \
  --start-rank 2 \
  --master-addr 10.0.0.1 \
  --master-port 32100 \
  --register-to http://frontend:8080 \
  --registration-token changeme

--register-to is optional but recommended - it makes the follower visible in the admin UI as an agent-type node tagged with node.role=vllm-follower. Without it the worker just runs vLLM and exits silently when vLLM does. The role label discourages SmartRouter from placing other models on the follower; pair it with model selectors like {"!node.role":"vllm-follower"} if you also run regular LocalAI models on the same fleet.

Worked example: 2-node Kimi-K2.6 deployment

Two A100 nodes (10.0.0.1, 10.0.0.2), 8 GPUs total, data_parallel_size=8 with 4 ranks per node:

# /models/kimi.yaml on the head (10.0.0.1)
name: kimi-k2-6
backend: vllm
parameters:
  model: moonshotai/Kimi-K2.6-Instruct
engine_args:
  data_parallel_size: 8
  data_parallel_size_local: 4
  data_parallel_address: 10.0.0.1
  data_parallel_rpc_port: 32100
  enable_expert_parallel: true
  all2all_backend: deepep_high_throughput

# On 10.0.0.2 (follower)
local-ai p2p-worker vllm moonshotai/Kimi-K2.6-Instruct \
  --data-parallel-size 8 --data-parallel-size-local 4 --start-rank 4 \
  --master-addr 10.0.0.1 --master-port 32100 \
  --register-to http://10.0.0.1:8080 --registration-token changeme

A curl http://10.0.0.1:8080/v1/chat/completions ... against the head will then dispatch across all 8 ranks.

Intel Arc / XPU notes

vLLM XPU supports DP (vllm/platforms/xpu.py:198 handles world_size_across_dp > 1; ranks bind to xpu:{local_rank} in xpu_worker.py:62, with xccl as the collective backend). Each rank still needs a distinct discrete GPU - the iGPU on a hybrid host is not a viable second device.

Older XE-HPG GPUs (e.g. Arc A770) need to bypass the cutlass attention path:

engine_args:
  attention_backend: TRITON_ATTN

docker-compose.vllm-multinode.intel.yaml at the repo root is the Intel equivalent of docker-compose.vllm-multinode.yaml - uses /dev/dri passthrough, ZE_AFFINITY_MASK to pin each rank to one device, and latest-gpu-intel images. Run via ./tests/e2e/vllm-multinode/smoke.sh --intel.

Caveats

• Tensor parallel within a node only. vLLM v1 does not support TP across nodes; combine tensor_parallel_size (within a node, via engine_args) with data_parallel_size (across nodes).
• Followers don’t host LocalAI gRPC. The follower process is vanilla vLLM, so /api/backend-logs/<modelId> does not stream follower output. Use journalctl / kubectl logs / compose logs for the follower’s stderr.
• Network reachability. The head’s data_parallel_rpc_port plus a range of ZMQ ports (typically data_parallel_rpc_port..+N) must be reachable from every follower. Open them in your firewall / security group.
• Topology must match exactly. A mismatch in --data-parallel-size between head and any follower will hang the handshake. Check the head’s vLLM logs for waiting for N DP ranks if startup stalls.

ds4 Layer-Split Distributed Inference

The ds4 backend (DeepSeek V4 Flash) supports layer-parallel distributed inference: a single model that is too large for one machine is split by transformer layer across several machines. Each machine must have the GGUF present locally, but loads only its own slice of the layers. This lets you run a model whose weights exceed any single host’s memory.

This is not routed through the SmartRouter: it is a model-internal split, configured manually (Phase 1). It is unrelated to the NATS/PostgreSQL distributed mode described above.

Topology

ds4 uses a coordinator/worker split:

• The coordinator owns tokenization, sampling, the prompt, and a low layer range (e.g. 0:19). It is LocalAI’s ds4 backend and listens on a host/port. Workers dial into it.
• One or more workers own higher layer ranges (e.g. 20:output). Each worker loads only its slice and dials the coordinator to register the range it can serve. The last worker normally owns the output head.
• Activations flow through the connected slices and back to the coordinator. The route is “ready” only once the coordinator plus all connected workers cover every layer.

This dial direction is the inverse of the llama.cpp RPC model, where the main server dials out to a list of rpc-server workers. With ds4 the workers dial in to the coordinator.

Coordinator setup

The coordinator is a normal LocalAI ds4 model whose YAML carries distributed options::

name: ds4flash
backend: ds4
options:
  - "ds4_role:coordinator"
  - "ds4_layers:0:19"
  - "ds4_listen:0.0.0.0:1234"

Once the model is loaded, the coordinator serves requests exactly like a single-node ds4 model: generation goes through the ordinary inference path and is transparently routed across the layer slices.

Worker setup

On each worker machine (with the GGUF present locally), start a worker pointed at the coordinator:

local-ai worker ds4-distributed -- \
  --role worker \
  --model /models/ds4flash.gguf \
  --layers 20:output \
  --coordinator <coordinator-host> 1234

local-ai worker ds4-distributed resolves the ds4 backend and execs the packaged ds4-worker binary, passing everything after -- straight through.

Layer-range semantics

• Ranges are inclusive: 0:19 is layers 0 through 19.
• N:output means layer N through the final layer plus the output head. The last worker normally owns the output head.
• The coordinator and all connected workers together must cover every layer. Until they do, the coordinator returns a gRPC UNAVAILABLE error on inference requests (so a worker that starts slightly after the coordinator is tolerated: once it connects and the route is complete, requests succeed). The wait is tunable via ds4_route_timeout.

Scaling

Adding worker capacity: Start additional worker instances pointing to the same frontend. They self-register automatically:

# Additional workers - no backend type needed
local-ai worker \
  --register-to http://frontend:8080 \
  --node-name worker-2 \
  --nats-url nats://nats:4222 \
  --registration-token changeme

local-ai worker \
  --register-to http://frontend:8080 \
  --node-name worker-3 \
  --nats-url nats://nats:4222 \
  --registration-token changeme

Multiple frontend replicas: Run multiple LocalAI frontends behind a load balancer. Since all state is in PostgreSQL and coordination is via NATS, frontends are fully stateless and interchangeable.

Model Scheduling

Model scheduling controls where models are placed and how many replicas are maintained. In the React WebUI it has its own Scheduling page (a top-level nav item, separate from the Nodes page). It combines two optional features:

Node Selectors

Pin models to nodes with specific labels. Only nodes matching all selector labels are eligible:

# Only schedule on NVIDIA nodes in the us-east zone
curl -X POST http://frontend:8080/api/nodes/scheduling \
  -H "Content-Type: application/json" \
  -d '{"model_name": "llama3", "node_selector": {"gpu.vendor": "nvidia", "zone": "us-east"}}'

Without a node selector, models can schedule on any healthy node (default behavior).

Replica Auto-Scaling

Control the number of model replicas across the cluster:

Auto-scaling is only active when min_replicas > 0 or max_replicas > 0.

# Scale llama3 between 2 and 4 replicas on NVIDIA nodes
curl -X POST http://frontend:8080/api/nodes/scheduling \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "llama3",
    "node_selector": {"gpu.vendor": "nvidia"},
    "min_replicas": 2,
    "max_replicas": 4
  }'

The Replica Reconciler runs as a background process on the frontend:

• Scale up: Adds replicas when all existing replicas are busy (have in-flight requests)
• Scale down: Removes idle replicas after 5 minutes of inactivity
• Maintain minimum: Ensures min_replicas are always loaded (recovers from node failures)
• Eviction protection: Models with auto-scaling enabled are never evicted below min_replicas
• Restart-safe: Per-model load metadata (backend type + ModelOptions) is persisted in the model_load_infos PostgreSQL table on the first successful dispatch, so a frontend restart or rolling upgrade does not require a fresh inference request to repopulate state before the reconciler can scale up replacement replicas.

All fields are optional and composable:

• Node selector only: pin model to matching nodes, single replica
• Replicas only: auto-scale across all nodes
• Both: auto-scale on matching nodes only

Declarative per-model scheduling (unattended installs)

In distributed mode you can declare per-model scheduling at startup, instead of using the WebUI/API. Config is authoritative: it is re-applied on every boot and overwrites the listed models (models not listed are left untouched).

Entry fields: model_name (required), node_selector (a label map; omit it to match every node), and then one of two replica modes (they are mutually exclusive):

• replicas: all - static spread: place exactly one replica on every matching node, proactively, regardless of load, and keep it in sync as nodes join and leave. Use this for “run model X everywhere (with this label)”.
• min_replicas / max_replicas - elastic auto-scaling: keep at least min_replicas running, and burst up to max_replicas only when all replicas are busy, scaling back down to the minimum when idle. max_replicas: 0 means no upper bound (grow to cluster capacity). To enable this mode you must set min_replicas >= 1 or max_replicas >= 1 - an entry with only max_replicas: 0 (and no replicas: all) does nothing.

Net effect at a glance:

node_selector constrains which nodes a model may use; with no selector the model may use all healthy nodes. So “spread model X across all nodes” is just replicas: all with no node_selector. replicas: all targets one replica per matching node; with the default per-node cap of one replica per model this lands exactly one on each node (see the note below about LOCALAI_MAX_REPLICAS_PER_MODEL).

YAML example (scheduling.yaml):

# One replica on every GPU-labelled node (static spread, tracks join/leave):
- model_name: gpt-oss
  node_selector:
    tier: gpu
  replicas: all

# One replica on EVERY node in the cluster (no selector = all nodes):
- model_name: embeddings
  replicas: all

# Elastic on CPU nodes: always >=1, burst to capacity under load, 0 = no cap:
- model_name: whisper
  node_selector:
    tier: cpu
  min_replicas: 1
  max_replicas: 0

LOCALAI_DISTRIBUTED=true \
LOCALAI_MODEL_SCHEDULING_CONFIG=/etc/localai/scheduling.yaml \
local-ai run

Inline equivalent:

LOCALAI_MODEL_SCHEDULING='[{"model_name":"gpt-oss","node_selector":{"tier":"gpu"},"replicas":"all"}]'

Notes:

• Because the config is authoritative, each listed model’s entire scheduling row is replaced on every boot, including the optional prefix-cache routing overrides (route_policy, balance_abs_threshold, balance_rel_threshold, min_prefix_match). For a model you manage via this config, set those fields here too if you need non-default values; values set only through the API are reset on the next restart. Models not listed in the config are never touched.
• replicas: all places one replica per matching node by relying on the default per-node cap of one replica per model. If you raise LOCALAI_MAX_REPLICAS_PER_MODEL on a worker above 1, the target count can be met by stacking replicas on fewer nodes rather than spreading one to each.

Label Management API

Scheduling API

Comparison with P2P

Troubleshooting

Worker not registering:

• Verify the frontend URL is reachable from the worker (curl http://frontend:8080/api/node/register)
• Check that --registration-token matches on both frontend and worker
• Ensure auth is enabled on the frontend (LOCALAI_AUTH=true)

NATS connection errors:

• Confirm NATS is running and reachable (nats-server --signal ldm or check port 4222)
• Check that --nats-url uses the correct hostname/IP from the worker’s network perspective

PostgreSQL connection errors:

• Verify the connection URL format: postgresql://user:password@host:5432/dbname?sslmode=disable
• Ensure the database exists and the user has CREATE TABLE permissions (for auto-migration)
• Check that pgvector extension is installed if using RAG features

Node shows as unhealthy or offline:

• The HealthMonitor marks nodes offline when heartbeats are missed. Check network connectivity between worker and frontend.
• Verify --heartbeat-interval is not set too high
• Offline nodes automatically restore to healthy when they re-register (no re-approval needed)

Backend not installing:

• Check the worker logs for backend.install events

Port conflicts on workers:

• Each model gets its own gRPC process on an incrementing port (50051, 50052, …)
• The HTTP file transfer server runs on the base port - 1 (default: 50050)
• Ensure the port range is not blocked by firewalls or used by other services
• Verify the backend gallery configuration is correct
• The worker needs network access to download backends from the gallery

Roadmap: Routing and Caching Enhancements

The scheduling algorithm above is load-based (least in-flight, then least-recently-used). Work is underway to make routing prefix-cache-aware: bias each request toward the replica that already holds the relevant KV/prefix cache (multi-turn conversations and shared system prompts), so backends reuse cache instead of recomputing it. The first step is a router-side radix tree of prompt-prefix hashes mapped to nodes, with longest-prefix match, a load guard that preserves round-robin behavior under imbalance, and NATS sync across frontends. It is purely a routing-layer hint (no backend changes) and never routes worse than today’s round-robin.

Further enhancements, surfaced from a survey of SGLang, vLLM production-stack, Ray Serve, llm-d, AIBrix, and NVIDIA Dynamo, are tracked under the routing roadmap epic (#10063):

• Reported/precise KV-event mode (#10064): subscribe to actual backend KV-cache events for exact residency instead of inferring it from routing history.
• Multi-tier cache-overlap scoring (#10065): credit GPU/CPU/disk cache tiers separately.
• Pluggable scorer/filter/picker pipeline (#10066): composable multi-signal routing (cache, queue depth, KV utilization, latency).
• Load-shaping (#10067): anti-herding (softmax/temperature) and dispatch-time freshness.
• Prefill/decode disaggregation routing (#10068): route prefill and decode to separate pools with KV transfer.
• Per-user fairness (VTC) (#10069): balance per-user token usage against pod load.
• Minor tuning + MCP parity (#10070): per-model TTL override, probabilistic LRU updates, and MCP scheduling-config tool parity.