Operate overview

/app/operate is the front door to the Operate console. It answers one question — is anything wrong — without you having to open four other pages.

The page opens with one sentence: Everything is running., 2 things need you. or, when nothing needs a decision but a row has something to read, Running, with 1 thing to look at. On a new installation, with no backend, no model and nothing loaded, it says Nothing is running yet. and offers Install a backend and Browse models.

Under the sentence are four rows. A row with a problem opens by itself and holds the button that deals with it. A row with nothing to say stays one line. Click a row to open or close it.

Needs you

Only things that want a decision:

• a backend with an update available, with an Update button
• an operation that failed, with Retry and Dismiss
• a node that is not answering, with a link to the nodes

Update starts the same update as the Backends page. Retry installs the failed model or backend again, after moving the failure into the Activity record, and Dismiss moves it there without retrying. When nothing needs you the row says so in one line. There is no green panel: a status page that shouts when everything is fine teaches you to stop reading it.

The attention count on the Status tab is the number of items in this row.

Capacity

One line says how full GPU memory is (system memory on a host without a GPU), and how much room the models disk has left. On a cluster it adds the memory of the nodes that are answering, and the row lists each node’s memory when opened.

The row opens by itself when memory is 90% full or more, or when the models disk has less than a tenth free or less than 20 GB. It then lists the models loaded on this machine, each with an Unload button. Unloading asks first, stops the backend process of that model and frees its memory. The next request that uses the model loads it again.

Under the rows, a bar shows the memory in use now. LocalAI keeps no memory history, so the chart under the bar is drawn from readings the page took itself, one with each Operate summary poll (every 15 seconds), while Operate is open. It starts with the second reading, keeps at most 240 readings (about an hour), and is labelled “Since you opened Operate”. The axis starts at zero, the capacity is a labelled line, and a data table sits behind the chart. Leaving Operate empties it. Temperature and power are not shown because the resources endpoint does not report them.

Running now

Operations in progress, and the models loaded on this machine. On a single-node install the row lists the heaviest five models: backend, resident memory, CPU share and uptime, with View logs and Stop model… in the row menu. Open this machine leads to the full list. With distributed mode on, models run on workers rather than on the controller, so the row links to the Swarm page instead.

Recent failures

Requests and failed requests over the last 24 hours, and p95 latency, counted server-side by GET /api/traces/summary:

curl http://localhost:8080/api/traces/summary?hours=24 \
  -H "Authorization: Bearer <admin-key>"

{
  "total": 18402,
  "errors": 37,
  "p95_ms": 842,
  "window_hours": 24,
  "buckets": [{ "start": "2026-08-02T09:00:00Z", "count": 1520, "errors": 3 }]
}

hours defaults to 24 and is capped at 168. Only 5xx responses and transport errors count as failures — a 4xx is the caller getting it wrong, not the installation being unhealthy. p95_ms is a nearest-rank percentile, not the slowest request. The endpoint exists so a dashboard wanting three numbers does not fetch the whole trace list to count it. An installation that has recorded nothing says so rather than showing zeroes dressed as telemetry. The row opens when at least one request failed, and links to Traces and Usage.

The page does not report models that failed to load, because LocalAI records no load failures to read.

This machine

On a single-node install, Operate → This machine (/app/nodes) shows the host and everything loaded on it:

• GPU memory as one bar, with what is free and a note that the next model must fit in that, or a loaded model has to be stopped first. With several GPUs each one gets its own line. A host without a GPU says so instead of drawing an empty bar.
• Models in host memory: one bar split by running model, so you can see which model is holding memory. LocalAI reports the resident memory of each backend process, not the GPU memory of each model, so the split is of host RAM and no model is shown with a GPU size.
• The machine: VRAM, RAM, CPU and the models disk, each with a bar, a percentage and what is left. A reading the host does not report is shown as “No data”, never as zero.
• Running models: search, sort by memory, CPU or uptime, open a model’s logs, or stop it. Stopping asks for confirmation; the model loads again on its next request.

The page polls GET /system and GET /api/resources every five seconds. The per-model readings come from the process block of GET /system; the host CPU and disk readings come from the cpu and disk fields of GET /api/resources.

Add a machine reveals the command to start LocalAI in distributed mode, and a link to the steps for adding a worker. Once distributed mode is on, the same route becomes the Nodes page and the This machine tab gives way to the Swarm tab.

Models and backends no longer live under a nested Host page. Use Models → Installed for model runtime and configuration actions, and Operate → Backends → Installed for installed backend actions. The overview links into the canonical Operate sections rather than duplicating those inventories.

Old /app/manage bookmarks remain supported. They redirect with replace semantics to the matching Installed Models or Installed Backends view while preserving legacy search, filter, selection, variant, and development flags.

The tab bar

Operate has one row of tabs above the page: Status (this overview), This machine, Swarm (only with distributed mode on), Runtime, Traffic and Settings. A tab that holds several pages shows a second row of links under the bar: Runtime holds Backends, Activity, Logs and (on a single install) Failover; Traffic holds Usage, Traces and Middleware; Settings holds Settings and Users (with authentication on); Swarm holds Nodes, Placement rules, Failover and P2P. Adding a node opens from the Nodes page. Every page keeps its own URL, and a page such as a node detail keeps its tab highlighted. On a phone the bar scrolls sideways. The API link at the end of the bar opens the API documentation.

Several tabs carry a live value: pending backend updates and running operations on Runtime, the healthy node count on Swarm, running models on This machine, the attention count on Status and the error count on Traffic. Host capacity lives on the overview instead of appearing as a separate destination.

Those values are orientation, not an alarm. The bar only exists on Operate routes, so anything urgent also appears in the Needs you row and on the operations badge attached to the sidebar entry, which is always visible.

Authentication & Authorization

LocalAI supports two authentication modes: legacy API key authentication (simple shared keys) and a full user authentication system with roles, sessions, OAuth, and per-user usage tracking.

Legacy API Key Authentication

The simplest way to protect your LocalAI instance is with API keys. Set one or more keys via environment variable or CLI flag:

# Single key
LOCALAI_API_KEY=sk-my-secret-key localai run

# Multiple keys (comma-separated)
LOCALAI_API_KEY=key1,key2,key3 localai run

Clients provide the key via any of these methods:

• Authorization: Bearer <key> header
• x-api-key: <key> header
• xi-api-key: <key> header
• token cookie

Legacy API keys grant full admin access - there is no role separation. For multi-user deployments with role-based access, use the user authentication system instead.

API keys can also be managed at runtime through the Runtime Settings interface.

Protected and anonymous HTTP routes

When you configure database authentication or legacy API keys, LocalAI makes the HTTP surface private by default. An anonymous request succeeds only for an approved method and path or an explicit deployment override. If you configure neither authentication mode, the authentication middleware does not restrict requests.

Public discovery APIs

The following discovery requests are available anonymously:

• GET /.well-known/localai.json
• GET /api/instructions
• GET /api/instructions/{name}
• Swagger GET requests at /swagger and under /swagger/

These discovery APIs describe the server’s API surface. The endpoints that they advertise still require credentials unless this section lists them as public or bootstrap routes.

Anonymous bootstrap routes

LocalAI also permits the requests needed for health checks, credential acquisition, and the login UI. These routes do not make the rest of the API public.

• Health checks: GET /healthz and GET /readyz.
• Authentication status and token login: GET /api/auth/status and POST /api/auth/token-login.
• Local registration and login: POST /api/auth/register and POST /api/auth/login.
• GitHub OAuth: GET /api/auth/github/login and GET /api/auth/github/callback.
• OIDC: GET /api/auth/oidc/login and GET /api/auth/oidc/callback.
• CORS preflight requests: OPTIONS on every path. A cross-origin preflight cannot carry credentials by HTTP spec, so these requests are never gated on auth; the CORS middleware answers them, which grants no API access.
• SPA shell routes: GET /, HEAD /, and GET requests at /app, /browse, /login, /invite/*, and /explorer. Subpaths under /app/ and /browse/ are also available through GET.
• SPA assets: GET /favicon.svg and GET requests under /assets/, /locales/, and /static/.
• Branding reads: GET /api/branding and GET requests under /branding/asset/. Branding mutations still require admin credentials.

Routes that require credentials

Without an explicit deployment override, every other route requires credentials. This includes GET /version, all model and backend API routes, and all inference routes. MCP and moderation aliases are also private:

• POST /v1/mcp/chat/completions, POST /mcp/v1/chat/completions, and POST /mcp/chat/completions
• POST /v1/moderations and POST /moderations

Generated output URLs also require credentials. This applies to every URL under /generated-audio/, /generated-images/, /generated-videos/, and /generated-3d/.

Explicit deployment overrides

Embedded deployments can add path prefixes to ApplicationConfig.PathWithoutAuth. Each prefix bypasses global authentication for every HTTP method below that prefix. Route-specific authorization still applies when the route registers it. Keep these overrides as narrow as possible; the default list is empty.

Two legacy flags provide a separate GET-only compatibility override when legacy API keys are configured:

• LOCALAI_DISABLE_API_KEY_REQUIREMENT_FOR_HTTP_GET=true enables the override.
• LOCALAI_HTTP_GET_EXEMPTED_ENDPOINTS sets the regular expressions for the exempt GET routes.

The endpoint expressions have no effect unless you enable LOCALAI_DISABLE_API_KEY_REQUIREMENT_FOR_HTTP_GET. Review custom expressions carefully because they can expose protected reads.

User Authentication System

The user authentication system provides:

• User accounts with email, name, and avatar
• Role-based access control (admin vs. user)
• Session-based authentication with secure cookies
• OAuth login (GitHub) and OIDC single sign-on (Keycloak, Google, Okta, Authentik, etc.)
• Per-user API keys for programmatic access
• Client keys for paired clients: a one-time pairing code gives a client a key that reaches only the routes of its feature list. A key can use the standard inference API for a list of models that the owner chose (client_inference), call the add-ons that the owner chose (client_addons), or both. See Paired clients: inference and add-ons
• Admin route gating - management endpoints are restricted to admins
• Per-user usage tracking with token consumption metrics

Enabling Authentication

Set LOCALAI_AUTH=true or provide a GitHub OAuth Client ID or OIDC Client ID (which auto-enables auth):

# Enable with SQLite (default, stored at {DataPath}/database.db)
LOCALAI_AUTH=true localai run

# Enable with GitHub OAuth
GITHUB_CLIENT_ID=your-client-id \
GITHUB_CLIENT_SECRET=your-client-secret \
LOCALAI_BASE_URL=http://localhost:8080 \
localai run

# Enable with OIDC provider (e.g. Keycloak)
LOCALAI_OIDC_ISSUER=https://keycloak.example.com/realms/myrealm \
LOCALAI_OIDC_CLIENT_ID=your-client-id \
LOCALAI_OIDC_CLIENT_SECRET=your-client-secret \
LOCALAI_BASE_URL=http://localhost:8080 \
localai run

# Enable with PostgreSQL
LOCALAI_AUTH=true \
LOCALAI_AUTH_DATABASE_URL=postgres://user:pass@host/dbname \
localai run

Enabling auth on an existing instance

Enabling user authentication does not assign existing data to an account. With authentication off, everything the chats made (chats, saved agents, triggers, connector bindings, MCP connectors, notifications and the files of the workspace) belongs to one anonymous owner, and that owner is not an account. Authenticated users, admins included, get separate storage, so the anonymous data stays in the database or on disk but no account can see it, in the API or the Web UI.

What happens to each part when you switch authentication on:

• Chats, saved agents, notifications and files stay under the anonymous owner. No user sees them. Copy what you need into the new account first (see the steps below).
• Triggers and connector bindings of the anonymous owner stop. The owner no longer exists as a user, so the trigger runner skips the next run and switches each trigger off with the reason “its owner no longer exists”, and a connector bot answers “not available” and is switched off.
• Collections made without a user belong to the root storage area. With authentication on, each user has their own collections (see Collections).
• Old agents, tasks, skills, Telegram connectors and saved chats of the previous agent pool are converted at start and given to the account that owned them (see Chats migration). With authentication on, an old item that belongs to no user is not converted: the report at /app/chats/conversion lists it as skipped, and you make it again under the intended account.

Before switching an existing instance:

1. Switch off the triggers and Telegram bindings you want to keep, while auth is still off.
2. Export what you want to keep. Export each chat as JSON (GET /api/chats/{id}/export?format=json) and each saved agent (GET /api/chats/agents/{id}/export). Recreate triggers and bindings by hand.
3. Stop LocalAI and back up its data path and any separately configured collection storage or databases.
4. Test the switch on a copy before changing the production instance.
5. Restrict access during bootstrap and register the intended administrator first.
6. Import the chats and agents under the intended user (POST /api/chats/import, POST /api/chats/agents/import). Recreate collections there as needed.
7. Grant the new accounts the features they need. Saved agents, triggers, connectors and agent jobs are off by default for a user (see Chats permissions); an administrator has all of them.
8. Open the conversion report and check each item, and verify credentials and required data, before resuming triggers and bots.

The first registered user becomes admin and bypasses approval or invite requirements. Setting LOCALAI_ADMIN_EMAIL promotes that email address, but does not reserve the first-admin role for it. Keep registration inaccessible to other users until bootstrap is complete.

An export holds the transcript and the settings of a chat, and the definition of an agent. It does not hold files, trigger state, bot tokens or the long-term state of the model. Keep the backup: LocalAI has no automatic migration of the anonymous owner’s data.

A trigger or a Telegram bot acts as its owner, with the owner’s rights: the agent gets only what the owner may use at that moment, under the owner’s quota and model allowlist. It does not use an API key of its own. An administrator who switches the chats feature off for a user, or the triggers or connectors feature, also stops what that user’s triggers and bots do, at their next run or message; the trigger and the binding stay, and work again when the feature is given back.

Configuration Reference

Note: network-backed storage. File-based SQLite relies on POSIX file locking, which is unreliable over network filesystems (SMB/CIFS/NFS, e.g. Azure Files / Azure Container Apps shared volumes). On such storage the auth DB can fail to migrate with database is locked. Use PostgreSQL (LOCALAI_AUTH_DATABASE_URL=postgres://...) when the data directory lives on shared or network storage, or place database.db on a local volume.

SQLite storage settings

One SQLite file holds the accounts and everything else that LocalAI keeps in the auth database (the chats, their files, the triggers and the notifications). Many parts of the server write to it at the same time, so LocalAI opens it with these settings:

WAL adds two files beside the database, database.db-wal and database.db-shm. Copy or back up all three together, or run PRAGMA wal_checkpoint(TRUNCATE) first.

You can change a setting in the database URL. A value that you give always wins, for example LOCALAI_AUTH_DATABASE_URL="/data/database.db?_pragma=journal_mode(DELETE)&_pragma=busy_timeout(30000)". Use journal_mode(DELETE) on a volume where WAL does not work (see the note above on network storage).

One process only. SQLite coordinates writers through file locks on one host. Run one LocalAI process per database file. Two servers on the same file, or on a file that is on a network share, are not supported: use PostgreSQL when more than one instance must share the accounts. The pool and the timeouts bound the waiting inside one process; they do not make a shared file safe.

A part of the code that holds a cursor open (rows that it reads one by one) and also writes needs a second connection for the write. Keep the cursors short, or read the rows into memory first, so the 16 connections cannot all be taken by cursors.

Disabling Local Authentication

If you want to enforce OAuth/OIDC-only login and prevent users from registering or logging in with email/password, set LOCALAI_DISABLE_LOCAL_AUTH=true (or pass --disable-local-auth):

# OAuth-only setup (no email/password)
LOCALAI_DISABLE_LOCAL_AUTH=true \
GITHUB_CLIENT_ID=your-client-id \
GITHUB_CLIENT_SECRET=your-client-secret \
LOCALAI_BASE_URL=http://localhost:8080 \
localai run

When disabled:

• The login page will not show email/password forms (the UI checks the providers list from /api/auth/status)
• POST /api/auth/register returns 403 Forbidden
• POST /api/auth/login returns 403 Forbidden
• OAuth/OIDC login continues to work normally

Roles

There are two roles:

• Admin: Full access to all endpoints, including model management, backend configuration, system settings, traces, the saved agents, triggers and connectors of every user (read-only, with switch-off and delete), and user management.
• User: Access to inference endpoints only - chat completions, embeddings, image/video/audio generation, TTS, MCP chat, and their own usage statistics.

The first user to sign in is automatically assigned the admin role. Additional users can be promoted to admin via the admin user management API or by setting LOCALAI_ADMIN_EMAIL to their email address.

Registration Modes

Invite Links

Admins can generate single-use, time-limited invite links from Settings → Users and keys → Invites in the web UI (choose 1, 7 or 30 days), or via the API:

# Create an invite link (default: expires in 7 days)
curl -X POST http://localhost:8080/api/auth/admin/invites \
  -H "Authorization: Bearer <admin-key>" \
  -H "Content-Type: application/json" \
  -d '{"expiresInHours": 168}'

# List all invites
curl http://localhost:8080/api/auth/admin/invites \
  -H "Authorization: Bearer <admin-key>"

# Revoke an unused invite
curl -X DELETE http://localhost:8080/api/auth/admin/invites/<invite-id> \
  -H "Authorization: Bearer <admin-key>"

Share the invite URL (/invite/<code>) with the user. The link is shown once, when you create it: the invite list keeps only the first characters of the code, so a link you did not copy can only be revoked and made again. When the user opens it, the registration form is pre-filled with the invite code. LocalAI validates the code only when the user submits registration. Invite codes are single-use - once consumed, they cannot be reused. Expired or used invites are rejected.

For GitHub OAuth, the invite code is passed as a query parameter to the login URL (/api/auth/github/login?invite_code=<code>) and stored in a cookie during the OAuth flow.

Admin-Only Endpoints

When authentication is enabled, the following endpoints require admin role:

Model & Backend Management:

• GET /api/models, POST /api/models/install/*, POST /api/models/delete/*
• GET /api/backends, POST /api/backends/install/*, POST /api/backends/delete/*
• GET /api/operations, POST /api/operations/*/cancel, POST /api/operations/*/pause, POST /api/operations/*/dismiss
• GET /api/operations/history, DELETE /api/operations/history
• GET /models/available, GET /models/galleries, GET /models/jobs/*
• GET /backends, GET /backends/available, GET /backends/galleries

System & Monitoring:

• GET /api/traces, GET /api/traces/summary, GET /api/traces/{id}, POST /api/traces/clear
• GET /api/backend-traces, GET /api/backend-traces/{id}, POST /api/backend-traces/clear
• GET /api/backend-logs/*, POST /api/backend-logs/*/clear
• GET /api/resources, GET /api/settings, POST /api/settings
• GET /system, GET /backend/monitor, POST /backend/shutdown, POST /backend/load
• POST /v1/slots/save, POST /v1/slots/restore (write and read prompt-cache files on the host)

P2P:

• GET /api/p2p/*

User-Accessible Endpoints (all authenticated users):

• POST /v1/chat/completions, POST /v1/embeddings, POST /v1/completions
• POST /v1/images/generations, POST /v1/audio/*, POST /tts, POST /vad, POST /video
• GET /v1/models, POST /v1/tokenize, POST /v1/detokenize, POST /v1/detection
• POST /v1/mcp/chat/completions, POST /v1/messages, POST /v1/responses
• POST /stores/*, GET /api/cors-proxy
• GET /version, GET /api/features, GET /metrics
• GET /api/auth/usage (own usage data)

Web UI Access Control

When auth is enabled, the React UI sidebar dynamically shows/hides sections based on the user’s role:

• All users see: Home, Chat, Images, Video, TTS, Sound, Talk, Usage, API docs link
• Admins also see: Models, the Build console (Training, Recognition and the other tools that need an administrator), and the Operate console (Backends, Activity, Nodes, Traffic, Settings, and Users and keys)

Admin-only pages are also protected at the router level - navigating directly to an admin URL redirects non-admin users to the home page.

On Home, the memory line with the list of loaded models is shown to admins only, because it reads the admin endpoints GET /system and GET /api/resources. Other users do not see the line, rather than a “No models loaded” that may not be true.

In Chat, a non-admin user holding the mcp feature can select the MCP servers of a model whose configuration declares them (MCP → Servers, plus that model’s MCP prompts and resources). Users without the mcp feature only see client-side MCP. The model information panel and the automatic context size stay admin-only, because they read the full model configuration.

Sign-in, invite and account pages

The sign-in screen offers only what GET /api/auth/status reports:

• A GitHub or SSO button appears only when that provider is configured. With LOCALAI_DISABLE_LOCAL_AUTH=true there is no email form.
• Email and password sign-in asks for one field at a time: the email, then the password. Both are sent in one call, so the server still never says whether an email exists.
• Registration asks for the email, then the name, password and (when the registration mode allows one) an invite code. A sign-up that waits for approval returns to the sign-in screen with a notice that an admin must approve it.
• On a server with no users yet, the first screen is “Create the admin account”.
• On a server protected only by legacy API keys, the screen asks for the key and nothing else.
• An invite link (/invite/<code>) opens registration with the code filled in and read-only.

The Account page has four tabs: Profile (name and picture address), Security (change a local password; for a GitHub or SSO account it says the password is managed by the provider), API keys, and Usage (your requests, tokens by model, and the limits an admin set on you, for the last 30 days). Settings → Users and keys has the people, the invites and your own API keys.

GitHub OAuth Setup

1. Create a GitHub OAuth App at Settings → Developer settings → OAuth Apps → New OAuth App
2. Set the Authorization callback URL to {LOCALAI_BASE_URL}/api/auth/github/callback
3. Set GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET environment variables
4. Set LOCALAI_BASE_URL to your publicly-accessible URL

OIDC Setup

Any OIDC-compliant identity provider can be used for single sign-on. This includes Keycloak, Google, Okta, Authentik, Azure AD, and many others.

Steps:

1. Create a client/application in your OIDC provider
2. Set the redirect URL to {LOCALAI_BASE_URL}/api/auth/oidc/callback
3. Set the three environment variables: LOCALAI_OIDC_ISSUER, LOCALAI_OIDC_CLIENT_ID, LOCALAI_OIDC_CLIENT_SECRET

LocalAI uses OIDC auto-discovery (the /.well-known/openid-configuration endpoint) and requests the standard scopes: openid, profile, email.

Provider examples:

# Keycloak
LOCALAI_OIDC_ISSUER=https://keycloak.example.com/realms/myrealm

# Google
LOCALAI_OIDC_ISSUER=https://accounts.google.com

# Authentik
LOCALAI_OIDC_ISSUER=https://authentik.example.com/application/o/localai/

# Okta
LOCALAI_OIDC_ISSUER=https://your-org.okta.com

For OIDC, invite codes work the same way as GitHub OAuth - the invite code is passed as a query parameter to the login URL (/api/auth/oidc/login?invite_code=<code>) and stored in a cookie during the OAuth flow.

User API Keys

Authenticated users can create personal API keys for programmatic access:

# Create an API key (requires session auth)
curl -X POST http://localhost:8080/api/auth/api-keys \
  -H "Cookie: session=<session-id>" \
  -H "Content-Type: application/json" \
  -d '{"name": "My Script Key"}'

User API keys inherit the creating user’s role. Admin keys grant admin access; user keys grant user-level access.

You can pause a key without deleting it, and resume it later. Only the key’s owner can pause it. A paused key is rejected like an invalid key until it is resumed. Pause it until you resume it, or until a time in the future, after which it works again by itself:

# Pause until resumed
curl -X PATCH http://localhost:8080/api/auth/api-keys/<key-id> \
  -H "Cookie: session=<session-id>" \
  -H "Content-Type: application/json" \
  -d '{"disabled": true, "paused_until": null}'

# Pause until a given time (RFC 3339, must be in the future)
curl -X PATCH http://localhost:8080/api/auth/api-keys/<key-id> \
  -H "Cookie: session=<session-id>" \
  -H "Content-Type: application/json" \
  -d '{"disabled": false, "paused_until": "2030-01-01T00:00:00Z"}'

# Resume
curl -X PATCH http://localhost:8080/api/auth/api-keys/<key-id> \
  -H "Cookie: session=<session-id>" \
  -H "Content-Type: application/json" \
  -d '{"disabled": false, "paused_until": null}'

The key list returns disabled and, when set, pausedUntil for each key. The Account page in the web UI has a Pause and Resume button for each key.

A key can be created with a lifetime: send expiresIn (30d, 90d or 1y) or expiresAt (RFC 3339). Without either, the server applies LOCALAI_DEFAULT_API_KEY_EXPIRY when it is set, and the key does not expire otherwise. The full key is returned once, in the response to the create call; the list returns only its prefix, the creation time, lastUsed and expiresAt.

In the web UI, the API keys tab of the Account page and of Settings → Users and keys shows the signed-in person’s own keys: a new key is shown once with a Copy button, and the lifetime list sends expiresIn. Revoking a key waits ten seconds in the browser before it sends the DELETE, and Undo inside that time sends nothing; leaving the page sends a revoke that is still waiting. LocalAI lists a person’s keys only to that person, so an admin cannot list or revoke other users’ keys, in the UI or through the API. The Usage page can group requests by API key for everyone.

Auth API Endpoints

Usage Tracking

When authentication is enabled, LocalAI automatically tracks per-user token usage for inference endpoints. Usage data includes:

• Prompt tokens, completion tokens, and total tokens per request
• Model used and endpoint called
• Request duration
• Timestamp for time-series aggregation

Viewing Usage

Usage is accessible through the Usage page in the web UI (visible to all authenticated users) or via the API:

# Get your own usage (default: last 30 days)
curl http://localhost:8080/api/auth/usage?period=month \
  -H "Authorization: Bearer <key>"

# Admin: get all users' usage
curl http://localhost:8080/api/auth/admin/usage?period=week \
  -H "Authorization: Bearer <admin-key>"

# Admin: filter by specific user
curl "http://localhost:8080/api/auth/admin/usage?period=month&user_id=<user-id>" \
  -H "Authorization: Bearer <admin-key>"

Period values:

• day - last 24 hours, bucketed by hour
• week - last 7 days, bucketed by day
• month - last 30 days, bucketed by day (default)
• all - all time, bucketed by month

Response format:

{
  "usage": [
    {
      "bucket": "2026-03-18",
      "model": "gpt-4",
      "user_id": "abc-123",
      "user_name": "Alice",
      "prompt_tokens": 1500,
      "completion_tokens": 800,
      "total_tokens": 2300,
      "request_count": 12
    }
  ],
  "totals": {
    "prompt_tokens": 1500,
    "completion_tokens": 800,
    "total_tokens": 2300,
    "request_count": 12
  }
}

Usage Dashboard

The web UI Usage page provides:

• Period selector - switch between day, week, month, and all-time views
• Summary cards - total requests, prompt tokens, completion tokens, total tokens
• By Model table - per-model breakdown with visual usage bars
• By User table (admin only) - per-user breakdown across all models
• Sources tab - per-API-key and per-source breakdown (described below)

Per-API-key Breakdown

The Sources tab on the Usage page surfaces a third dimension of the same data: traffic broken down by API key and by request source. Three source classes are tracked:

• API key - request authenticated with a named user API key (Authorization: Bearer lai-..., x-api-key, or token cookie). Each key shows up with its label (snapshotted at write time, so revoked keys still display the original name).
• Web UI - request authenticated with a browser session cookie.
• Legacy - request authenticated with an env-configured LOCALAI_API_KEY. Visible to admins only.

The Sources tab is visible to every authenticated user. Non-admins see only their own keys plus their own Web UI traffic (legacy is filtered server-side). Admins see every key from every user.

The tab is laid out as:

• A source mix ribbon showing the percentage split across the three classes.
• A top-N + Other stacked time chart (top 7 sources by total tokens; the rest roll up).
• A searchable, sortable table of every key plus the Web UI and Legacy pseudo-rows. Click a row to filter the chart to that source.

Endpoints

Both endpoints accept the same period parameter (day, week, month, all) as /api/auth/usage.

# Your own per-source usage for the last week
curl "http://localhost:8080/api/auth/usage/sources?period=week" \
  -H "Authorization: Bearer <key>"

# Admin: filter to a single API key across all users
curl "http://localhost:8080/api/auth/admin/usage/sources?period=month&api_key_id=<key-id>" \
  -H "Authorization: Bearer <admin-key>"

Response shape:

{
  "buckets": [
    { "bucket": "2026-05-19", "source": "apikey",
      "api_key_id": "uuid", "api_key_name": "ci-runner",
      "total_tokens": 20000, "request_count": 142, "...": "..." },
    { "bucket": "2026-05-19", "source": "web",
      "total_tokens": 300, "request_count": 11, "...": "..." }
  ],
  "totals": {
    "by_source": {
      "apikey": { "tokens": 1234567, "requests": 8420 },
      "web":    { "tokens":   92000, "requests":   211 }
    },
    "by_key": [
      { "api_key_id": "uuid", "api_key_name": "ci-runner",
        "tokens": 2100000, "requests": 8420,
        "last_used": "2026-05-20T12:34:56Z" }
    ],
    "grand_total": { "tokens": 1334777, "requests": 8645 }
  },
  "truncated": false
}

The by_key list is server-sorted by tokens descending and capped at 200 entries. When more keys would qualify, the response sets "truncated": true so the UI can show a notice.

Migration of pre-feature data

Usage rows recorded before this feature have no source column. On startup, InitDB backfills them as legacy when the synthetic legacy-api-key user_id was used, and web for everything else. The migration is idempotent; existing aggregations remain correct after the upgrade.

Combining Auth Modes

Legacy API keys and user authentication can be used simultaneously. When both are configured:

1. User sessions and user API keys are checked first
2. Legacy API keys are checked as fallback - they grant admin-level access
3. This allows a gradual migration from shared API keys to per-user accounts

Build Requirements

User authentication needs no build tag and no C compiler. The SQLite database is opened with a pure-Go driver, so a binary built with CGO_ENABLED=0 and no tags (the release binaries) can keep accounts in a SQLite file, and the same driver serves the chats and their files. PostgreSQL is chosen by the postgres:// form of the database URL, as before.

go build ./cmd/local-ai

The auth build tag is only left on some tests. The pure-Go driver reads and writes the file format of SQLite, so an existing database.db opens as it is.

Runtime Settings

LocalAI provides a web-based interface for managing application settings at runtime. These settings can be configured through the web UI and are automatically persisted to a configuration file, allowing changes to take effect immediately without requiring a restart.

Accessing Runtime Settings

Open Operate → Settings in the web UI (/app/settings). The page lists every setting below under eight groups by intent, and keeps your edits in a pending bar until you apply them.

Groups

The groups reorganise the fields the page used to show under fifteen sections. The mapping:

Search at the top of the page matches a setting’s name, description and key, the group, and the old section name, so a search for “Watchdog” still finds the watchdog settings and each result says where it used to be.

/app/settings?group=<id> opens a group (memory, speed, backends, access, debug, agents, tools, swarm, look or addons), and a hash after it brings a block of the group into view: #settings-builtin-tools (the built-in chat tools) on the Tools group. The old address of the command network, /app/settings?group=tools#settings-egress, opens the Settings tab of the add-on that holds it now (see below), or the Tools group when no add-on does.

The settings of an add-on are not on this page: each add-on keeps them on the Settings tab of its details on the Add-ons page. When at least one add-on has settings, an Add-ons group at the end of the list names those add-ons, each with a link to its Settings tab. See Host slots. An add-on can also ask LocalAI to draw some of LocalAI’s own settings on that tab, with a host control. The rule: this page keeps the settings of the features that exist without any add-on (the commands on this machine, the built-in web tools), and points to the add-on where an add-on serves them; the settings that exist only for an add-on are on its tab. The Sandbox network below is one: with the Sandbox installed, the Tools group shows a row that links to its Settings tab, and neither the group nor the search lists those settings. A built-in tool row that an add-on with the tool-providers control serves links to that add-on’s tab.

Editing, applying and undoing

Edits are not sent as you type. A bar at the bottom of the page counts them and offers Discard, Show diff and Apply. The diff lists each old and new value, and runs the checks the browser can make: a duration parses the way the server parses it, a GPU memory budget is one the server accepts, a gallery list is valid JSON with a url in each entry, and warnings repeat what the server says (a restart is needed, an empty P2P token stops P2P, evicting while busy can interrupt requests, CSRF protection off). A check that would make the server refuse the value stops Apply.

Apply sends only the settings that changed. After it, Undo (for ten seconds) saves the previous values again. This is a new save, not a rollback: anything that changed in between stays changed.

A setting shows Changed and its built-in default, with a Reset link, only when the default is known (the defaults of the local-ai run flags). Settings without a stated default, such as threads and the context size, never show the marker. An environment variable or flag can change the value the server starts with, so the marker compares with the built-in default, not with the startup value.

A row says Applies now when the save handler applies the setting immediately (the watchdog and the memory reclaimer restart with the new values; so does the P2P stack when its settings change), and Needs restart for most collections settings. A setting without either note is saved, and the code does not say when it takes effect.

History lists the settings you applied from this browser, newest first, up to the last 50. LocalAI keeps no settings log of its own, so changes made elsewhere do not appear, and a secret (the P2P token, shared API keys, the collections database URL) is listed as changed without its value. Revert puts the old value back as an edit.

POST /api/settings accepts a partial body and merges it over the saved settings, so the page sends only the keys it changes. In the request, the csrf field carries the disable flag: the page shows the inverse as “CSRF protection”. The galleries are sent as galleries and backend_galleries, and the shared API keys as api_keys, each as a JSON list.

Available Settings

Watchdog Settings

The watchdog monitors backend activity and can automatically stop idle or overly busy models to free up resources.

• Watchdog Enabled: Master switch to enable/disable the watchdog
• Watchdog Idle Enabled: Enable stopping backends that are idle longer than the idle timeout
• Watchdog Busy Enabled: Enable stopping backends that are busy longer than the busy timeout
• Watchdog Idle Timeout: Duration threshold for idle backends (default: 15m)
• Watchdog Busy Timeout: Duration threshold for busy backends (default: 5m)

Changes to watchdog settings are applied immediately by restarting the watchdog service.

Backend Configuration

• Max Active Backends: Maximum number of active backends (loaded models). When exceeded, the least recently used model is automatically evicted. Set to 0 for unlimited, 1 for single-backend mode
• Force Eviction When Busy: Allow evicting models even when they have active API calls (default: disabled for safety). Warning: Enabling this can interrupt active requests
• LRU Eviction Max Retries: Maximum number of retries when waiting for busy models to become idle before eviction (default: 30)
• LRU Eviction Retry Interval: Interval between retries when waiting for busy models (default: 1s)

Note: The “Single Backend” setting is deprecated. Use “Max Active Backends” set to 1 for single-backend behavior.

LRU Eviction Behavior

By default, LocalAI will skip evicting models that have active API calls to prevent interrupting ongoing requests. When all models are busy and eviction is needed:

1. The system will wait for models to become idle
2. It will retry eviction up to the configured maximum number of retries
3. The retry interval determines how long to wait between attempts
4. If all retries are exhausted, the system will proceed (which may cause out-of-memory errors if resources are truly exhausted)

You can configure these settings via the web UI or through environment variables. See VRAM Management for more details.

Performance Settings

• Threads: Number of threads used for parallel computation (recommended: number of physical cores)
• Context Size: Context size for models that set no context_size of their own. A value in a model’s configuration always wins, and so does any value you set here over the automatic size. Unset (the default), a model that can serve chats gets 32768 tokens, capped by what it was trained for and by the memory of the machine, and other models get at most 8192. The chat needs at least 16384: see the context window a chat needs.
• Concurrent Backend Operations: How many backend installs, upgrades and deletes run at once. 1 runs them one after the other (default: 1, minimum 1). Operations on the same backend never overlap, whatever the value. Model operations run in their own queue and are not held up by backend operations. In distributed mode the same limit bounds how many nodes receive one operation at the same time. The limit is per frontend. A change applies to operations that start after the change
• Artifact Download Concurrency: Maximum number of artifact files downloaded at once. 1 downloads sequentially (default: 1)
• F16: Enable GPU acceleration using 16-bit floating point
• VRAM Budget: Cap on VRAM used for model allocation (for example 80% or 12GB; empty means no cap). See VRAM Management

Debug and Logging

• Debug Mode: Enable debug logging (deprecated, use log-level instead)

API Security

• CORS: Enable Cross-Origin Resource Sharing
• CORS Allow Origins: Comma-separated list of allowed CORS origins. When CORS is explicitly enabled with a nonempty allowlist, matching origins are also trusted by CSRF protection for cross-site state-changing requests. Explicit wildcard patterns apply to both policies. The default permissive HTTP CORS policy does not bypass CSRF; authentication and permissions still apply.
• CSRF: Enable CSRF protection middleware. Outside explicitly trusted CORS origins, cross-site browser requests are exempt only when the server successfully authenticates a credential from Authorization, x-api-key, or xi-api-key. Supplying an arbitrary header when authentication is disabled does not bypass protection; cookie authentication also does not grant this exemption. Requests without Sec-Fetch-Site (such as CLI clients) remain allowed by this middleware.
• API Keys: Manage API keys for authentication (one per line or comma-separated)
• Offers from the assistant (chats_suggestions, default on, applies at once): the default for users who did not choose. On, the assistant may offer, at most once a day and once a chat, never in the first turns or during a task, to remember a name or bio, make a shortcut, or update a Home widget on a schedule. A user’s own switch (Account, About you and the assistant, Help and suggestions) wins. See Make LocalAI yours.
• MCP connectors may reach private networks (mcp_connectors_allow_private, default off, applies at once): lets the MCP servers that users add as connectors use addresses on private networks (this machine, link-local, private ranges, cloud metadata). Off, a connector reaches only public hosts, and the check is made when the connection is opened, on every address and every redirect. Turn it on only if your users are trusted: a connector can then probe the internal network through the server.

For multi-user authentication with roles, OAuth, and usage tracking, see Authentication & Authorization.

P2P Settings

Configure peer-to-peer networking for distributed inference:

• P2P Token: Authentication token for P2P network
• P2P Network ID: Network identifier for P2P connections
• Federated Mode: Enable federated mode for P2P network

Changes to P2P settings automatically restart the P2P stack with the new configuration.

Gallery Settings

Manage model and backend galleries:

• Model Galleries: a read-only list of the model library sources, with name, id, address, state, whether it is a default, and its mirrors. Change them in the Manage libraries panel of the Models page (see Gallery mirrors)
• Backend Galleries: the same read-only list for backend library sources; change them in the Manage libraries panel of the Backends page
• Load and pre-warm galleries on boot: Load model galleries and pre-warm their remote size and VRAM estimates when LocalAI starts. Disable this setting to skip both startup operations.
• Autoload Backend Galleries: Automatically load backend galleries on startup

Chats and Collections Settings

Configure the collections (the knowledge base) and the models the chats use for media:

• Collections: Enable or disable the collections service (the Knowledge page)

• Default model for collections: Default LLM model of the collections service

• Chat image model (chats_vision_model, default empty, applies at once): the model the chats use to describe an image (describe_image). Empty, or a model that is not installed: the chat’s own model when it can look at images, else the first installed model, by name, that can.

• Chat transcription model (chats_transcription_model, default empty, applies at once): the model the chats use to transcribe audio. Empty, or a model that is not installed: the first installed model, by name, that can transcribe.

• Embedding Model: Model used for knowledge base embeddings (default: granite-embedding-107m-multilingual)

• Max Chunking Size: Maximum chunk size for document ingestion (default: 400)

• Chunk Overlap: Overlap between document chunks (default: 0)

• Collection DB Path: Custom path for the collections database

• Shared pages (shared_pages_enabled, default on): lets users open the pages of the shared folders they have access to. Each user still allows each page before it runs. It applies at once: off, the pages are listed but do not open, and an open shared page stops at its next action. Start with LOCALAI_DISABLE_SHARED_PAGES=true to have it off from the start.

• Page area on Home (home_page_area_enabled, default on): lets users show one of their pages below the composer on Home (pageArea in home.json). It applies at once: off, Home ignores pageArea and the area says an administrator turned it off. The files and pages are kept. Start with LOCALAI_DISABLE_PAGE_AREA=true to have it off from the start.

Tools

The Tools tab lists the built-in chat tools, from GET /api/chats/admin/tools: bash, web_search, web_fetch, list_agents and the two LocalAI groups (localai_client, localai_admin). An add-on may replace each of the first three. Each row says who serves the tool now (built-in, or an add-on) and has its switch:

• Commands on this machine (chats_host_exec, off, ask or on, applies at once; LOCALAI_CHATS_HOST_EXEC): who gets the chats’ built-in bash when no Sandbox add-on serves bash. It runs commands directly on this machine with no isolation and with its network: they can read what LocalAI can read and reach the internet, your network and the LocalAI API. ask gives it to administrators, and each command waits for their yes in the chat; on gives it to every user with no card. Empty is the default: off with accounts, ask without. Every command is audited. See Commands on this machine.
• Web search (chats_web_search, default on, applies at the next turn; its row in the tab) and Read web pages (chats_web_fetch, default on; its row in the tab): offer the built-in web_search and web_fetch tools to chats. See Web search and web pages in chats.
• LocalAI client tools (chats_localai_client, default on, applies at the next turn; its row in the tab) and LocalAI admin tools (chats_localai_admin, default on; its row in the tab): offer the chats the LocalAI Assistant’s tool groups. The client group runs inference for every user within their permissions; the admin group manages this server and is for administrators, who switch it on per chat or per saved agent. If the deprecated LOCALAI_DISABLE_ASSISTANT variable or --disable-local-ai-assistant flag is set, the admin group stays off and this setting cannot turn it on until you remove them.
• Load LocalAI tools on demand (chats_tool_search, auto, on or off, default auto, applies at the next turn): send the model tool_search, tool_describe and tool_call in place of the two LocalAI tool groups. auto does it when the schemas of the groups reach a tenth of the model’s context window, on always, off never. With on, the tools of the user’s MCP connectors load on demand as well. See Tool search.
• Saved agents in chats (chats_saved_agents, default on, applies at the next turn; its row in the tab): let a chat list your saved agents (list_agents) and start one as a sub-agent. Each chat must still switch it on in its Tools menu, and the user needs the agents feature. See Saved agents as sub-agents.
• Work folder for helper files (chats_work_folder, default on, applies at the next turn): name each chat’s work folder (Chats/<chat>/work/) at every turn, start every shell in it with its path in $WORK, so the scripts, logs and drafts the model makes go there and not next to your files, and add a one-line nudge when a helper file is saved elsewhere. Off: no steering; the Files pane of a chat still shows working files apart. See Working files and the work folder.
• Search inside documents (chats_index_files, default on, applies at once): read the text of PDF, Word, Excel and PowerPoint files when they are uploaded or written, so file search and the chats’ search_files find a document by what it says. Off: nothing new is read, and text read before is still used for files that did not change. See Search inside documents.
• Text kept per document (chats_index_max_text_bytes, default 2097152, 65536 to 16777216, applies at once): the most text the index keeps for one document. Text past it is not searched; read_file in a chat still reads the whole document.
• Extract: most bytes per archive (chats_extract_max_bytes, default 1073741824, 1048576 to 68719476736, applies at once): the most data one archive may expand to when a user chooses Extract here on the Files page. It is counted on the bytes that are really read. An archive over the limit is refused and nothing is written.
• Extract: most entries per archive (chats_extract_max_entries, default 20000, 1 to 1000000, applies at once): the most files and folders one archive may have. An archive over the limit is refused and nothing is written.
• Extract: allow 7z and RAR (chats_extract_7z_rar, default off, applies at once): let Extract here open 7z and RAR archives. Their readers run in the server and cannot limit their memory, so a crafted file can stop the server. Turn it on only when you trust the users who can upload files. See Extract an archive.
• SearXNG instance (chats_web_searxng_url, default empty): the address of your SearXNG instance. With no provider order set, web_search then uses SearXNG alone. The instance must allow the JSON format.
• Search providers (chats_web_search_providers, default empty): the providers of web_search in the order they are tried, comma separated: searxng, duckduckgo, brave_html, bing_html. Leave one out to never use it. Empty: searxng when an instance is set, else duckduckgo,brave_html,bing_html. An unknown name is refused. Test search on the web_search row says what each provider did (ok, challenged by a bot check, blocked, no results). See Search providers.
• Web pages may be on private networks (chats_web_allow_private, default off): let web_fetch reach addresses inside your network. The cloud metadata addresses stay refused. Turn it on only if you trust every user of chats.
• Ports for web pages (chats_web_ports, default 80,443): the ports web_fetch may use, comma separated. A value that is not a list of ports is refused.
• Command network (applies at the next connection of a command; an allow the proxy cached lives at most 30 seconds). The commands a chat runs in isolation reach the network only through the proxy of the add-on that runs them, and these settings bound it. They are on the Settings tab of the add-on that runs the commands (its network-policy host control), not on this page; chats_egress_audit_days is on both, since it also keeps the audit of the commands on this machine. They do not apply to commands on this machine (chats_host_exec), which have the network of the server. See Network access.
  • chats_egress_ceiling (off, registries or all, default off): the widest preset any chat may use. registries admits the package registries only; all admits every public host. A chat chooses its own preset up to this.
  • chats_egress_default (off, registries or all, default off): the preset a new chat starts with, capped by the ceiling.
  • chats_egress_allow_private (default off): let a host that an administrator allowed, by a rule or a card answer, reach addresses inside your network. A preset alone never does, and the cloud metadata addresses and the server’s own addresses stay refused. Turning it on needs chats_egress_private_confirm in the same request, with the phrase I understand that commands can reach my private network; the phrase is not stored. The add-on’s Settings tab asks for it. With add-ons enabled the add-on owner checks the same phrase, so a direct host-settings-update call needs chats_egress_private_confirm in its patch too.
  • chats_egress_ports (default 80,443): the ports a command may reach, comma separated. Ports 22 and 25 are refused.
  • chats_egress_extra_registries (default empty): exact host names added to the package registries, comma separated, at most 64; for example an internal mirror (which also needs the private switch, since its address is private). No patterns. They count as registries: a chat reaches them on HTTPS with the Package registries choice, and on any request with All public sites.
  • chats_egress_allowed_hosts (default empty): exact host names or whole-label wildcard patterns an administrator allows, comma separated, at most 64. Each entry is a host name, or host:port to allow that one port (the ports setting still applies); names are lower-cased and written in punycode, and a partial-label pattern, a scheme, a path or an IP address outside the public internet is refused (an IP literal of a private network is accepted only while chats_egress_allow_private is on, and then still needs a rule or a card answer from an administrator like any private address). A listed host is allowed, on any kind of request, in every chat whose effective preset is Package registries or All public sites, with no card. With the network off, or with the limit (chats_egress_ceiling) off, the list does nothing, and a limit below a chat’s choice caps it as for the presets. Private, loopback, link-local, metadata and server addresses stay refused unless the private switch rules allow them, the TLS server name is still checked (on any port, when the tunnel starts with TLS), and every use is audited with the reason allowed_host. A change applies at the next check (an allow the proxy cached lives at most 30 seconds). The add-on’s Settings tab has a list editor next to the extra registries, and the Network panel of a chat shows the list read only. get_chat_egress of the LocalAI Assistant returns it as allowed_hosts; no assistant tool changes it. Patterns are *.example.com (descendants at any depth, not the apex) and * (public destinations only), optionally pinned to an exact port. They normalize IDNA, case and trailing dots. Wildcard ports and partial-label globs are refused; extra registries remain exact. Unlike the legacy exact-host exception, wildcard grants cannot exceed the ceiling: registries requires a listed registry over TLS, all permits public sites, and off blocks everything. Private access is never granted by a wildcard, even with the private switch on.
  • Settings > Tools and POST /api/settings refuse a bad value of chats_egress_ports, chats_egress_extra_registries or chats_egress_allowed_hosts with the reason, and a runtime settings file with one keeps the value in force. A bad value that reaches a frontend another way (a hand edit, a value a newer version reads more strictly) neither empties nor widens the list: the frontend keeps the last valid value it used and logs a warning once. A frontend that never saw a valid value uses the default (no hosts, ports 80,443). The last valid value is kept in the memory of each frontend.
  • chats_egress_audit_days (1 to 365, default 30): how long the audit of the network requests, and the audit of the commands run on this machine (chats_host_exec), keep a line. A value of 0 means a setting never set, and the page shows 30 days.
• Jobs from a paired client (client_jobs_allow_all_tools, default off, applies to jobs that already run; set it with POST /api/settings or in the runtime settings file, because Settings > Tools has no control for it and it has no environment variable): a job that an add-on starts for a paired client reads and reports only. Turn this on to let such jobs use the write and act tools that the agent of their task is set up with, as an ordinary job of that task may. It never gives a grant the agent does not hold. It does not change jobs that people or triggers start.
• Steps per chat turn (chats_max_iterations, 1 to 500, default 40, applies at the next turn): how many rounds of tool calls the model may make in one chat turn before it must answer. This covers chats, saved agents, triggers and connector turns. A turn that reaches the limit shows as Paused with a Continue button (see The status line). A model whose configuration sets agent.max_iterations uses its own limit.
• Automatic turns per goal (chats_goal_max_turns, 1 to 200, default 20, applies when a turn ends): how many turns a chat with a goal may start by itself before the goal pauses. Resuming the goal starts a new count (see Goals).
• Steps per goal (chats_goal_max_steps, 1 to 10000, default 400, applies when a turn ends): how many rounds of tool calls all the turns of one goal may make together before the goal pauses.
• Loops of the chats (apply at once; see Loops):
  • chats_loop_min_interval (a duration from 30s to 24h, default 1m): the shortest interval of a loop, and the shortest wait a self-paced loop may ask for.
  • chats_loop_max_runs (1 to 10000, default 100): the runs after which a loop ends.
  • chats_loop_max_per_chat (1 to 20, default 3) and chats_loop_max_per_user (1 to 200, default 10): how many loops that are not ended one chat and one user may have.
  • chats_loop_daily_runs (1 to 100000, default 200): how many loop runs one user may start in a UTC day, over all their loops. After that the loops wait for the next day.
• Triggers of the chats (apply at once; see Triggers):
  • chats_trigger_run_limit (a duration from 1m to 24h, default 15m): the longest one trigger run may take. A longer run is stopped and fails. A trigger can set a shorter time limit of its own (run_limit); this setting is the ceiling, so a trigger that asks for more runs at this one. A change applies to the runs that start after it and to the check of a run that is found running after a restart.
  • chats_trigger_max_per_user (1 to 1000, default 50): how many triggers a user may have. A user who has more keeps them but cannot add another.
  • chats_trigger_max_daily_cap (1 to 1000, default 1000): the highest daily limit one trigger may have. A trigger with a higher limit runs at this one.
• Add-ons that replace a built-in tool (chats_tool_providers, a map such as {"bash": "sandbox", "xlsx": "office"}, applies at once): when several add-ons replace one tool, the one that serves it. The keys are the replaceable tools (bash, web_search, web_fetch) and the replaceable built-in skills (xlsx, docx, pptx, pdf). A chosen add-on that cannot serve a chat (stopped, off for the chat) leaves the name to the built-in, not to another add-on. With none chosen, the add-on installed first serves (ties go by add-on ID), and the tab marks it as the default and says that you should choose. See Tools that replace a built-in tool.

Note: Most collection settings require a restart to take effect. The settings of the removed agent pool, agent jobs and agent hub (agent_job_retention_days, agent_pool_enable_logs, agent_pool_agent_hub_url and the like) are ignored when found in runtime_settings.json.

Configuration Persistence

All settings are automatically saved to runtime_settings.json in the LOCALAI_CONFIG_DIR directory (default: BASEPATH/configuration). This file is watched for changes, so modifications made directly to the file will also be applied at runtime.

Settings Precedence

Every runtime setting follows a single precedence rule:

1. Environment variables and CLI flags (highest priority)
2. Configuration files (runtime_settings.json, api_keys.json)
3. Default values (lowest priority)

The same rule is applied identically in all three places where settings can change:

• At boot: values persisted in runtime_settings.json fill in every setting that was not explicitly set via an environment variable or CLI flag.
• On POST /api/settings (the web UI Settings page): if a setting is controlled by an environment variable, it cannot be modified through the web interface. The settings page will indicate when a setting is controlled by an environment variable.
• On manual file edits: the configuration watcher hot-applies edits to runtime_settings.json with the same env-over-file semantics as boot, so editing the file by hand behaves like restarting with that file. For example, hand-editing vram_budget installs the new VRAM allocation cap live, without a restart.

Known Limitations

• An environment variable explicitly set to its default value is indistinguishable from an unset one, so the value from runtime_settings.json wins in that case. In particular, an explicit LOCALAI_THREADS=0 behaves like an unset LOCALAI_THREADS.
• A field previously changed via the API (or the web UI) looks environment-set to the file watcher, so a manual file edit of that same field is not hot-applied: it takes effect on the next restart.

Example Configuration

The runtime_settings.json file follows this structure:

{
  "watchdog_enabled": true,
  "watchdog_idle_enabled": true,
  "watchdog_busy_enabled": false,
  "watchdog_idle_timeout": "15m",
  "watchdog_busy_timeout": "5m",
  "max_active_backends": 0,
  "force_eviction_when_busy": false,
  "lru_eviction_max_retries": 30,
  "lru_eviction_retry_interval": "1s",
  "threads": 8,
  "context_size": 2048,
  "artifact_download_concurrency": 4,
  "max_concurrent_backend_operations": 2,
  "f16": false,
  "debug": false,
  "cors": true,
  "csrf": false,
  "cors_allow_origins": "*",
  "p2p_token": "",
  "p2p_network_id": "",
  "federated": false,
  "galleries": [
    {
      "url": "https://index.localai.io/models",
      "mirrors": ["github:mudler/LocalAI/gallery/index.yaml@master"],
      "name": "localai"
    }
  ],
  "backend_galleries": [
    {
      "url": "https://index.localai.io/backends",
      "mirrors": ["github:mudler/LocalAI/backend/index.yaml@master"],
      "name": "localai"
    }
  ],
  "autoload_galleries": true,
  "autoload_backend_galleries": true,
  "vram_persistent_cache": true,
  "api_keys": []
}

API Keys Management

API keys can be managed through the runtime settings interface. Keys can be entered one per line or comma-separated.

Important Notes:

• API keys from environment variables are always included and cannot be removed via the UI
• Runtime API keys are stored in runtime_settings.json
• For backward compatibility, API keys can also be managed via api_keys.json
• Empty arrays will clear all runtime API keys (but preserve environment variable keys)

Dynamic Configuration

The runtime settings system supports dynamic configuration file watching. When LOCALAI_CONFIG_DIR is set, LocalAI monitors the following files for changes:

• runtime_settings.json - Unified runtime settings
• api_keys.json - API keys (for backward compatibility)
• external_backends.json - External backend configurations

Changes to these files are automatically detected and applied without requiring a restart, using the same precedence rule described in Settings Precedence.

Best Practices

1. Use Environment Variables for Production: For production deployments, use environment variables for critical settings to ensure they cannot be accidentally changed via the web UI.

2. Backup Configuration Files: Before making significant changes, consider backing up your runtime_settings.json file.

3. Monitor Resource Usage: When enabling watchdog features, monitor your system to ensure the timeout values are appropriate for your workload.

4. Secure API Keys: API keys are sensitive information. Ensure proper file permissions on configuration files (they should be readable only by the LocalAI process).

5. Test Changes: Some settings (like watchdog timeouts) may require testing to find optimal values for your specific use case.

Troubleshooting

Settings Not Applying

If settings are not being applied:

1. Check if the setting is controlled by an environment variable
2. Verify the LOCALAI_CONFIG_DIR is set correctly
3. Check file permissions on runtime_settings.json
4. Review application logs for configuration errors

Watchdog Not Working

If the watchdog is not functioning:

1. Ensure “Watchdog Enabled” is turned on
2. Verify at least one of the idle or busy watchdogs is enabled
3. Check that timeout values are reasonable for your workload
4. Review logs for watchdog-related messages

P2P Not Starting

If P2P is not starting:

1. Verify the P2P token is set (non-empty)
2. Check network connectivity
3. Ensure the P2P network ID matches across nodes (if using federated mode)
4. Review logs for P2P-related errors

Traffic

The Traffic tab of the Administration console answers three questions: how much is the server used, what failed, and how are the machine and the models doing. It opens on an overview and has a second row of links: Overview, Usage, Models, GPU and host, Traces, Middleware and Prometheus. One time window (24 hours, 7 days, 30 days or all time) is shared by the Overview, Usage and Models pages.

Every figure comes from a record LocalAI really keeps, and each page says which. Where LocalAI keeps no record, the page leaves the figure out and does not draw a zero.

Overview

Five figures in a row: requests, failed requests, latency p95, tokens in and tokens out. Under them, three charts: requests per bucket, failed requests (the trace buffer, in twelve columns) and tokens in and out per bucket. Each chart has a data table behind it, and you can read a value with the arrow keys. A table of the five busiest models closes the page.

• Failed counts a transport error or a 5xx answer. A 4xx answer is the caller being refused, and LocalAI does not count it as a failure.
• p95 is the 95th percentile of request duration in the trace buffer. LocalAI does not compute a p50 or a p99, so there are none.
• With tracing off, the failed and p95 figures say so and link to the setting. The ledger figures do not depend on tracing.
• The trace summary reports at most 7 days. For the 30-day and all-time windows, failed requests and latency cover the last 7 days, and the page says so.

Usage

The ledger as a table, grouped by model, by user (admins) or by API key (when authentication is on). Rows sort by any column, can be searched, and can be filtered by model. A row opens in place on its own chart. A user who is not an admin sees only their own numbers and their quotas, and the page tells them whether the current pace stays inside each quota.

Export CSV and Export JSON save the rows the table holds. The export runs in the browser. The ledger does not record status, endpoint or node, so those are not groups. Estimated cost is optional: you type a price per million tokens, it stays in your browser, and LocalAI has no price of its own.

Models

One row per model: requests and tokens from the ledger, failed operations and the mean operation time from the backend-operation buffer, and the memory the backend process holds now. That is the resident host memory of the process, when the server reports it. GPU memory per model is not reported, and neither is a per-model latency percentile or a memory history.

GPU and host

The current reading, refreshed every 5 seconds: memory per GPU, system memory, the CPU share and the models disk. LocalAI does not report GPU utilisation or temperature. Two charts show readings the page took itself, since you opened this page: the memory pool and the CPU share. They are kept in the browser, at most 240 readings, and leaving the page empties them. On a cluster the page also lists every node with its memory and CPU share.

Traces

The recent API requests and backend operations, with the tracing settings above the list. Filter by failed or slow requests, search, sort, export and clear. Opening an API request shows its page: the status and the error LocalAI recorded, a timeline of the request with the backend operations that ran while it was open, and the request and response bodies. Bodies stay closed until you press Reveal, because they can hold prompts and personal data. Request headers are never listed. The two buffers share no request id, so operations are matched on time and model, and the page says so. A trace leaves the buffer when newer requests push it out; its page then says it is no longer there.

With tracing off, the page explains what is lost and offers Turn on tracing. You can also start LocalAI with LOCALAI_ENABLE_TRACING=true.

Middleware

The order a request passes through: Proxy, Admission, Filtering, Routing, Model. The server fixes that order. Selecting a step shows only its rules. See Middleware for what each step does.

Prometheus

GET /metrics is admin only and returns the Prometheus text format. The page shows the URL and a scrape configuration with a copy button, checks the endpoint against the running server, and lists the metrics LocalAI can export. Only api_call is always present: a histogram of request duration in seconds by HTTP method and route. The others appear while the feature behind them runs. Metrics are off when LocalAI runs with LOCALAI_DISABLE_METRICS_ENDPOINT=true.

Activity

Model installs, backend installs, updates, removals and cluster staging all run in the background. Two surfaces in the WebUI report them: the Activity tab of the notification panel, and the Activity page, which holds the full picture.

Both are admin-only.

To see models currently loaded for inference, open Home (/app) and scroll to Loaded models. That list includes each model’s backend and a stop control. A loaded model can be idle; Activity reports management jobs rather than whether a model is resident or generating output.

The Activity tab

The Notifications entry in the sidebar (the bell in the header on a phone) opens the notification panel. For an administrator the panel has two tabs, Notifications and Activity. Everyone else sees the panel without tabs, as before. When authentication is off, everyone is an administrator.

The panel opens on Activity while something is running, queued or failed, and on Notifications otherwise. A tab you pick stays picked while the panel is open. The tab has three sections, each shown only when it has something in it:

• Needs attention: failed operations, kept until you dismiss them. Each row shows the reason, with Retry and Dismiss.
• In progress: one row for each running or queued operation. A row has a progress ring, the title (Installing qwen3-4b, Updating llama-cpp, Removing whisper), one detail line, a thin bar with the percentage, and Cancel. A queued operation says Waiting. The detail line joins what the operation reports, for example Node 3 of 5 · Downloading · 1.2 GB / 1.9 GB · 48 MB/s: the node a fan-out install has reached, the phase (Resolving files, Downloading, Verifying, Finalizing, Saving configuration), the bytes and the speed. A part the operation does not report is left out.
• Recent: operations that finished successfully in the last day, newest first, at most five, each with how long ago it finished (Installed qwen3-4b, Updated llama-cpp, Removed whisper). A fast install shows here, so it does not vanish before you see it. A cancelled operation, and a failure you dismissed, are not listed. The Activity page keeps them in its record. Nothing is written to the notification list for a finished operation.

See all activity at the bottom opens the Activity page. The tab scrolls inside the panel when it is long.

The sidebar entry shows the operations too. While operations run, the icon sits in a small ring that fills with the average progress of the running operations, and it turns while only queued operations wait. If an operation failed, the ring is red and has a dot. The number badge counts unread notifications only; operations never change it. With nothing running or failed, the entry looks as it always did.

Retry and Cancel behave as on the Activity page (see below). The Cancel wait of 8 seconds applies here too, with the same Undo toast. Closing the panel sends a cancel that is still waiting, as leaving the page does; changing tab does not.

Where the chats feature is off there is no notification inbox. An administrator still gets the entry, named Activity, while an operation runs or failed, shortly after one finished, and for the rest of the visit once one has been seen. It opens the Activity tab alone, with no notification list, filters or browser-notification switch, and it asks the chats for nothing. Anyone else, and an administrator with nothing to show, sees no entry; use the Activity page.

The Activity page

Operate → Activity (/app/activity). The page has up to three sections, each shown only when it has something in it.

In progress

One row per running or queued operation, each naming what is being done and to what. A percentage and progress bar appear only once the operation is running and has reported progress, so a queued operation and a removal show none. Artifact-backed gallery models also report the phase, downloaded and total bytes, and an estimate of the time left once enough of the transfer has been observed to work one out. Every backend install, and every gallery model that lists its files directly, reports no phase and no byte counters; those cards show the installer’s own status line instead, which names the file being fetched and its size.

An install that involves workers shows a per-node list, with one row per worker:

• a status pill: Queued, Downloading, Worker busy, Done or Failed;
• the file being transferred, with current/total bytes and a percentage;
• any error the worker returned.

An install that fans out to more than one worker also carries an N nodes tag and a toggle over the list. Lists of up to four nodes start expanded, longer ones start collapsed behind Show N nodes, and the toggle reads Hide per-node detail while the list is open.

Worker busy means the worker took longer than --backend-install-timeout to acknowledge but is most likely still working. It clears on its own when the worker finishes.

Needs attention

Model and backend operations that failed and have not been acknowledged yet, each row carrying the error returned by the installer, with Retry and Dismiss. Cluster staging never appears here: a staging job reports no error to the page, so a staging failure has to be read from the logs.

Record

What finished, newest first, one row each: the name, what happened (installed in 1m 12s, removed, cancelled, or failed: with the error), the time of day it finished, and a link into Models or Backends. A cancelled install also has Start again, which installs the same target again: a download that was paused continues where it stopped, and one that was cancelled starts over. The record does not say which of the two it was, so the button says what it does in both cases. A cancelled removal has no such button. Model and backend installs and removals are recorded; cluster staging is not, so a staging run leaves nothing behind here once it finishes.

Filters

Four chips above the sections filter the whole page: All, Models, Backends and Cluster. In the live sections, Cluster covers staged model files and any install that involves workers, whether it targets one node or fans out across several. In the Record it covers only the node-targeted case: a finished fan-out install is filed under Models or Backends, not under Cluster.

Cancelling, retrying and dismissing

These are on the operation rows. Pause, Cancel, Retry and Dismiss are labelled buttons. The Activity tab has Cancel, Retry and Dismiss; Pause is on the page only. The Backends page also shows the progress of a backend install in the backend’s own row, with its Cancel.

• Pause stops a running download and keeps the bytes already fetched, so installing the same model or backend again continues from there.

• Cancel is offered while an operation is queued, whatever it is, and while an install is running. It is not offered once a removal has started: a removal in progress cannot be interrupted, so the window to call one off is the queue, before a worker picks it up. Cancelling there stops it before anything is touched. For artifact-backed gallery models, cancelling an active download leaves its partial files in place so a later install resumes rather than starting over. A cancelled operation leaves the live sections; it appears in the record as cancelled.

  Cancel waits for 8 seconds before it is sent. The row says “Cancelling unless you undo” and a toast offers Undo. The server cannot take a cancel back, so the wait in the browser is the whole undo: while it runs nothing has been stopped and the download carries on. Closing the toast cancels at once, asking to cancel a second download ends the first one’s wait, and leaving the page sends a cancel that is still waiting. A job that finishes during the wait is left alone.

• Retry is offered on a failed install or update. It acknowledges the failure, which moves it into the record, and starts the same work again: an install again, or an update again for a failed backend update (on one node, for a node-scoped one). It is not offered on a failed removal, which is not restarted by reinstalling, or on a failed staging job.

• Dismiss acknowledges the failure without retrying. The operation moves into the record with a failed outcome; it is not deleted. This is why the same failure can be found either under Needs attention or in the Record, depending on whether it has been acknowledged.

History

The record holds the last 50 finished operations. Where it is kept depends on how LocalAI is running:

• Standalone: in memory, so it is empty again after a restart. For durable per-model install state, use the model and backend listings rather than this page.
• Distributed mode: in PostgreSQL, alongside the operations themselves. It is the same record on every replica, it survives restarts, and a replica added by a scale-out or a rolling deploy reports it in full rather than starting empty.

Clear history, beside the page title whenever the record has anything in it, empties it. Running operations and unacknowledged failures are untouched. In distributed mode it clears the record for every replica, not just the one serving the page.

The sidebar count

The Operate entry in the sidebar carries a badge whenever something is in flight, showing how many operations are running or queued. If any failure is waiting to be acknowledged, the badge turns red and counts the failures instead, so the number always reports the thing that most wants your attention.

API

Both surfaces are backed by these endpoints. All of them are admin-only when authentication is enabled.

Pausing preserves the download’s .partial file. Start the same model or backend installation again to resume from the saved bytes when the origin supports HTTP Range requests. Cancelling intentionally discards partial data.

Both GET endpoints wrap their list in an operations key rather than returning a bare array:

# What has finished
curl http://localhost:8080/api/operations/history \
  -H "Authorization: Bearer <admin-key>"

{
  "operations": [
    {
      "id": "localai@qwen3-4b",
      "name": "qwen3-4b",
      "jobID": "6f1b0c2e-...",
      "isBackend": false,
      "taskType": "installation",
      "outcome": "completed",
      "startedAt": "2026-07-27T10:02:11.482913574Z",
      "finishedAt": "2026-07-27T10:03:23.117402881Z"
    }
  ]
}

nodeID is present when the operation was scoped to a worker, and error when the outcome is failed. The outcome is one of completed, failed or cancelled. The taskType is installation, deletion or, for a backend update, upgrade.

# Forget the record
curl -X DELETE http://localhost:8080/api/operations/history \
  -H "Authorization: Bearer <admin-key>"

The DELETE answers 500 with an error if the record could not be cleared, which in distributed mode means the rows are still there. It never reports success on a record it did not clear.

Backend Monitor

LocalAI provides endpoints to monitor and manage running backends. The /backend/monitor endpoint reports the status and resource usage of loaded models, /backend/load pre-loads a model into memory, and /backend/shutdown allows stopping a model’s backend processes. In distributed mode, a named shutdown stops both a process local to the controller and every placement registered on workers.

All three are admin-only.

Monitor API

• Method: GET
• Endpoints: /backend/monitor, /v1/backend/monitor

Request

The model to monitor is passed as a query parameter:

For backwards compatibility, a JSON body with the same field is still accepted when the model query parameter is not set, but new clients should use the query parameter.

Response

Returns a JSON object with the backend status:

If the gRPC status call fails, the endpoint falls back to local process metrics:

Usage

curl "http://localhost:8080/backend/monitor?model=my-model"

Example response

{
  "state": 2,
  "memory": {
    "total": 1073741824,
    "breakdown": {
      "weights": 536870912,
      "kv_cache": 268435456
    }
  }
}

Load API

Pre-loads a model into memory ahead of its first request, so that request pays no cold-start load cost. It is the inverse of the Shutdown API and works for any model, not just realtime pipelines.

• Method: POST
• Endpoints: /backend/load, /v1/backend/load

Request

Behavior

• For a regular model, its own backend is loaded.
• For a realtime pipeline model (a config with a pipeline: block), every configured sub-model (VAD, transcription, LLM, TTS, sound_detection, voice_recognition) is loaded concurrently instead of the pipeline stub, which has no backend of its own.

The call blocks until loading finishes and reports which model names became resident, so partial failures are visible.

Usage

curl -X POST http://localhost:8080/backend/load \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model"}'

Example response

{ "loaded": ["my-model"], "message": "model loaded" }

On failure the call returns 500 with loaded listing whichever sub-models did load and message naming the failures.

Shutdown API

• Method: POST
• Endpoints: /backend/shutdown, /v1/backend/shutdown

Request

Usage

curl -X POST http://localhost:8080/backend/shutdown \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model"}'

Response

Returns 200 OK with the shutdown confirmation message on success.

Stopping a backend removes its watchdog timers and eviction state. A timeout from a stopped backend does not shut down a replacement at a different address.

Error Responses

Tracing

LocalAI can retain recent API exchanges and backend operations for inspection on the Traces page in the management interface. Enable tracing in runtime settings or with the existing tracing configuration.

API requests appear while they are still running. Their elapsed duration updates when the page refreshes, and the result column marks them as in progress until the response completes. In-flight requests live only in memory; the completed exchange is what LocalAI adds to the bounded, persistent history.

API and backend trace histories are persisted in separate directories below the configured data path. They are restored after a clean service restart, whether or not authentication is enabled.

Each history remains independently bounded by tracing_max_items. When a history reaches that limit, LocalAI removes its oldest records from memory and disk. The existing clear actions on the Traces page remove both the in-memory history and its persisted records.

Opening an API request on the Traces page shows the request as its own page, at /app/traces/<id>: the status, the error LocalAI recorded, a timeline with the backend operations that ran while the request was open, and the request and response bodies. Bodies stay closed until revealed, and request headers are not listed. See Traffic.

Profiling and request diagnostics

LocalAI provides two independently opt-in diagnostics: a private Go profiling listener and structured phase timing events. Both default to disabled. Debug logging alone enables neither feature.

Scope and configuration

These options apply to local-ai run, including a distributed frontend. They do not enable diagnostics in separately launched workers or provide backend-native llama.cpp profiling. Diagnostics has no MCP, admin, or public API surface and no UI controls. State is per-instance, not shared between replicas. Profiles describe the frontend Go process, not the whole cluster.

The address accepts numeric IPv4 loopback or bracketed IPv6 loopback, such as 127.0.0.1:6060 or [::1]:6060. Ports must be decimal values from 1–65535. Validation rejects hostnames, including localhost, wildcard addresses, empty hosts, LAN addresses, public addresses, and Unix sockets. Validation applies even when profiling is disabled. Negative sampling settings fail validation; nonzero sampling settings require --pprof. --preload-backend-only validates the options but does not start profiling or sampling.

For mutex sampling, a value of N records approximately one in N contention events; 1 records every event. Block sampling targets one blocking event per configured number of nanoseconds spent blocked; 1 records every blocking event. These settings are not percentages or capture durations. Sampling adds process-wide overhead, especially at 1.

One diagnostics server owns process-global runtime sampling in each run process. It applies both configured rates once after binding successfully and resets both rates to zero on shutdown or fatal serving failure. It does not restore previous rates; Go provides no getter for the previous block rate. Do not combine competing sampling owners in the same process. There is no dynamic toggle, continuous capture, or automatic profile storage. Configuration changes require a separately approved restart or rollout.

The listener binds before normal request serving. Validation or bind errors fail startup; subsequent fatal serving errors propagate through the run lifecycle. The private listener gets five seconds to drain before profiling connections are force-closed, which can interrupt an active capture. Its cleanup runs on startup failure, normal return, and through the existing signal handler (which exits without running deferred cleanup). A fatal profiler serving error cancels the run context and is returned after application cleanup. Application initialization or shutdown work that ignores cancellation can delay that return; the private-listener budget is not a deadline for the entire application. With profiling disabled, diagnostics opens no socket, starts no monitoring goroutine, and changes no runtime sampling settings. Disabled phase timing allocates no diagnostic identities or events and adds no diagnostic filesystem operations.

Access and data sensitivity

The separate listener uses a private mux, not the public API router or the default Go mux. It provides the profiling index, CPU, trace, symbol, and named runtime profiles, including heap, goroutine, mutex, and block. /debug/pprof/cmdline is unavailable, including query and subtree variants, because command-line arguments can contain credentials. The standard index can still mention that endpoint.

Timing events omit credentials, bearer candidates, model names, prompts, bodies, raw headers, filenames, paths, and raw errors. This sanitation applies only to diagnostic timing payloads. It does not redact unrelated logs or profiles. Events use the constant message diagnostic_phase at Info. Operator-selected warn/error filtering suppresses these events, even when phase timing is enabled.

Bounded Kubernetes investigation

A rollout or build configuration change is a separate, explicit, owner-approved action. Select one frontend and a defined observation window. Approve profiling, timing, and any nonzero sampling settings before the window. Do not change replicas, clocks, scheduling, or storage mounts to force a result. The examples below capture data only; they do not mutate a live deployment.

Your cluster runtime must support port-forwarding to the pod’s loopback listener. Do not add a Service, ingress, or NodePort for profiling. Forward directly to the selected frontend pod, not a load-balanced Service. The local port must be free. Keep kubectl’s default local-only binding; do not add a wildcard --address.

1. Start the direct pod port-forward in a separate terminal, replacing both placeholders:

   kubectl -n <namespace> port-forward pod/<selected-frontend-pod> 6060:6060

2. Protect newly created files in your capture terminal:

   umask 077

3. Capture ten seconds of CPU activity during the approved window:

   curl --fail --output cpu.pprof 'http://127.0.0.1:6060/debug/pprof/profile?seconds=10'
   go tool pprof cpu.pprof

CPU profiles show execution, not time blocked on I/O or locks. Capture naturally slow requests and matched control requests against the same frontend. Record the observation window and generated diagnostic IDs alongside protected captures. Profiles are process-wide; they do not automatically attribute samples to a request ID.

Mutex and block profiles require their respective nonzero sampling settings before capture. The following ten-second delta captures bound collection time, not the duration of process-global sampling. Sequential commands cover different windows; compare them accordingly.

curl --fail --max-time 20 --output mutex.pprof 'http://127.0.0.1:6060/debug/pprof/mutex?seconds=10'
curl --fail --max-time 20 --output block.pprof 'http://127.0.0.1:6060/debug/pprof/block?seconds=10'
go tool pprof mutex.pprof
go tool pprof block.pprof

Mutex profiles locate contention attributed to lock holders; block profiles locate sampled blocking operations. Neither substitutes for the request-owned lock and filesystem timings below. A goroutine snapshot can show I/O or lock waits that CPU profiles miss. It describes one instant, not a measured duration. Its text can be large and sensitive; the client timeout bounds waiting, not file size.

curl --fail --max-time 10 --output goroutines.txt 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=2'

Stop the port-forward after capture. Disable diagnostics and sampling afterward through the separately approved rollout process. Protect, review, and delete the capture files under your data-handling policy. If latency does not reproduce, report non-reproduction. Do not infer a predetermined filesystem, lock, or routing root cause.

Event identity and phase boundaries

Each event contains id, kind, phase, state, outcome, elapsed, and count. The server generates diagnostic UUIDs; it does not trust or reuse client X-Correlation-ID headers as diagnostic identities. Requests use kind request; config reload operations use kind config_reload with fresh operation IDs. Request identity survives the instrumented internal context reconstructions. Background callers remain unobserved unless explicitly enabled.

States are start, end, and mark; outcomes are ok, error, canceled, and skipped. Durations use monotonic elapsed time, not cross-host clock subtraction. Counts describe aggregate work, not filenames or model labels. Lock-protected helpers emit buffered aggregates after unlocking, so log order alone does not establish execution order.

Nested timings overlap and are not additive. Repeated extraction segments share one request ID; they are not duplicate whole-request measurements. Extraction ends before downstream inference. It is not socket arrival or full request latency. The existing trace start follows global authentication and body reading; it is not socket arrival either.

Reload observer boundaries

Reload timing measures ModelConfigLoader.loadModelConfigsFromPath, not the unrelated dynamic JSON watcher. Distributed resync constructs a temporary authoritative loader with its own mutex. That loader’s reload events do not prove contention on the shared loader lock or ownership of another request’s wait. No event establishes cross-host timing or worker execution.

The immutable constructor option WithReloadDiagnostics controls early lock, enumeration, and metadata timing. Ordinary LoadOptionDiagnostics callbacks run at their original position after enumeration and metadata, under the loader lock. Their resolved recorder controls later YAML-read and parse/default timing. An option-only caller cannot observe earlier phases. A late nil recorder disables subsequent instrumented work, not constructor-enabled early measurements.

Unchanged recorder identity preserves the reload operation ID across early and later phases. A different non-nil late recorder receives a separate reload operation ID; these are separate observations, not one correlated span. Callbacks retain ordering, composition, and last-write semantics. They retain every preexisting invocation, including each parsed model’s ModelConfig.SetDefaults; diagnostics does not add a replay or snapshot functional options.

See the CLI reference for other server options. The troubleshooting guide covers broader operational checks.

Response correlation

With --request-phase-timing (LOCALAI_REQUEST_PHASE_TIMING=true), requests that reach model extraction receive X-LocalAI-Diagnostic-ID. The header contains the same server-generated UUID v4 as that request’s phase events. LocalAI sets it before extraction errors or streaming output can write the response. Repeated extraction and failover attempts retain one ID for the logical request.

Client-supplied diagnostic or correlation headers do not select this ID. The ID contains no model name, authentication token, or request content. It is a correlation value, not an authorization credential or a distributed trace ID. Requests rejected before extraction, including global authentication failures, do not receive this header. Disabling request-phase timing emits no diagnostic header and creates no diagnostic identity. Enabling pprof or debug logging alone does not enable the header.

Go callers can use diagnostics.RequestID(ctx) to read the existing request ID without creating one. The accessor returns an empty string for nil contexts, unobserved contexts, and background reload observations.

Chat model fallback

OpenAI chat completions bind the request before considering legacy model discovery. An explicit bound model skips Bearer-as-model lookup and default candidate listing; model existence validation still runs. If the bound model is empty, extraction retains context, path, query, form, Bearer, and first-available fallback precedence. The existing candidate filter and ordering still select the default. Malformed JSON returns a parsing error without fallback discovery. Each failover attempt binds fresh input from the replayed body. Other API routes retain their existing extraction order.

Middleware: PII filtering and intelligent routing

LocalAI ships a request-middleware layer that sits between the HTTP API and the backend dispatcher. Two subsystems share that layer because they share the same lifecycle hook: PII filtering scans the request body before it reaches a backend, and the intelligent router rewrites input.Model so a single client-facing model name fans out across multiple downstream targets.

Both are inspected and configured from the same admin page (/app/middleware), which draws the order a request passes through as five steps (Proxy, Admission, Filtering, Routing, Model) and shows only the rules of the step you select, backed by the same REST surface (/api/middleware/*, /api/pii/*, /api/router/*) and the same MCP tools.

Request lifecycle

client ── auth ── route-model ── per-model PII ── backend ── client
                       │              │
                       │              └─── event log
                       └─── decision log

The router runs first (it picks the target model so per-model PII has something to gate on), per-model PII runs next (gated by the resolved config), and the backend executes. Filtering is request-side only - the request body is scanned and rewritten before forwarding; the response is not touched (NER over a streamed response is left as a follow-up). Each subsystem writes to its own admin-visible log: /api/router/decisions for routing, /api/pii/events for redaction and block actions.

PII filtering

PII redaction is NER-based and runs request-side (input). It is off by default, flipping to on for any cloud-proxy backend because that traffic crosses the network to a third-party provider. Pick a default detector so those models are actually scanned. Explicit pii.enabled in a model’s YAML always wins over the backend default.

Filtering runs on every text-accepting endpoint that has an adapter wired: /v1/chat/completions and /v1/messages (chat), /v1/completions, /v1/embeddings, /v1/edits, the Open Responses POST /v1/responses (instructions, string input, and in input items the message and reasoning text, function_call arguments and function_call_output output), and the Ollama /api/chat, /api/generate and /api/embed endpoints, plus the MITM proxy request body. Image, audio (TTS/STT), video, rerank, and the realtime WebSocket are not filtered yet (different prompt-PII semantics; realtime is not HTTP middleware).

The Responses WebSocket mode (GET /v1/responses) is not HTTP middleware either, so it cannot filter: a response.create for a model with PII filtering enabled is refused with the error type pii_unsupported_transport instead of being served unfiltered. Use POST /v1/responses for those models.

Stored history loaded through previous_response_id or item_reference is resolved after filtering and is not scanned against the current model’s policy. That history can contain unfiltered text from another model or model-generated output. Tool output supplied directly in the request is scanned as text, or as its complete JSON encoding for non-string values, including arrays.

A request’s messages are scanned as one document (joined in order), so the NER detector keeps conversational context: whether 4421 is a PIN or jdoe_42 is a username is usually decided by the question asked in the previous message, and a bidirectional encoder only sees that context when the messages share a forward pass. Detected spans are mapped back to the individual message they fall in, so redaction still rewrites each message field in place and events carry message-local offsets.

The earlier regex pattern tier (pii.patterns, the built-in pattern catalogue, --pii-config, the /api/pii/patterns|test|decide endpoints) and response/streaming-side redaction have been removed. Detection is now driven entirely by token-classification (NER) models. Legacy keys no-op with a startup warning.

Detector models

A detector is a token_classify model (e.g. an openai-privacy-filter GGUF) that carries the detection policy in a top-level pii_detection: block - defined once, on the model itself:

name: privacy-filter-multilingual
backend: privacy-filter
embeddings: true              # TOKEN_CLS pooling
known_usecases:
  - token_classify
pii_detection:
  min_score: 0.5              # drop detections below this confidence
  default_action: mask        # applied to any detected group with no entry
  entity_actions:             # which PII to block vs mask vs allow-log
    PASSWORD: block
    CREDITCARD: block
    EMAIL: mask

mask rewrites the matched span to [REDACTED:ner:<GROUP>] in the request body before forwarding. block returns HTTP 400 (error.type=pii_blocked) without forwarding. allow detects and logs (a PIIEvent is still recorded) but leaves the text unchanged. The entity-group names are whatever the model emits (the privacy-filter family uses uppercase names like EMAIL, PASSWORD, CREDITCARD).

Extending a detection to the next word

Some identifying values come in pairs where the model reliably tags one half and misses the other. The typical case is an address line: the postal code is detected with high confidence, the town right after it is tagged with a low score or not at all, and masking the code alone leaves the town in the clear ([REDACTED:ner:ZIPCODE] Ludwigshafen). extend_to_next_word lists entity groups whose accepted detections also cover the word that directly follows them on the same line, with the same action and group:

pii_detection:
  min_score: 0.4
  extend_to_next_word:
    - ZIPCODE                 # "67059 Ludwigshafen" -> "[REDACTED:ner:ZIPCODE]"

A word is a run of letters, digits, combining marks, -, ' and inner dots (a trailing dot is left out). The extension skips spaces and tabs but never a line break, and it does nothing when no word follows. Only one word is added, so a multi-word town (“Frankfurt am Main”) keeps its tail.

The extension is deliberately blunt: the next word can also be a number or a date (ZIPCODE followed by 12 Mar 2026 takes 12), so it trades some over-masking for fewer leaks. Use it only for groups whose neighbour is identifying.

extend_to_previous_word is the mirror for the word before a detection. A name model often tags the surname with high confidence and scores the first name in front of it below min_score (Gast Greta [REDACTED:ner:LASTNAME]):

pii_detection:
  extend_to_previous_word:
    - LASTNAME                # "Gast Greta Waldmeister" -> "Gast [REDACTED:ner:LASTNAME]"

Here the preceding word must consist of letters (plus - and '): a word with digits - an amount, a date, a room number - is never taken, nor a protected term. The search skips spaces and tabs but never a line break.

The word before a hit is not always a first name: a title (Herr), a preposition or any other plain word in front of the surname is taken too (Gast Greta Waldmeister loses Greta, Herrn Waldmeister loses Herrn). Like the next-word option it trades over-masking for fewer leaks. Do not combine both directions on the same name group: each hit then also takes the word after it, so a name followed by a number swallows that number.

Protected terms

The opposite failure is a value the model tags although it must reach the upstream model: a business name read as a surname (Hotel Seeblick -> Hotel [REDACTED:ner:LASTNAME]) turns an invoice into one the model cannot categorise. protected_terms lists such values; protected_terms_files adds files inside the models path with one term per line (# starts a comment), re-read whenever they change, so an external process can keep the list current without a restart:

pii_detection:
  protected_terms:
    - Hotel Seeblick
  protected_terms_files:
    - privacy-filter/known-suppliers.txt

Each whole-word, case-insensitive occurrence (whitespace inside a term matches any whitespace run) is replaced by a neutral placeholder before the detector runs and restored afterwards. The detector never sees the value, so it cannot tag it or let it colour the context around it, and the parts of a detection that fall on the placeholder are cut out. Event offsets still address the original text. Terms shorter than three characters are ignored, and a missing or unreadable file is skipped with a warning (fewer protected terms means more masking, never less).

A term that is only part of a longer word is not protected: with the term Anna, Anna-Lena and O'Anna are still passed to the detector. Protected terms apply to the NER tier only. A pattern detector (backend: pattern) always scans the original text and may still mask a match inside a protected term.

A protected term passes everywhere in the text: if the same string is also a person’s name, that occurrence passes too. Keep the list to values that are safe to send.

Pattern detector tier

NER is the wrong tool for high-entropy, highly-regular secrets - API keys, tokens, private-key blocks. A trained NER model has no “API key” class, so it fragments a key into the nearest categories it does know and can leave the secret part exposed. Those secrets are exactly what a regex catches cheaply.

A pattern detector is a detector model (backend: pattern) that matches secrets with a restricted regex subset compiled to Go’s RE2 engine - linear-time, no backtracking, no ReDoS. It runs entirely in-process: no model download, no backend, zero VRAM. Install the gallery’s secret-filter for a ready-made set, or define your own:

name: secret-filter
backend: pattern
known_usecases: [token_classify]        # so it appears in the detector picker
pii_detection:
  default_action: block                 # a leaked credential shouldn't leave
  builtins:                             # built-in catalogue (enable by name)
    - anthropic_api_key
    - openai_api_key
    - github_token
    - aws_access_key
    - private_key_block
  patterns:                             # operator-defined, restricted subset
    - name: INTERNAL_TOKEN
      match: "tok-[A-Za-z0-9]{32,64}"
      action: block                      # optional per-pattern override
      min_len: 36                        # optional length floor

A match is reported under its group (built-in group name, or the pattern name), so entity_actions / default_action apply exactly as for NER.

The restricted grammar (validated at load - an invalid pattern is rejected, not silently ignored):

• Allowed: literals, character classes […] and \w \d \s, alternation, anchors ^ $ \b, and quantifiers ? * + {m,n}.
• Rejected: . (any-char), capturing groups, and {n,m} bounds over 4096.
• Required anchor: every pattern must contain a fixed literal run of at least 3 characters (e.g. sk-ant-, ghp_, AKIA). This admits real key shapes but rejects open-ended ones - an email or a bare \w+ has no such anchor and belongs to the NER tier.

Use both tiers together: reference an NER detector and a pattern detector in a model’s pii.detectors (or as instance defaults); their hits union, and a block from either rejects the request.

Consuming models

Any model opts in by enabling PII and referencing one or more detectors - no per-consumer policy:

name: claude-strict
backend: cloud-proxy
proxy:
  mode: passthrough
  provider: anthropic
  upstream_url: https://api.anthropic.com/v1/messages
  api_key_env: ANTHROPIC_API_KEY
pii:
  enabled: true               # default-on for cloud-proxy; explicit for audit
  detectors:
    - privacy-filter-multilingual
  reversible_redactions: true # restore request PII if the model echoes its wrapped token
  reversible_token_prefix: "[REDACTED:" # optional; this is the default
  reversible_token_suffix: "]"          # optional; this is the default

reversible_redactions enables bijective, request-scoped replacement. Each masked value is sent to the backend as a stable wrapped token such as [REDACTED:EMAIL_001] instead of a generic redaction marker. If the model includes that token in its response, LocalAI restores the original value before returning JSON or SSE to the caller. The substitution map exists only for that request and is never logged or persisted. Leave the option unset (the default) for irreversible [REDACTED:...] masking.

The prefix and suffix reduce collisions with ordinary model output and can be customized with reversible_token_prefix and reversible_token_suffix. Reversible redactions provide less confidentiality than irreversible masking: any third party that can observe both the redacted request and restored response may be able to infer the original values.

block_unscanned_media: true closes the gap the text detectors leave open: images, audio, video and files in a request are never scanned, so by default they reach the backend unchecked. With the option set, a request that carries any such content is rejected with HTTP 400 (error.type=pii_blocked, audit event pattern __unscanned_media__) before a detector runs. Use it for egress policies where “cannot be inspected” must mean “not sent”, e.g. on cloud-proxy models. It applies to the OpenAI chat, Open Responses, Anthropic messages and Ollama chat routes, and only while PII is enabled for the model.

Multiple detectors union their detections; overlapping spans resolve to the strongest action (block > mask > allow). A configured detector that can’t be loaded fails the request closed (HTTP 503, error.type=pii_ner_unavailable) rather than silently skipping the check. The same NER path runs on the MITM proxy request body for intercepted hosts. Reversible response restoration currently applies to LocalAI API routes; the MITM proxy keeps its own output-redaction policy.

Instance-wide default detector

The Detector models table on the Middleware → Filtering page lists every token_classify detector model (neural NER models and in-process pattern matchers alike) and exposes a per-row Default toggle. Toggling a detector on adds it to the instance-wide default detector set - one or more models applied to any PII-enabled model that names none of its own pii.detectors. It is persisted through POST /api/settings and read live, so a change takes effect on the next request without a restart. A default that names a model no longer loaded still appears (marked not loaded) so it can be toggled off.

The default set can also be supplied out-of-band with the LOCALAI_PII_DEFAULT_DETECTORS environment variable (comma-separated model names, e.g. privacy-filter-nemotron,secret-filter). When set it takes precedence over the value persisted via the UI (env > file), which is the right behaviour for immutable container deployments that pin filtering policy at boot rather than via the admin UI.

This is what makes cloud-proxy / MITM redaction work out of the box: those backends default to PII-enabled but ship no detector list, so without a default detector the filter runs with nothing to scan. Set one here and cloud-proxy traffic is scanned with no per-model config.

Resolution precedence (the single decision point is ResolvePIIPolicy, shared by the chat middleware and the MITM listener so both agree):

1. An explicit pii.enabled on the model wins - true or false.
2. Otherwise PII is on if the backend defaults it on (cloud-proxy).
3. Detectors are the model’s own pii.detectors; if it lists none, the instance-wide default detector(s) are used.

A model that resolves enabled but ends up with no detector at all (a cloud-proxy model with no model detectors and no instance default) scans nothing - set a default detector to close that gap.

Admin page

The /app/middleware page (admin role only) has four tabs - Filtering, Routing, MITM Proxy (see the MITM doc), and Events. The Filtering tab has a Detector models table (every token_classify filter model, with the per-row Default toggle above and an edit link to each detector’s config, plus an Add detector model button) and a per-model table listing only the models PII can actually apply to - chat / completion / embeddings / edit consumers and cloud-proxy models, not VAD/STT/image models or the detector models themselves. Each row reports the effective enabled state as an inline toggle - flipping it writes an explicit pii.enabled to that model’s YAML (a server-side deep-merge that preserves pii.detectors and every other field), so a cloud-proxy model shown on by backend default can be turned off, and vice-versa - plus the resolved detector(s) - with a (default) marker when they come from the instance-wide default rather than the model’s YAML - why it is on (YAML / backend default), and the recent event count. Detection policy (entity→action, min score) is still edited on each detector model’s config (Models → edit → PII), not globally.

Analyze / redact API

The same detection pipeline is also exposed as a standalone service, so a client can scan or sanitise a string without routing a full chat request through it (the inline path above). Two endpoints, both requiring a normal API key (the pii_filter feature - not admin):

• POST /api/pii/analyze - detect only. Returns the matched entity spans (entity_type, source ner|pattern, start/end, score, action) and a blocked flag, without modifying the text.
• POST /api/pii/redact - apply the configured policy. Returns redacted_text (with masked spans replaced by [REDACTED:<id>]) and masked; when a block action fires it returns 400 with type: pii_blocked and the offending entities - never a redacted body.

Both take the same request: text plus a detector selection - either explicit detector model names in detectors, or a consuming model whose effective policy is used: the model’s own pii.detectors, else the instance-wide default detectors, exactly as the inline filter resolves them. A model with PII disabled - or enabled but with no detector anywhere - is a 400: the inline filter would scan nothing for it, and the API says so rather than implying a clean scan. The detection policy lives on the detector models exactly as for the inline filter. The raw matched value is never returned (an admin may pass reveal: true to include the audit hash_prefix).

text is scanned as a single document. To reproduce the inline filter’s conversation-context behaviour for multi-message content, join the messages with blank lines into one text - NER detection quality depends on that context (a bare 4421 is nothing; after “what are the last four digits of your card?” it is a PIN).

# Redact with an explicit pattern/NER detector
curl -sX POST http://localhost:8080/api/pii/redact \
  -H 'Authorization: Bearer $API_KEY' -H 'Content-Type: application/json' \
  -d '{"text":"reach me at jane@acme.io","detectors":["my-ner-model"]}'
# => {"redacted_text":"reach me at [REDACTED:ner:EMAIL]","masked":true,...}

# Analyze using a consuming model's configured detectors
curl -sX POST http://localhost:8080/api/pii/analyze \
  -H 'Authorization: Bearer $API_KEY' -H 'Content-Type: application/json' \
  -d '{"text":"sk-ant-api03-…","model":"gpt-4"}'
# => {"entities":[{"entity_type":"ANTHROPIC_KEY","source":"pattern",...,"action":"block"}],"blocked":true}

Calls are audited in the same event log, tagged with an origin of pii_analyze / pii_redact (the inline filter records middleware, the MITM proxy records proxy), so GET /api/pii/events?origin=pii_redact shows just the redact-API rows.

REST surface

MCP tools

The same surface is mirrored through the LocalAI Assistant MCP server:

Detection policy is part of a detector model’s config, so it is managed through the model-config tools (edit_model_config), not a dedicated PII tool.

Intelligent routing

A router model is a model whose YAML carries a router: block. When a client addresses it ("model": "smart-router"), the middleware classifies the prompt, picks a downstream candidate model, rewrites input.Model to the candidate, and the standard model-resolution path runs against that resolved target. ACL checks, disabled-state, and per-model PII all apply to the resolved model - the router does model selection only.

Depth-1 invariant

Candidates must not themselves be router models. A smart-router → claude-strict → cloud-proxy chain is fine (claude-strict is a regular cloud-proxy model). A smart-router → other-router → real-model chain is rejected at runtime by the middleware (the dispatcher returns HTTP 500 with a depth-1 invariant error). This keeps the dispatch graph acyclic and predictable.

Fallback

If no candidate’s label set covers the active label set from the classifier, or the classifier errors out, the router uses cfg.Router.Fallback. An empty fallback causes the dispatch to fail with HTTP 500 rather than silently routing somewhere unintended - fail-fast, not silent-bypass.

Available classifiers

LocalAI ships four classifier implementations. Pick one with classifier: in the router YAML:

All four share policies, candidates, and fallback. The existing score, colbert, and knn classifiers also use classifier_cache_size. score and colbert take a classifier_model (+ activation_threshold, optional embedding_cache); knn instead takes a knn: block and a corpus seeded through the API.

Native decision models (decisions)

Use classifier: decisions with an installed native decision model explicitly configured with known_usecases: [decisions] and a backend supporting the Score RPC. A chat model with ordinary continuation scoring is not a substitute. The router calls the internal ModelSystemOne adapter, never a loopback HTTP endpoint, NER extractor, or text-generation fallback.

name: policy-router
router:
  classifier: decisions
  classifier_model: my-decision-model
  activation_threshold: 0.5
  policies:
    - label: code
      description: Writing or debugging code
    - label: private
      description: Handling private information
  candidates:
    - model: coding-model
      labels: [code]
    - model: private-coding-model
      labels: [code, private]
  fallback: general-model

Each policy becomes a separate noul question with explicit false and true criteria. Stable question IDs map answers back to policy declaration order. The native numeric noul value is P(true); a probabilities map is not required. label_scores are independent values in [0,1], not a distribution normalized across policies. Every label with probability >= threshold is active, and score is the maximum probability. The default threshold is 0.5 (omitted or zero); positive configured thresholds must be finite and <=1. There is no exclusive-choice mode or top-one rescue. All-below-threshold means abstention. The first candidate covering all active labels wins; abstention, malformed answers, native context rejection, and backend errors use the normal configured fallback. Parent request cancellation returns without selecting a fallback.

Limits: 1–64 unique nonblank policies, nonblank descriptions, and 64 KiB each for serialized internal request and raw response. Missing/null/non-numeric, wrong-type, unknown-ID and out-of-range answers fail classification; numeric zero is valid. The native engine owns question framing and context enforcement. The router does not estimate native context from raw JSON token counts or trim text on that assumption. A request within the byte limit can still exceed the model’s context and fall back.

The native adapter has a separate process-wide eight-operation ceiling, covering healthy calls as well as abandoned work, independently of the general backend admission limit (default 1024). Saturation fails classification and uses the configured fallback; it does not queue. Model loading has no cancellable API: cancellation releases the waiting caller, not underlying resources. Eight stuck loads therefore prevent further native decisions until underlying work finishes. This bound is a resource-safety measure, not a throughput claim.

The classifier registry invalidates on router configuration, resolved native model configuration/usecase, or persisted model revision changes. Decisions do not memoize prompt results or support embedding_cache/composite knn blocks; these combinations are rejected. classifier_cache_size has no effect on this classifier. The same central factory serves chat, Anthropic, realtime and the existing /router/decide oracle. Decision traces contain no prompt text.

The Score classifier

The score classifier works like this:

1. Build a Qwen/ChatML system prompt that lists every policy label with its description and primes the model to emit a label as the assistant turn.
2. Ask the classifier model to score every policy label as the first-token(s) continuation. This uses the Score gRPC primitive (backend.proto::Score), which returns per-candidate log-probabilities length-normalized so candidates of unequal token length stay comparable.
3. Softmax the length-normalized log-probabilities into a probability distribution over labels.
4. Threshold the distribution: every label whose probability passes activation_threshold joins the active label set.
5. Pick the FIRST candidate whose Labels is a superset of the active set. Admins order candidates smallest → largest so a single-label query routes to the smallest capable model, while a query that activates multiple labels falls to a candidate that covers them all.

This is the Arch-Router approach extended for multi-label. The distribution carries more signal than the argmax - reading off the spread lets one prompt activate multiple policies and route to a model capable of all of them.

Recommended classifier model

Arch-Router-1.5B is the canonical choice. It’s a Qwen-2.5-1.5B-Instruct base trained specifically on routing-policy continuation, so the ChatML system-prompt

• label-continuation pattern produces well-separated label probabilities without prompt tuning. The Q4_K_M GGUF runs on CPU, GPU, and Intel SYCL.

The classifier model must support the Score gRPC primitive (today: the llama-cpp and vLLM backends) and use the ChatML chat template. Any small ChatML instruct model works under those constraints, but expect flatter probability distributions which translate to a higher activation_threshold to keep noise out of the active label set.

On llama-cpp, scoring rides the server’s task queue alongside generation and embeddings, so the classifier may share a model config with chat/completion/embeddings - a dedicated scorer model is no longer required. Repeated calls with the same prompt also reuse the prompt’s KV cache across candidates.

The Colbert classifier

The colbert classifier reranks each policy description against the prompt via the rerankers backend and activates the labels whose relevance scores clear activation_threshold (default 0.5 for reranker-style scores in [0, 1]).

router:
  classifier: colbert
  classifier_model: bge-m3-colbert  # gallery entry; loads BAAI/bge-m3 in ColBERT mode
  activation_threshold: 0.5
  policies:
    - label: code-generation
      description: writing, debugging, reading, or explaining code
    - label: casual-chat
      description: small talk, greetings, jokes
  candidates: [...]

The reranker scores the description (natural English) rather than asking a small LM to score the label as a next-token continuation, so it tends to be more robust when policy labels are abstract slugs (compliance-review, tier-2-support). The trade-off is one reranker round-trip per request - bge-m3 in ColBERT mode is fast enough on GPU that this is comparable to the Score path for most workloads. The embedding_cache block applies identically.

The reranker model’s type: (in the model YAML) selects which underlying scoring head loads - colbert for late-interaction MaxSim, cross-encoder for cross-attention scoring. The classifier itself is indifferent; pick the head that fits your latency / quality budget.

The KNN classifier

The knn classifier routes by similarity-weighted vote over a curated corpus of labelled example prompts. Where score and colbert ask a model’s opinion per request, knn consults recorded experience: each corpus entry is an example prompt plus the policy labels it should activate. It needs no classifier model — just an embedding model and a seeded corpus.

router:
  classifier: knn
  fallback: gpt-4o-proxy          # used whenever the prompt is unlike all corpus entries
  knn:
    embedding_model: nomic-embed-text-v1.5
    # embedding_revision: "2026-07" # bump for remote/in-place weight changes LocalAI cannot identify
    k: 3                          # neighbours that vote (default 3)
    similarity_threshold: 0.80    # the epistemic gate (default 0.80)
    vote_threshold: 0.5           # weighted vote share a label needs (default 0.5)
    # store_name: router-corpus-smart-router   # default "router-corpus-<router>"
  policies:
    - label: code-generation
      description: writing or debugging code
    - label: casual-chat
      description: small talk
  candidates:
    - model: qwen3-0.6b
      labels: [casual-chat]
    - model: qwen-coder
      labels: [code-generation, casual-chat]

For each request:

1. Embed the prompt with knn.embedding_model.
2. Fetch the k nearest corpus entries (cosine similarity).
3. Epistemic gate: entries below similarity_threshold cannot vote. If none clears it, the classifier activates no labels and the router uses fallback — a prompt unlike all labelled experience is treated as undecidable, not guessed. The decision log records nearest_similarity so you can see how far away the closest labelled example was.
4. Each surviving neighbour votes for its labels, weighted by its similarity; every label whose vote share clears vote_threshold joins the active set. Candidate matching then proceeds exactly as for the other classifiers. With k: 1 this degenerates to “nearest example’s labels”.

The numeric configuration is bounded so one request cannot allocate an unbounded neighbour result and the cosine-weighted vote remains meaningful:

Negative, non-finite, and above-range values are rejected when the model configuration is loaded. The k cap bounds the store priority queue and returned neighbour arrays. Similarity is restricted to non-negative cosine weights because negative weights would make vote totals and shares invalid; vote share itself is necessarily between zero and one.

Every knn decision (in the decision log and the /api/router/decide response) also carries neighbors — the k retrieved corpus entries by descending similarity, including ones below the gate, each as {id, similarity, labels}. The id is the entry’s content hash (the first 8 bytes of the SHA-256 of its text, hex-encoded): stable across reseeds and re-embeds, and text-free — whoever seeded the corpus can recompute text→id on their own copy to group decisions by corpus region (e.g. for external per-region reliability accounting) without corpus text ever leaving the server.

Seeding and curating the corpus (API-only)

Corpus entries may contain example user content, so they are managed exclusively through the admin API — the UI never sends or displays them, and no endpoint returns entry texts (inspection is label counts only):

# Seed labelled exemplars (embedded server-side; indexed immediately)
curl -X POST http://localhost:8080/api/router/smart-router/corpus \
  -H "Content-Type: application/json" \
  -d '{"entries": [
        {"text": "why does this segfault when I free the buffer twice", "labels": ["code-generation"]},
        {"text": "hey hows it going", "labels": ["casual-chat"]}
      ]}'

# Inspect — counts only, never texts
curl http://localhost:8080/api/router/smart-router/corpus/stats

# Wipe (file + live index); reseed afterwards
curl -X DELETE http://localhost:8080/api/router/smart-router/corpus

Entry labels must be declared in policies (same invariant as candidate labels). Empty and duplicate labels are rejected, and duplicate texts are skipped rather than double-weighted. Label your exemplars with outcomes, not topics, when routing for difficulty: an entry recording “the small model handled prompts like this” is exactly as useful as one recording that it failed — grade a sample of production traffic against your candidates and seed both.

Persistence

The corpus is persisted as one JSONL file per router under <data path>/router-corpus/ (text, labels, vector, embedding-model name, and embedding fingerprint) — the file is the source of truth and survives restarts; the local-store index is rebuilt from it at classifier build time without re-embedding. Before each KNN lookup, LocalAI checks a stored vector against the live index. If the store restarts empty after eviction or an idle timeout, LocalAI restores the index from the file without re-embedding. A synchronization error fails the lookup.

The fingerprint follows the effective embedding-model config and local artifact identity, so changing the model or replacing its local weights re-embeds the corpus on the next process load. For remote embedding services whose weights can change invisibly, bump knn.embedding_revision explicitly.

If an embedding fingerprint changes after that corpus is already present in the live in-memory index, LocalAI fails the classifier build instead of querying mixed embedding spaces. Restart LocalAI to rebuild the empty live index and re-embed the persisted entries.

Tuning notes

• similarity_threshold is the safety knob. Too low and the router confidently extrapolates from unrelated exemplars; too high and everything falls back. Watch nearest_similarity in the decision log: fallback rows clustering just under the threshold mean the corpus needs entries near that traffic (or the gate is too tight).
• k trades robustness for corpus density: k: 3 tolerates one mislabelled neighbour; raise it only when every label region has several exemplars. The maximum is 1024.
• embedding_cache is ignored for knn (with a warning) — the classifier is already an embedding KNN lookup; wrapping it in another would embed twice for no additional information.

YAML reference

name: smart-router
known_usecases:
  - chat
router:
  # `score` (Arch-Router-style next-token scoring), `colbert` (rerank
  # policy descriptions), or `knn` (vote over a labelled corpus).
  # See "Available classifiers" above.
  classifier: score

  # A model loaded by LocalAI that supports the Score gRPC primitive
  # (llama-cpp and vLLM ship implementations). Arch-Router-1.5B is the
  # canonical choice.
  classifier_model: arch-router-1.5b

  # Bounded LRU keyed on (case-folded, whitespace-trimmed) prompt - prompts
  # repeat in agent loops; the cache amortises the classifier round-trip
  # across them. 0 here means "use the default" (1024); the cache cannot be
  # disabled from YAML today.
  classifier_cache_size: 256

  # Softmax probability floor a label must clear to join the active label set.
  # 0 = use the package default (0.15). 0.40 is a better empirical
  # starting point on Arch-Router-1.5B - see the tuning note below.
  activation_threshold: 0.40

  # Used when no candidate covers the active label set, or the classifier
  # itself errors. Empty here = fail-fast with HTTP 500.
  fallback: qwen3-0.6b

  # The label vocabulary. Descriptions are fed verbatim into the
  # classifier's system prompt - short, action-oriented sentences work
  # best ("writing or debugging code", "small talk").
  policies:
    - label: code-generation
      description: writing, debugging, reading, or explaining code in any programming language
    - label: casual-chat
      description: small talk, greetings, jokes, or general conversation with no specific task
    - label: math-reasoning
      description: arithmetic, equations, percentage calculations, or step-by-step word problems

  # Routing table - order matters (smallest → largest). See "Score
  # classifier" above for the matching rule.
  candidates:
    - model: qwen3-0.6b
      labels: [casual-chat]
    - model: qwen_qwen3.5-2b
      labels: [code-generation, casual-chat, math-reasoning]

Tuning activation_threshold

The threshold is the single knob you’ll want to tune per (classifier-model, policy-set) pair. On Arch-Router-1.5B with the three-policy setup above, sweeping the threshold over a hand-labeled 30-prompt corpus produced:

The classifier’s argmax matches the dominant label 93% of the time on this corpus - what the threshold controls is how much secondary-label noise leaks into the active label set. Low thresholds push single-label queries to multi-label-capable (larger) candidates unnecessarily; 0.40 keeps the dominant label dominant without losing genuine compound activations.

Re-tune per (classifier-model, policy-set) pair. The /api/score endpoint (see below) is the convenient probe - it returns the raw length-normalized log-probabilities so you can sweep thresholds offline without driving real chat completions.

Embedding cache (L2)

Classification is the most expensive thing the middleware does. The score classifier already memo-caches verbatim repeats (case- and whitespace-folded prompt → decision); the embedding cache is the L2 tier that catches semantically similar prompts - “How do I exit vim?” and “i need to quit vim” can share a decision instead of running the classifier twice.

Pairs naturally with a larger / slower classifier model: the steady-state cost on cache hits collapses to one embedding round-trip plus a KNN search, both well under 100ms with nomic-embed-text-v1.5 + local-store.

Configuration

Add an embedding_cache: block to a router model:

router:
  classifier: score
  classifier_model: arch-router-1.5b
  policies: [...]
  candidates: [...]

  embedding_cache:
    embedding_model: nomic-embed-text-v1.5    # any loaded embedding model
    similarity_threshold: 0.80                # cosine sim floor for a hit (default 0.80)
    confidence_threshold: 0.60                # min top-label prob to cache a decision (default 0.60)
    # store_name: router-cache-smart-router   # optional override; defaults to "router-cache-<router>"

Omit the block entirely to disable. The cache adds two new failure modes (embedder unavailable, store unavailable) - both fall through to the inner classifier so routing keeps working.

How it works

For each request:

1. Embed the probe prompt via the configured embedding_model.
2. KNN top-1 against the per-router local-store collection.
3. If similarity ≥ similarity_threshold, return the cached decision (Cached=true, CacheSimilarity=<sim> in the decision log).
4. Miss → run the inner classifier. If decision.score >= confidence_threshold, insert (embedding, decision) into the store. Low-confidence decisions are deliberately skipped so they can’t poison future paraphrases.

The local-store collection is named router-cache-<router-model-name> by default — each router gets its own collection so two routers can’t cross-contaminate. The collection is in-memory only: local-store keeps no on-disk artefact, so the embedding cache starts empty on every restart and re-learns from live traffic. (The KNN classifier’s corpus does NOT have this limitation — its corpus file is the source of truth and re-indexes on startup; see “The KNN classifier” above.)

Tuning notes

• Similarity threshold: 0.80 is the package default - re-tune per (embedding model, corpus). The histogram on the Routing tab shows where the cosine distribution actually sits; pick a threshold above the cross-intent cluster and below the paraphrase cluster.
• Confidence threshold: 0.60 corresponds roughly to “the classifier is committed to a top label.” Don’t lower this - caching unsure decisions propagates the uncertainty.
• Cache flush: invalidates automatically when the router YAML changes (the classifier cache is fingerprinted by yaml.Marshal), but the underlying local-store collection still holds the old payloads. Manual flush via local-store admin or by renaming store_name if you need a hard reset.
• Latency budget: an embedding round-trip (typically 30-80ms for small embedding models) plus KNN search (~5ms) is added to every miss on top of the classifier latency. Cache hits skip the classifier entirely. Break-even is around 7-10% hit rate; agent loops with repeated phrasing easily exceed this.

Admin page

The /app/middleware page has a Routing tab listing every router model’s classifier, policies, candidates, and fallback. The Events tab shows the decision log - one row per classified request with correlation ID, requested model, served model, classifier name, active labels, top-label score, and latency.

Routing decisions are stored in an in-process ring buffer (default capacity 5,000). The decision log is for audit and tuning - the canonical usage log lives in /api/usage and correlates by request ID.

REST surface

MCP tools

Mutating the rest of the routing config — adding a candidate, changing the classifier model — goes through the model-config surface (edit_model_config / PATCH /api/models/config-json/:name); reload with POST /models/reload to pick up YAML edits without restarting.

Operational notes

• Reload after YAML edits. The router configs are loaded at startup and cached. POST /models/reload re-reads from disk; the next request rebuilds the classifier from the new config (the classifier cache is fingerprinted by yaml.Marshal(RouterConfig) so it invalidates automatically).
• Classifier latency on Arch-Router-1.5B Q4_K_M is ~500ms steady for 3 policies on Intel SYCL. The score primitive re-decodes the full prompt for every candidate today (the KV cache is cleared between candidates); the prompt-KV-sharing optimization is on the perf TODO list in backend/cpp/llama-cpp/grpc-server.cpp::Score. Until then, classifier_cache_size is the highest-leverage knob for repeat-query workloads (agent loops).
• Decision log size: 5,000-entry ring buffer per process. The log is in-process and not persisted - pair with the usage log for long-horizon audit.

Related features

• Cloud passthrough proxy - combine the router with proxy-* backends to send simple prompts to local models and complex ones to cloud providers.
• MITM proxy - apply the same PII filter to Claude Code, Codex CLI, and any HTTPS client without LocalAI holding their API keys.
• Authentication - admin role is required for mutating endpoints and the /app/middleware page; in no-auth single-user mode the synthetic local user has admin role automatically.

Creating a Decisions router in the UI

In Middleware → Routing → Create routing model, select Decisions (native probabilities) under Classifier. The Classifier Model picker lists installed, enabled native decision models explicitly declaring known_usecases: [decisions] on a backend supporting Score, including llama-cpp and vllm-cpp. NER-only models and routing dispatchers are not eligible. Selecting a model saves its exact configured name; it does not install weights.

Decisions needs no ChatML template and returns independent label probabilities, not an exclusive choice. Start with an activation threshold of 0.5 (zero uses the Decisions default of 0.5). Changing classifiers clears the dependent model selection but preserves your threshold, including the template’s initial 0.40; set it deliberately before saving. Existing saved selections reopen unchanged.

Cloud passthrough proxy

LocalAI can forward chat-completion and Anthropic Messages requests to an external provider instead of running them through the local gRPC backend pipeline. Configure a model with backend: cloud-proxy and a proxy.upstream_url, and LocalAI bypasses templating, MCP injection, and the local model loader entirely - the upstream sees the body the client sent (with only the top-level model field optionally rewritten).

The streaming PII filter still runs over the upstream’s SSE stream, so cloud egress remains subject to the same redaction rules a local model would apply.

When to use this

• Mix local and cloud models in the same LocalAI instance - clients hit one endpoint, LocalAI dispatches per model.
• Apply LocalAI’s auth, usage tracking, and PII redaction to cloud traffic before the body leaves the network.
• Use the intelligent router to send small or simple prompts to a local model and complex ones to Claude or GPT-4o.

To fall back to a local model when the upstream is down, list the proxy model in a failover chain.

How it works

1. Request hits LocalAI on /v1/chat/completions (OpenAI-shaped) or /v1/messages (Anthropic-shaped).
2. The standard auth and routing middleware runs.
3. Per-model PII redaction runs request-side as it would for any model.
4. The handler detects the cloud-proxy backend in passthrough mode and loads the cloud-proxy gRPC backend, which owns the outbound HTTP.
5. The backend POSTs the body to proxy.upstream_url with provider-aware authentication, then streams the SSE response back to core.
6. The streaming PII filter rewrites per-token text in flight; the upstream’s event names and metadata pass through unchanged.

Passthrough mode is wire-format-faithful - it does not translate request shapes between providers. A client posting an OpenAI-shaped body to an Anthropic upstream will get a confused upstream. Use the matching wire format, or switch to translate mode (below).

Configuration

The cloud-proxy backend has one knob - the provider it should authenticate against - and two modes:

proxy.provider selects the auth scheme and (in translate mode) the wire format. Supported values: openai, anthropic.

If the upstream requires an API key, configure one of api_key, api_key_env (an environment variable name), or api_key_file (a file path). If the upstream requires no API key, omit all three fields.

api_key is a plaintext administrator-managed credential, supported by both cloud-proxy and localai-proxy:

proxy:
  api_key: your-upstream-key

The model editor masks this input but preserves its actual value on unrelated saves. Masking is not encryption or vault storage: model YAML/JSON, admin configuration responses, exports, and backups can contain the plaintext key, just as they can contain model env values. Restrict access and protect files and backups. Use api_key_env or api_key_file for external secret management.

Direct keys require updated core and proxy backend binaries. In distributed installations, update workers and their proxy backends before frontends; older binaries can ignore this additive field and send unauthenticated requests. This does not add distributed model-environment propagation.

OpenAI passthrough

name: gpt-4o-proxy
backend: cloud-proxy

# When set, replaces the client's "model" field before forwarding.
# Useful when the LocalAI alias differs from the upstream's canonical name.
proxy:
  mode: passthrough
  provider: openai
  upstream_url: https://api.openai.com/v1/chat/completions
  api_key_env: OPENAI_API_KEY
  upstream_model: gpt-4o
  request_timeout_seconds: 120

# PII filtering defaults to ON for cloud-proxy backends. Override by setting
# pii.enabled: false explicitly. Per-pattern action overrides go in
# pii.patterns; see the Middleware admin page or the Middleware feature doc.
pii:
  enabled: true

Then start LocalAI with the API key in the environment:

export OPENAI_API_KEY=sk-...
local-ai run

Clients hit http://localhost:8080/v1/chat/completions with "model": "gpt-4o-proxy" and the request lands on OpenAI’s API.

Anthropic passthrough

name: claude-sonnet-proxy
backend: cloud-proxy

proxy:
  mode: passthrough
  provider: anthropic
  upstream_url: https://api.anthropic.com/v1/messages
  api_key_env: ANTHROPIC_API_KEY
  upstream_model: claude-3-5-sonnet-20241022
  request_timeout_seconds: 300

pii:
  enabled: true
  # Block - not just mask - leaked credentials before they reach the upstream.
  patterns:
    - id: api_key_prefix
      action: block

Anthropic clients hit http://localhost:8080/v1/messages with "model": "claude-sonnet-proxy".

OrcaRouter passthrough

OrcaRouter is a hosted OpenAI-compatible gateway that routes to many upstream models under a single key. It speaks the OpenAI chat-completions wire format, so point a cloud-proxy model at it with provider: openai:

name: orcarouter-auto
backend: cloud-proxy

proxy:
  mode: passthrough
  provider: openai
  upstream_url: https://api.orcarouter.ai/v1/chat/completions
  api_key_env: ORCAROUTER_API_KEY
  upstream_model: orcarouter/auto
  request_timeout_seconds: 300

pii:
  enabled: true

Then start LocalAI with the key in the environment:

export ORCAROUTER_API_KEY=sk-orca-...
local-ai run

Clients hit http://localhost:8080/v1/chat/completions with "model": "orcarouter-auto" and LocalAI forwards the request to OrcaRouter.

orcarouter/auto lets OrcaRouter pick the best model for each prompt. To pin a specific model, set upstream_model to its id from the OrcaRouter catalog (in provider/model-name form). Leave upstream_model empty to pass the client’s model field through unchanged.

Other OpenAI-compatible providers

Most third-party providers (Together, Groq, DeepInfra, OpenRouter, …) speak the OpenAI chat-completions wire format. Use provider: openai with the provider’s URL and, if required, its API key:

name: llama-3-70b-via-together
backend: cloud-proxy

proxy:
  mode: passthrough
  provider: openai
  upstream_url: https://api.together.xyz/v1/chat/completions
  api_key_env: TOGETHER_API_KEY
  upstream_model: meta-llama/Llama-3-70b-chat-hf

Images

LocalAI cannot ask an upstream which inputs its model accepts. A cloud-proxy model is text only until its configuration says that it can look at images:

name: gpt-4o-proxy
backend: cloud-proxy
known_usecases: [chat, vision]
proxy:
  mode: passthrough
  provider: openai
  upstream_url: https://api.openai.com/v1/chat/completions
  api_key_env: OPENAI_API_KEY
  upstream_model: gpt-4o

In the model editor, the Cloud Proxy section has the switch This model can look at images, which adds vision to known_usecases or removes it. Declare it only when the upstream model accepts images: LocalAI does not check.

With the declaration, /v1/models/capabilities lists the model with the vision capability and image input, and the chats send an attached image to the model inside the message, as an image_url part with a data: URL. Passthrough mode forwards the part as it is. Without it, the chats give the model the file path and the describe_image tool. When no other model can look at images, describe_image tries the cloud model anyway, since the upstream may accept them. If the upstream refuses, the tool tells the user to turn on the switch above, choose another model, or ask an administrator to set a Chat image model.

Translate mode sends the upstream the text of the messages only, so a translate-mode model never takes images, whatever it declares, and the editor does not show the switch for it.

Upstreams without an API key

For an OpenAI-compatible upstream that accepts requests without authentication, omit api_key, api_key_env, and api_key_file:

name: internal-chat-proxy
backend: cloud-proxy

proxy:
  mode: passthrough
  provider: openai
  upstream_url: http://inference.internal:8000/v1/chat/completions
  upstream_model: my-model

Replace the example URL and model name with your upstream’s values. LocalAI loads this configuration without resolving a key and adds no upstream Authorization header. This also applies to OpenAI-compatible upstreams in translate mode.

Omitting both fields differs from setting api_key_env to an empty or unset environment variable: the latter causes a backend load error.

LocalAI’s client authentication is separate. Clients must still authenticate to LocalAI when its authentication is enabled. LocalAI does not forward their Authorization header to the upstream.

An upstream without API keys can still require another authentication or payment protocol. Omitting these fields does not implement that protocol.

Translate mode

In translate mode the cloud-proxy backend converts LocalAI’s internal proto to the provider’s wire format. This lets a client speak one shape (e.g. OpenAI Chat Completions) against an upstream that expects another (e.g. Anthropic Messages).

name: claude-via-openai-clients
backend: cloud-proxy

proxy:
  mode: translate
  provider: anthropic
  upstream_url: https://api.anthropic.com/v1/messages
  api_key_env: ANTHROPIC_API_KEY
  upstream_model: claude-3-5-sonnet-20241022

Translate mode currently routes only pure-text completions - tool calls, image blocks, and per-request usage tokens are dropped through the internal Predict() signature. Use passthrough mode when your clients need the upstream’s full feature set.

In translate mode, an Anthropic response with stop_reason: "refusal" is returned to the client as an error instead of an empty successful reply, for both non-streaming and streaming requests. A streamed response may already have delivered partial content when the refusal arrives. Responses that end normally (end_turn) are unaffected, even when their content is empty.

For OpenAI-compatible upstreams, translate mode rewrites nullable type unions in tool schemas ("type": ["null", "array"]) to the plain type ("type": "array") before sending. LocalAI’s built-in tools mark every list and pointer parameter as nullable, and some providers validate tool schemas against a narrower dialect: Mistral, for example, rejects the whole request with Invalid tool schema: None is not of type 'array' (code 3260). Optional parameters stay optional through required. Unions with more than one non-null type keep those types. A required or enum that is null (an empty Go list, as on the built-in list_files tool) is left out for the same reason.

Anthropic prompt caching

proxy.cache_prompt: true makes the translator add Anthropic prompt-cache breakpoints (cache_control: {type: ephemeral}) to the stable prefix of every request: the system block, the last tool, and the final message block (at most three of Anthropic’s four allowed breakpoints). Anthropic then serves that repeated prefix at the cache-read rate (~0.1x input) on subsequent calls, which sharply cuts cost on agentic or multi-turn workloads that re-send a large, unchanging system-plus-tools prefix each turn.

The flag only applies with mode: translate and provider: anthropic; it has no effect in passthrough mode, for other providers, or when unset (the system field is then still emitted as a bare string).

name: claude-cached
backend: cloud-proxy

proxy:
  mode: translate
  provider: anthropic
  upstream_url: https://api.anthropic.com/v1/messages
  api_key_env: ANTHROPIC_API_KEY
  upstream_model: claude-3-5-sonnet-20241022
  cache_prompt: true

Loading secrets from a file

api_key_file is an alternative to api_key_env when your secret manager mounts keys as files (e.g. Kubernetes secrets, Docker secrets, Vault Agent):

proxy:
  api_key_file: /run/secrets/openai_api_key

The file is read at backend load time and trimmed of surrounding whitespace. api_key, api_key_env, and api_key_file are mutually exclusive.

Combining with the intelligent router

A router model can spread traffic across local and cloud candidates. The score classifier reads the policy descriptions and routes per request:

name: smart-router
router:
  classifier: score
  classifier_model: arch-router-1.5b
  fallback: qwen-3-7b-local
  activation_threshold: 0.40
  policies:
    - label: casual
      description: small talk, greetings, short answers
    - label: code
      description: writing or debugging code in any programming language
    - label: heavy-reasoning
      description: long-form analysis, complex math, multi-step reasoning
  candidates:
    - model: qwen-3-7b-local
      labels: [casual]
    - model: gpt-4o-proxy
      labels: [casual, code]
    - model: claude-sonnet-proxy
      labels: [casual, code, heavy-reasoning]

The router rewrites input.Model to the chosen candidate; per-model PII, ACLs, and the cloud-proxy fork all run against the resolved target.

See Middleware: PII filtering and intelligent routing for the full router and PII-filter reference.

Proxying to another LocalAI (localai-proxy)

cloud-proxy forwards chat and Messages requests only. To serve a model from another LocalAI instance for every API it has, use backend: localai-proxy. The backend receives the request from the local pipeline like any other backend and sends it to the REST API of the upstream LocalAI. Because it is a normal backend, a localai-proxy model can be a stage of a realtime pipeline or a target of a failover chain.

name: remote-llm
backend: localai-proxy
known_usecases: [chat]
proxy:
  # Base URL of the upstream LocalAI. Do not add /v1 or an endpoint path:
  # the backend adds the path for each API.
  upstream_url: https://argus.lan:8080
  # The model name on the upstream. When empty, the name of this config.
  upstream_model: gemma-3-12b
  # Optional. The upstream API key, from an environment variable
  # (or api_key_file). Sent as "Authorization: Bearer <key>".
  api_key_env: ARGUS_API_KEY
  # Optional. Time limit for each non-streaming request. Streams have no limit.
  request_timeout_seconds: 120

A model that does live transcription in a realtime pipeline also names a realtime pipeline on the upstream. The backend opens a transcription session on the upstream /v1/realtime endpoint with that pipeline:

name: remote-stt
backend: localai-proxy
known_usecases: [transcript]
options:
  - realtime_pipeline:asr-pipeline
proxy:
  upstream_url: https://argus.lan:8080
  upstream_model: parakeet

Set known_usecases on every localai-proxy model. Failover uses it to match targets, and LocalAI cannot guess the usecases of a remote model. For a chat model, known_usecases: [chat] has one more effect: LocalAI sends the chat messages to the upstream /v1/chat/completions endpoint, and the upstream applies its own chat template, tool parsing and reasoning parsing. Without chat, or when the config has its own templates, LocalAI renders the prompt locally and sends it to /v1/completions. proxy.mode and proxy.provider have no effect on this backend.

Supported APIs:

• Text: chat and completions (also streamed), embeddings, rerank, tokenize, detokenize, score.
• Audio: TTS (also streamed), sound generation, transcription (also streamed), live transcription (with realtime_pipeline), diarization, VAD, sound classification, audio transformations.
• Image, video and 3D: image generation, upscaling, video generation, 3D generation and animation. The backend downloads the files that the upstream generates.
• Vision: object detection, depth, face verification and analysis, voice verification, analysis and embeddings.
• Stores: set, get, delete, find.

Methods that have no REST API on the upstream return the gRPC error Unimplemented (“localai-proxy: has no upstream counterpart”). The upstream returning 501 Not Implemented maps to the same code. Both mean a capability gap, not a broken target: audio encoding and decoding, audio-to-audio streams, token classification (PII NER), model metadata, fine-tuning, quantization and model export fall in this bucket. A failover chain skips a target that returns Unimplemented and tries the next target, but does not mark the target down. This applies to every API, also to the APIs that report Unimplemented to the client as HTTP 501 (images, video, 3D, detection, depth, face and voice).

Errors from the upstream: a 5xx response (other than 501) or a connection failure becomes Unavailable, and a failover chain marks the target down. A 4xx response becomes InvalidArgument, and LocalAI returns it to the client without a retry or a trip — except 429 Too Many Requests, which becomes ResourceExhausted: the request itself is fine, the upstream is just out of capacity, so a failover chain retries it on the next target and trips the rate-limited one, moving traffic off it until it recovers.

Known limits:

• Voice-profile paths pass through unresolved. When LocalAI resolves a TTS voice to a local file (for example a voice clone reference), the backend sends that path to the upstream, where it does not exist. Use voices that the upstream knows by name.
• Depth exports are not supported. The upstream writes them to its own disk, so a depth request with exports or a destination file returns Unimplemented. Depth maps and points without exports work.
• The REST transcription API has no end-of-utterance (eou) flag, so transcriptions through the proxy never set it. Live transcription through realtime_pipeline sets eou at the end of each utterance.
• Sound generation from a source audio file is not supported.
• Chat and completions do not forward grammars, so JSON mode and other grammar-constrained output are not enforced by the upstream. Images, audio and video attached to messages are not forwarded either. The backend logs a warning for each request that loses one of these fields.
• Streamed TTS cannot detect an upstream synthesis failure that ends the stream cleanly. The client receives the audio produced so far as a complete response, and a failover chain does not retry it. A stream that is cut off is reported as an error.
• upstream_url is the root of the upstream server. If it has a /v1 path, the backend removes /v1 and everything after it, and logs a warning.

Limitations

• Passthrough does no wire-shape translation. Use mode: translate (with the constraints documented above) or send requests that match the upstream’s format.
• No output-side PII for non-streaming responses. Streaming responses are filtered in flight; buffered responses pass through verbatim. Request-side PII covers both.
• No retry or backoff. Transient upstream failures bubble up to the client as 502 Bad Gateway.
• No request shape validation. If the upstream rejects the body, its error envelope is forwarded to the client unchanged.
• No capability detection. LocalAI cannot tell whether the upstream model accepts images. Declare vision in known_usecases when it does (see Images).

Operational notes

• Cloud-proxy backends load like any other gRPC backend - they consume one process per loaded model and appear in the backend management view, but they hold no GPU memory.
• Usage stats and the trace log capture cloud-proxy requests like any other request. Token counts come from the upstream’s usage field when present.
• Set request_timeout_seconds defensively - a hung upstream otherwise ties up an HTTP handler until the client disconnects.

MITM proxy for Claude Code / Codex CLI

LocalAI can act as a local HTTPS proxy that redacts PII from your Claude Code, OpenAI Codex CLI, or any HTTPS client without holding their API keys. The proxy intercepts only the LLM API endpoints you allowlist (default: api.anthropic.com, api.openai.com); everything else - OAuth, telemetry, package fetches - passes through as a plain TCP tunnel.

Use this when:

• You want to use Claude Code with a Claude Pro/Max subscription but still apply the same PII redaction LocalAI applies to API-key traffic.
• You run Codex CLI on a corporate laptop and need an audit trail of prompts.
• You want LocalAI to enforce egress policies for AI traffic without becoming the API endpoint clients talk to.

The proxy is off by default. Operators opt in by setting --mitm-listen and distributing the generated CA cert.

How it works

1. The proxy generates a private CA on first start (persisted to disk).
2. Clients set HTTPS_PROXY=http://localai:port and add the CA to their trust store (e.g. NODE_EXTRA_CA_CERTS for Node-based CLIs like Claude Code and Codex).
3. The CLI sends CONNECT api.anthropic.com:443 to the proxy.
4. For allowlisted hosts, the proxy mints a per-host leaf cert signed by the CA, terminates TLS, parses the HTTP request, applies the global PII redactor on /v1/messages or /v1/chat/completions, and forwards to the real upstream over its own TLS connection.
5. The streaming SSE response runs through the same pii.StreamFilter the cloud-proxy backend uses.
6. For non-allowlisted hosts, the proxy is a plain CONNECT tunnel - no TLS termination, no inspection, no CA trust required.

The CLI authenticates with its own subscription / API key as it normally would. LocalAI never holds the credential - it just observes and rewrites the request body.

Quick start

Start LocalAI with the MITM listener:

local-ai run --mitm-listen :8443

The first start generates a CA at <data-path>/mitm-ca/{ca.crt,ca.key}. Restarting reloads the same CA so clients keep trusting it.

Download the public CA cert:

curl -O http://localhost:8080/api/middleware/proxy-ca.crt

Configure Claude Code to use the proxy and trust the cert:

export HTTPS_PROXY=http://localhost:8443
export NODE_EXTRA_CA_CERTS=$(pwd)/proxy-ca.crt
claude

Now any claude chat session that touches api.anthropic.com/v1/messages gets its prompts and tool inputs scanned by LocalAI’s PII filter, and any PII the model emits in its streaming response is masked before reaching your terminal. Events appear in the LocalAI middleware admin page under Filtering → Recent events.

The same works for Codex CLI - set HTTPS_PROXY and NODE_EXTRA_CA_CERTS and run codex.

Configuration

The proxy is enabled with two startup settings:

There is no global intercept-hosts flag. The hosts whose TLS is terminated and scanned are declared per model, in the model YAML mitm.hosts: block. Each model that names one or more hosts owns those hosts; everything not listed by any model tunnels through untouched. A cloud-proxy model that should intercept Anthropic traffic looks like:

name: claude
backend: cloud-proxy
mitm:
  hosts:
    - api.anthropic.com

Hostnames are case-insensitive. Add custom upstreams (e.g. an OpenAI-compatible third-party provider) by adding their hostname to a model’s mitm.hosts: list and ensuring their endpoint paths match /v1/chat/completions or /v1/messages. You can create these models from the Add Model UI.

What gets redacted

The MITM proxy runs the same PII detection as the regular request middleware. Detection is NER-based (a token-classification detector model), not a fixed regex list: the older pattern tier has been removed. See /docs/administration/middleware/ for how detector models, entity groups, and the mask / block actions are configured, and for the instance-wide default detector.

A block action returns HTTP 400 with error.type=pii_blocked to the client. The CLI sees the rejection and shows it as a request error.

Events are persisted via the same pii.EventStore the rest of LocalAI uses, so the /api/pii/events endpoint and the middleware admin page include MITM events alongside direct-API events.

Security notes

• The CA private key is the master credential. Anyone with read access to <data-path>/mitm-ca/ca.key can forge TLS for any host the proxy could intercept. The file is mode 0600; keep it that way.
• The proxy listener accepts plaintext HTTP CONNECT requests - bind it to localhost (--mitm-listen 127.0.0.1:8443) unless you’ve added auth in front of the listener. There is no built-in API-key check on this port.
• The MITM CA is separate from any TLS cert LocalAI’s main HTTP API uses. Installing the MITM CA grants trust only for traffic that flows through this proxy.
• The proxy does not pin upstream certificates; it trusts the system certificate store. If your machine’s trust store is compromised, the proxy is too.
• TLS termination negotiates HTTP/2 by default (ALPN h2) and falls back to HTTP/1.1 for clients that don’t speak h2. Modern CLIs (Claude Code, Codex) and the Anthropic / OpenAI APIs all use h2.

Limitations

• Only /v1/messages and /v1/chat/completions get redacted. Other paths on the same host (OAuth, model listing) are forwarded verbatim.
• No request-shape translation. The proxy assumes the request body matches the host’s wire format; cross-shape forwarding is the cloud proxy backend’s job, not the MITM’s.
• No CA rotation in the MVP. To rotate, delete ca.key and ca.crt and re-distribute the new cert to every client.
• Cert pinning kills MITM. Neither Claude Code nor Codex CLI pins certificates today, but a future SDK update could. If a CLI starts refusing the proxied handshake, that’s the signal.

Comparison with the cloud-proxy backend

LocalAI ships two cloud-related proxy modes; pick by who holds the credential:

For shared deployments where LocalAI owns the API key and clients are unsophisticated (curl, simple webapps), use the cloud-proxy backend. For “give my Claude Code a privacy filter” use cases, use the MITM proxy.

Distributed Mode

Distributed mode enables horizontal scaling of LocalAI across multiple machines using PostgreSQL for state and node registry, and NATS for real-time coordination. Unlike the P2P/federation approach, distributed mode is designed for production deployments and Kubernetes environments where you need centralized management, health monitoring, and deterministic routing.

Architecture Overview

Frontends are stateless LocalAI instances that receive API requests and route them to worker nodes via the SmartRouter. All frontends share state through PostgreSQL and coordinate via NATS.

Workers are generic processes that self-register with a frontend. They don’t have a fixed backend type - the SmartRouter dynamically installs the required backend via NATS backend.install events when a model request arrives.

Scheduling Algorithm

The SmartRouter uses idle-first scheduling with preemptive eviction:

1. If the model is already loaded on a node → use it (per-model gRPC address)
2. Drop any node without room to store the model on its models filesystem (see Disk headroom)
3. If no node has the model → prefer nodes with enough free VRAM
4. Fall back to idle nodes (zero models), then least-loaded nodes
5. If no node has capacity → evict the least-recently-used model with zero in-flight requests to free a node
6. If all models are busy → wait (with timeout) for a model to become idle, then evict
7. Send backend.install NATS event with backend name + model ID → worker starts a new gRPC process on a dynamic port
8. SmartRouter calls gRPC LoadModel on the model-specific port, records in DB

Each model gets its own gRPC backend process, so a single worker can serve multiple models simultaneously (e.g., a chat model and an embedding model).

Prerequisites

• PostgreSQL (with pgvector extension recommended for RAG) - used for node registry, job store, auth, and shared state
• NATS server - used for real-time backend lifecycle events and file staging
• All services must be on the same network (or reachable via configured URLs)

Quick Start with Docker Compose

The easiest way to try distributed mode locally is with the provided Docker Compose file:

docker compose -f docker-compose.distributed.yaml up

This starts PostgreSQL, NATS, a LocalAI frontend, and one worker node. When you send an inference request, the SmartRouter automatically installs the needed backend on the worker and loads the model. See the file for details on adding GPU support, shared volumes, and additional workers.

Frontend Configuration

The frontend is a standard LocalAI instance with distributed mode enabled. These flags are added to the local-ai run command:

Add-ons

Every local-ai run process is its own add-on owner by default. With several frontend replicas, run the owner as one separate workload with its own volume, and start the frontends with --addons-role=frontend, so they forward add-on requests to it and hold no add-on state. The library sources and the backend library installs also go through the owner: a frontend follows the owner’s sources and installs backends on its own node. See Running the owner as its own workload.

The owner must join distributed mode with the same --nats-url, registration token and PostgreSQL database as the frontends. Then it takes part in the leader locks and sends model loads to the workers. An owner without these settings loads models in its own process (backend not found) and polls the connectors next to the leader frontend (Slack messages answered twice). See The owner in distributed mode.

The model load deadline scales with the checkpoint

The LoadModel deadline starts after the backend is installed and the model files are staged, so it covers only the worker backend’s own checkpoint read and pipeline init. That work is proportional to the bytes on disk, which makes any fixed deadline a model-size cliff rather than a timeout: a 70 GB video checkpoint on a Jetson Thor worker failed reproducibly against the old fixed 5m default (rpc error: code = DeadlineExceeded after 953.5s of wall clock, roughly 11m of which was backend install and staging), and simply raising the constant would only move the cliff to the next larger model while making a genuinely wedged small model hang for the whole inflated duration.

So the deadline is derived per model:

budget = 5m + 20s per GiB of checkpoint,  capped at 6h

The per-GiB rate is deliberately pessimistic — it corresponds to reading weights at about 54 MB/s, below what any supported storage sustains — because the two errors are not symmetric: a budget that is too long costs only failure latency on a load that was going to fail anyway, while a budget that is too short causes a guaranteed false failure on a load that was perfectly healthy.

The size is measured from the model files on the frontend’s disk, over the same set of paths that get staged to the worker. If those files are not present locally — a backend handed a bare HuggingFace repo id fetches its own weights on the worker — there is nothing to measure and the budget stays at the plain 5m default. Pin LOCALAI_NATS_MODEL_LOAD_TIMEOUT for those models if their load is slow.

When the budget is exceeded, the error names the budget, the checkpoint size it was derived from, and the knob that overrides it, instead of surfacing a bare context deadline exceeded.

The cold-load lock ceiling

The router also bounds how long a single cold load may hold the per-model advisory lock, so a worker that dies mid-install cannot pin every other replica’s request for that model. That bound is derived, not configured, and it is based on progress rather than on wall-clock time.

The load starts with a base budget of max(backend-install-timeout + model-load-timeout + 5m, 25m) — with the defaults, 15m + 5m + 5m = 25m. That budget covers the steps that report no progress: node selection, backend install, and the remote LoadModel call. Raising either timeout widens it in step, so a longer load deadline is never clipped.

That base is the hold’s starting budget, not its maximum. Because the derived load budget above can exceed it — a 70 GB checkpoint’s 28m20s against a 25m base — the hold is widened again as the router enters the load phase, by the derived budget plus the same 5m of slack. Without that step the ceiling would cancel a load that was still comfortably inside its own deadline.

While model files are staging, however, the deadline extends every time staging does real work, and expires only once staging has been silent for a 5-minute stall window. Real work means uploaded bytes, and also the resumable-upload verify phase: when a shard is already present on the worker from an earlier attempt, the frontend HEADs it and hashes the local copy to confirm it matches, then skips the transfer. That phase uploads nothing at all — on a 70 GB model resuming with 56 GB already staged it ran for six-plus consecutive minutes at ~45s per shard — so hashing counts as progress too. Otherwise a resumed transfer would be mistaken for a wedged one. Staging time is a function of checkpoint size and available bandwidth, not a constant: a 70 GB model at 26 MB/s needs about 45 minutes, and a 600 GB checkpoint needs hours. A fixed ceiling would therefore be a model-size cliff — every increase just moves the cliff to the next larger model. Extending on progress means a large model transfers for as long as it legitimately needs, while a worker that dies mid-transfer still releases the lock within the stall window.

An absolute cap of 24h ends the hold even if progress keeps arriving, so a degenerate peer trickling a few bytes at a time cannot pin the lock forever. No configuration is needed for either value; both are sized well above any legitimate transfer.

Requests for a model that is still loading

A cold load in distributed mode is a long-running background job: install the backend, stage multi-GB model files to the worker, then load the checkpoint. Staging a 35.7 GB GGUF onto a fresh worker takes roughly twenty minutes on a fast LAN — far longer than any HTTP request can be held open.

So the load does not run on the request. The first request for an unloaded model claims a durable model load job — that claim takes milliseconds and is the only part that holds the per-model advisory lock — and the job then runs in the background on the frontend replica that claimed it. Every other request for the same model, on any replica, attaches to that job as a waiter:

• It is served the moment the model is ready, with no client-side retry. A model already 90% staged usually needs no second request.
• It never starts a duplicate load and never blocks on the database lock. (Before this split, concurrent requests blocked on pg_advisory_lock for the whole load and were killed by the PostgreSQL role’s statement_timeout — SQLSTATE 57014 — so from the operator’s seat the model simply never loaded.)
• If the load fails, the waiter gets the real cause (worker out of disk), not an anonymous timeout.
• If the client disconnects, the load keeps going. It belongs to the job record, not to the request.
• The owner of a job holds a 30 second lease, which it renews on every heartbeat. The database clock decides whether a lease has expired, so a frontend with a wrong clock cannot expire a live lease or keep a dead one. If an owner cannot renew for a whole lease, it stops its own load.
• If an owner dies, its job is marked failed once the lease runs out, with or without a new request. A request that arrives then reads the cause. The model is held for a 2.5 minute stop window, because remote work may still run. After that the next request starts a new attempt. No manual cleanup is needed.
• A failure that is known to have ended the remote work (an error from the backend, or a failure before a node was chosen) is kept for 15 seconds only, so every waiter reads the same cause and the next request can retry.
• Each attempt has its own generation. If a job is replaced, the old owner notices at its next heartbeat and stops its load. Its late writes to the job and to the replica table are rejected.
• If the job table cannot be read, a cold load fails instead of running without a job. Models that are already loaded keep serving, because routing to a loaded replica does not read the job table.

When the wait budget (LOCALAI_MODEL_LOAD_WAIT, default 60s) runs out, the request is answered with 503, a Retry-After header, and a body that says exactly where the load is:

{
  "error": {
    "message": "model Qwen3.6-27B-MTP-GGUF is staging on node nvidia-thor (41%, ETA ~11m)",
    "type": "model_loading",
    "code": "model_loading"
  },
  "loading": {
    "model": "Qwen3.6-27B-MTP-GGUF",
    "state": "staging",
    "node": "nvidia-thor",
    "progress": 41.2,
    "bytes_sent": 14730000000,
    "total_bytes": 35776484480,
    "file_index": 1,
    "total_files": 2,
    "eta_seconds": 660
  }
}

The error envelope keeps OpenAI clients working unchanged; loading is additive, so a client that understands it renders progress instead of an error. eta_seconds is derived from the job’s own observed transfer rate and is omitted rather than guessed until enough bytes have moved for that rate to mean anything — a confidently wrong ETA on a twenty-minute wait is worse than none. state is one of pending (choosing a node), installing, staging (transferring files) or loading (the worker is reading the checkpoint).

The chat UI renders this state inline and retries automatically once the model reports ready. Poll GET /api/models/{id}/load-status for the same loading object at any time.

The worker bounds the work it runs

The frontend owns the job row, but the real work runs in a backend process on a worker. The worker therefore watches each load too. A load is an operation named by the job’s generation:

• The install request carries the operation id and the longest the load may run, as a duration, so a worker clock that is wrong changes nothing. The backend starts in its own process group.
• The frontend renews the operation every few seconds and completes it when the load finishes. The worker kills the whole process group when no renewal arrives for 90 seconds, or when the deadline passes. A backend that already reports READY is never killed: a lost completion message must not destroy a model that serves.
• A stop names the operation, the process key and, when known, the address and process instance. The worker refuses unless they all match its own records. There is no fallback to “any running backend”. A stop for a load that already finished leaves the serving model alone.
• A worker that is killed cannot stop its backends. It records each backend’s process group in a small file under its data directory, and the next worker kills every group listed there before it serves (Linux; the start time of the leader guards against a recycled pid). A new incarnation, reported on the next heartbeat, then confirms the failed loads on that node. The worker also skips a gRPC port that something already listens on, and a readiness answer only counts while the worker’s own backend process is alive.
• If the worker answers a renewal with “unknown operation” three times in a row, the frontend fails the load at once instead of waiting for the load budget. The worker lost the operation, so the work is gone.

When a load fails after remote work may have started (a timeout, a cancel, a lost lease), the owner stops the operation immediately. An acknowledged stop shortens the hold to the 15 second report window. A silent worker keeps the hold at the stop window (2.5 minutes), and the reconciler retries the stop on every pass until the worker answers or the window ends. Nothing needs manual cleanup.

A model held by a failed job answers 503 with Retry-After set to the seconds until the hold ends, and the real cause in the body.

Cancelling a load

POST /api/models/{id}/load-cancel (admin only) cancels one load attempt. The body names the exact attempt, as GET /api/models/{id}/load-status reports it:

{"job_id": "0b6e4a3c-5c1d-4d52-8f0a-0f3c9e0b8f11"}

The call is idempotent. A repeat retries the stop and never extends the hold. A load that has not been placed on a node yet can be cancelled too. Unloading a model on a node, draining a node, and removing a node all cancel the loads placed there through the same stop path, and an unload still unloads the loaded replicas. The replica rows of a cancelled attempt are removed as soon as the worker confirms the stop. The cancel_model_load tool of the assistant calls the same service.

load-status also reports job_id, lease_expires_in, cancel_requested, last_error, stopping, stop_deadline and retry_after. A database error is a 503, never an empty answer.