Backend Monitor

LocalAI provides endpoints to monitor and manage running backends. The /backend/monitor endpoint reports the status and resource usage of loaded models, /backend/load pre-loads a model into memory, and /backend/shutdown allows stopping a model’s backend process.

All three are admin-only.

Monitor API

• Method: GET
• Endpoints: /backend/monitor, /v1/backend/monitor

Request

The model to monitor is passed as a query parameter:

For backwards compatibility, a JSON body with the same field is still accepted when the model query parameter is not set, but new clients should use the query parameter.

Response

Returns a JSON object with the backend status:

If the gRPC status call fails, the endpoint falls back to local process metrics:

Usage

curl "http://localhost:8080/backend/monitor?model=my-model"

Example response

{
  "state": 2,
  "memory": {
    "total": 1073741824,
    "breakdown": {
      "weights": 536870912,
      "kv_cache": 268435456
    }
  }
}

Load API

Pre-loads a model into memory ahead of its first request, so that request pays no cold-start load cost. It is the inverse of the Shutdown API and works for any model, not just realtime pipelines.

• Method: POST
• Endpoints: /backend/load, /v1/backend/load

Request

Behavior

• For a regular model, its own backend is loaded.
• For a realtime pipeline model (a config with a pipeline: block), every configured sub-model (VAD, transcription, LLM, TTS, sound_detection, voice_recognition) is loaded concurrently instead of the pipeline stub, which has no backend of its own.

The call blocks until loading finishes and reports which model names became resident, so partial failures are visible.

Usage

curl -X POST http://localhost:8080/backend/load \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model"}'

Example response

{ "loaded": ["my-model"], "message": "model loaded" }

On failure the call returns 500 with loaded listing whichever sub-models did load and message naming the failures.

Shutdown API

• Method: POST
• Endpoints: /backend/shutdown, /v1/backend/shutdown

Request

Usage

curl -X POST http://localhost:8080/backend/shutdown \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model"}'

Response

Returns 200 OK with the shutdown confirmation message on success.

Error Responses

Middleware: PII filtering and intelligent routing

LocalAI ships a request-middleware layer that sits between the HTTP API and the backend dispatcher. Two subsystems share that layer because they share the same lifecycle hook: PII filtering scans the request body before it reaches a backend, and the intelligent router rewrites input.Model so a single client-facing model name fans out across multiple downstream targets.

Both are inspected and configured from the same admin page (/app/middleware), backed by the same REST surface (/api/middleware/*, /api/pii/*, /api/router/*) and the same MCP tools.

Request lifecycle

client ── auth ── route-model ── per-model PII ── backend ── client
                       │              │
                       │              └─── event log
                       └─── decision log

The router runs first (it picks the target model so per-model PII has something to gate on), per-model PII runs next (gated by the resolved config), and the backend executes. Filtering is request-side only - the request body is scanned and rewritten before forwarding; the response is not touched (NER over a streamed response is left as a follow-up). Each subsystem writes to its own admin-visible log: /api/router/decisions for routing, /api/pii/events for redaction and block actions.

PII filtering

PII redaction is NER-based and runs request-side (input). It is off by default, flipping to on for any cloud-proxy backend because that traffic crosses the network to a third-party provider. Pick a default detector so those models are actually scanned. Explicit pii.enabled in a model’s YAML always wins over the backend default.

Filtering runs on every text-accepting endpoint that has an adapter wired: /v1/chat/completions and /v1/messages (chat), /v1/completions, /v1/embeddings, /v1/edits, and the Ollama /api/chat, /api/generate and /api/embed endpoints, plus the MITM proxy request body. Image, audio (TTS/STT), video, rerank, and the realtime WebSocket are not filtered yet (different prompt-PII semantics; realtime is not HTTP middleware).

A request’s messages are scanned as one document (joined in order), so the NER detector keeps conversational context: whether 4421 is a PIN or jdoe_42 is a username is usually decided by the question asked in the previous message, and a bidirectional encoder only sees that context when the messages share a forward pass. Detected spans are mapped back to the individual message they fall in, so redaction still rewrites each message field in place and events carry message-local offsets.

The earlier regex pattern tier (pii.patterns, the built-in pattern catalogue, --pii-config, the /api/pii/patterns|test|decide endpoints) and response/streaming-side redaction have been removed. Detection is now driven entirely by token-classification (NER) models. Legacy keys no-op with a startup warning.

Detector models

A detector is a token_classify model (e.g. an openai-privacy-filter GGUF) that carries the detection policy in a top-level pii_detection: block - defined once, on the model itself:

name: privacy-filter-multilingual
backend: privacy-filter
embeddings: true              # TOKEN_CLS pooling
known_usecases:
  - token_classify
pii_detection:
  min_score: 0.5              # drop detections below this confidence
  default_action: mask        # applied to any detected group with no entry
  entity_actions:             # which PII to block vs mask vs allow-log
    PASSWORD: block
    CREDITCARD: block
    EMAIL: mask

mask rewrites the matched span to [REDACTED:ner:<GROUP>] in the request body before forwarding. block returns HTTP 400 (error.type=pii_blocked) without forwarding. allow detects and logs (a PIIEvent is still recorded) but leaves the text unchanged. The entity-group names are whatever the model emits (the privacy-filter family uses uppercase names like EMAIL, PASSWORD, CREDITCARD).

Pattern detector tier

NER is the wrong tool for high-entropy, highly-regular secrets - API keys, tokens, private-key blocks. A trained NER model has no “API key” class, so it fragments a key into the nearest categories it does know and can leave the secret part exposed. Those secrets are exactly what a regex catches cheaply.

A pattern detector is a detector model (backend: pattern) that matches secrets with a restricted regex subset compiled to Go’s RE2 engine - linear-time, no backtracking, no ReDoS. It runs entirely in-process: no model download, no backend, zero VRAM. Install the gallery’s secret-filter for a ready-made set, or define your own:

name: secret-filter
backend: pattern
known_usecases: [token_classify]        # so it appears in the detector picker
pii_detection:
  default_action: block                 # a leaked credential shouldn't leave
  builtins:                             # built-in catalogue (enable by name)
    - anthropic_api_key
    - openai_api_key
    - github_token
    - aws_access_key
    - private_key_block
  patterns:                             # operator-defined, restricted subset
    - name: INTERNAL_TOKEN
      match: "tok-[A-Za-z0-9]{32,64}"
      action: block                      # optional per-pattern override
      min_len: 36                        # optional length floor

A match is reported under its group (built-in group name, or the pattern name), so entity_actions / default_action apply exactly as for NER.

The restricted grammar (validated at load - an invalid pattern is rejected, not silently ignored):

• Allowed: literals, character classes […] and \w \d \s, alternation, anchors ^ $ \b, and quantifiers ? * + {m,n}.
• Rejected: . (any-char), capturing groups, and {n,m} bounds over 4096.
• Required anchor: every pattern must contain a fixed literal run of at least 3 characters (e.g. sk-ant-, ghp_, AKIA). This admits real key shapes but rejects open-ended ones - an email or a bare \w+ has no such anchor and belongs to the NER tier.

Use both tiers together: reference an NER detector and a pattern detector in a model’s pii.detectors (or as instance defaults); their hits union, and a block from either rejects the request.

Consuming models

Any model opts in by enabling PII and referencing one or more detectors - no per-consumer policy:

name: claude-strict
backend: cloud-proxy
proxy:
  mode: passthrough
  provider: anthropic
  upstream_url: https://api.anthropic.com/v1/messages
  api_key_env: ANTHROPIC_API_KEY
pii:
  enabled: true               # default-on for cloud-proxy; explicit for audit
  detectors:
    - privacy-filter-multilingual

Multiple detectors union their detections; overlapping spans resolve to the strongest action (block > mask > allow). A configured detector that can’t be loaded fails the request closed (HTTP 503, error.type=pii_ner_unavailable) rather than silently skipping the check. The same NER path runs on the MITM proxy request body for intercepted hosts. Response/output redaction is out of scope for now.

Instance-wide default detector

The Detector models table on the Middleware → Filtering page lists every token_classify detector model (neural NER models and in-process pattern matchers alike) and exposes a per-row Default toggle. Toggling a detector on adds it to the instance-wide default detector set - one or more models applied to any PII-enabled model that names none of its own pii.detectors. It is persisted through POST /api/settings and read live, so a change takes effect on the next request without a restart. A default that names a model no longer loaded still appears (marked not loaded) so it can be toggled off.

The default set can also be supplied out-of-band with the LOCALAI_PII_DEFAULT_DETECTORS environment variable (comma-separated model names, e.g. privacy-filter-nemotron,secret-filter). When set it takes precedence over the value persisted via the UI (env > file), which is the right behaviour for immutable container deployments that pin filtering policy at boot rather than via the admin UI.

This is what makes cloud-proxy / MITM redaction work out of the box: those backends default to PII-enabled but ship no detector list, so without a default detector the filter runs with nothing to scan. Set one here and cloud-proxy traffic is scanned with no per-model config.

Resolution precedence (the single decision point is ResolvePIIPolicy, shared by the chat middleware and the MITM listener so both agree):

1. An explicit pii.enabled on the model wins - true or false.
2. Otherwise PII is on if the backend defaults it on (cloud-proxy).
3. Detectors are the model’s own pii.detectors; if it lists none, the instance-wide default detector(s) are used.

A model that resolves enabled but ends up with no detector at all (a cloud-proxy model with no model detectors and no instance default) scans nothing - set a default detector to close that gap.

Admin page

The /app/middleware page (admin role only) has four tabs - Filtering, Routing, MITM Proxy (see the MITM doc), and Events. The Filtering tab has a Detector models table (every token_classify filter model, with the per-row Default toggle above and an edit link to each detector’s config, plus an Add detector model button) and a per-model table listing only the models PII can actually apply to - chat / completion / embeddings / edit consumers and cloud-proxy models, not VAD/STT/image models or the detector models themselves. Each row reports the effective enabled state as an inline toggle - flipping it writes an explicit pii.enabled to that model’s YAML (a server-side deep-merge that preserves pii.detectors and every other field), so a cloud-proxy model shown on by backend default can be turned off, and vice-versa - plus the resolved detector(s) - with a (default) marker when they come from the instance-wide default rather than the model’s YAML - why it is on (YAML / backend default), and the recent event count. Detection policy (entity→action, min score) is still edited on each detector model’s config (Models → edit → PII), not globally.

Analyze / redact API

The same detection pipeline is also exposed as a standalone service, so a client can scan or sanitise a string without routing a full chat request through it (the inline path above). Two endpoints, both requiring a normal API key (the pii_filter feature - not admin):

• POST /api/pii/analyze - detect only. Returns the matched entity spans (entity_type, source ner|pattern, start/end, score, action) and a blocked flag, without modifying the text.
• POST /api/pii/redact - apply the configured policy. Returns redacted_text (with masked spans replaced by [REDACTED:<id>]) and masked; when a block action fires it returns 400 with type: pii_blocked and the offending entities - never a redacted body.

Both take the same request: text plus a detector selection - either explicit detector model names in detectors, or a consuming model whose effective policy is used: the model’s own pii.detectors, else the instance-wide default detectors, exactly as the inline filter resolves them. A model with PII disabled - or enabled but with no detector anywhere - is a 400: the inline filter would scan nothing for it, and the API says so rather than implying a clean scan. The detection policy lives on the detector models exactly as for the inline filter. The raw matched value is never returned (an admin may pass reveal: true to include the audit hash_prefix).

text is scanned as a single document. To reproduce the inline filter’s conversation-context behaviour for multi-message content, join the messages with blank lines into one text - NER detection quality depends on that context (a bare 4421 is nothing; after “what are the last four digits of your card?” it is a PIN).

# Redact with an explicit pattern/NER detector
curl -sX POST http://localhost:8080/api/pii/redact \
  -H 'Authorization: Bearer $API_KEY' -H 'Content-Type: application/json' \
  -d '{"text":"reach me at jane@acme.io","detectors":["my-ner-model"]}'
# => {"redacted_text":"reach me at [REDACTED:ner:EMAIL]","masked":true,...}

# Analyze using a consuming model's configured detectors
curl -sX POST http://localhost:8080/api/pii/analyze \
  -H 'Authorization: Bearer $API_KEY' -H 'Content-Type: application/json' \
  -d '{"text":"sk-ant-api03-…","model":"gpt-4"}'
# => {"entities":[{"entity_type":"ANTHROPIC_KEY","source":"pattern",...,"action":"block"}],"blocked":true}

Calls are audited in the same event log, tagged with an origin of pii_analyze / pii_redact (the inline filter records middleware, the MITM proxy records proxy), so GET /api/pii/events?origin=pii_redact shows just the redact-API rows.

REST surface

MCP tools

The same surface is mirrored through the LocalAI Assistant MCP server:

Detection policy is part of a detector model’s config, so it is managed through the model-config tools (edit_model_config), not a dedicated PII tool.

Intelligent routing

A router model is a model whose YAML carries a router: block. When a client addresses it ("model": "smart-router"), the middleware classifies the prompt, picks a downstream candidate model, rewrites input.Model to the candidate, and the standard model-resolution path runs against that resolved target. ACL checks, disabled-state, and per-model PII all apply to the resolved model - the router does model selection only.

Depth-1 invariant

Candidates must not themselves be router models. A smart-router → claude-strict → cloud-proxy chain is fine (claude-strict is a regular cloud-proxy model). A smart-router → other-router → real-model chain is rejected at runtime by the middleware (the dispatcher returns HTTP 500 with a depth-1 invariant error). This keeps the dispatch graph acyclic and predictable.

Fallback

If no candidate’s label set covers the active label set from the classifier, or the classifier errors out, the router uses cfg.Router.Fallback. An empty fallback causes the dispatch to fail with HTTP 500 rather than silently routing somewhere unintended - fail-fast, not silent-bypass.

Available classifiers

LocalAI ships two classifier implementations. Pick one with classifier: in the router YAML:

Both classifiers share the same YAML shape: classifier_model, policies, candidates, fallback, activation_threshold, classifier_cache_size, and the optional embedding_cache block.

The Score classifier

The score classifier works like this:

1. Build a Qwen/ChatML system prompt that lists every policy label with its description and primes the model to emit a label as the assistant turn.
2. Ask the classifier model to score every policy label as the first-token(s) continuation. This uses the Score gRPC primitive (backend.proto::Score), which returns per-candidate log-probabilities length-normalized so candidates of unequal token length stay comparable.
3. Softmax the length-normalized log-probabilities into a probability distribution over labels.
4. Threshold the distribution: every label whose probability passes activation_threshold joins the active label set.
5. Pick the FIRST candidate whose Labels is a superset of the active set. Admins order candidates smallest → largest so a single-label query routes to the smallest capable model, while a query that activates multiple labels falls to a candidate that covers them all.

This is the Arch-Router approach extended for multi-label. The distribution carries more signal than the argmax - reading off the spread lets one prompt activate multiple policies and route to a model capable of all of them.

Recommended classifier model

Arch-Router-1.5B is the canonical choice. It’s a Qwen-2.5-1.5B-Instruct base trained specifically on routing-policy continuation, so the ChatML system-prompt

• label-continuation pattern produces well-separated label probabilities without prompt tuning. The Q4_K_M GGUF runs on CPU, GPU, and Intel SYCL.

The classifier model must support the Score gRPC primitive (today: the llama-cpp and vLLM backends) and use the ChatML chat template. Any small ChatML instruct model works under those constraints, but expect flatter probability distributions which translate to a higher activation_threshold to keep noise out of the active label set.

On llama-cpp, scoring rides the server’s task queue alongside generation and embeddings, so the classifier may share a model config with chat/completion/embeddings - a dedicated scorer model is no longer required. Repeated calls with the same prompt also reuse the prompt’s KV cache across candidates.

The Colbert classifier

The colbert classifier reranks each policy description against the prompt via the rerankers backend and activates the labels whose relevance scores clear activation_threshold (default 0.5 for reranker-style scores in [0, 1]).

router:
  classifier: colbert
  classifier_model: bge-m3-colbert  # gallery entry; loads BAAI/bge-m3 in ColBERT mode
  activation_threshold: 0.5
  policies:
    - label: code-generation
      description: writing, debugging, reading, or explaining code
    - label: casual-chat
      description: small talk, greetings, jokes
  candidates: [...]

The reranker scores the description (natural English) rather than asking a small LM to score the label as a next-token continuation, so it tends to be more robust when policy labels are abstract slugs (compliance-review, tier-2-support). The trade-off is one reranker round-trip per request - bge-m3 in ColBERT mode is fast enough on GPU that this is comparable to the Score path for most workloads. The embedding_cache block applies identically.

The reranker model’s type: (in the model YAML) selects which underlying scoring head loads - colbert for late-interaction MaxSim, cross-encoder for cross-attention scoring. The classifier itself is indifferent; pick the head that fits your latency / quality budget.

YAML reference

name: smart-router
known_usecases:
  - chat
router:
  # `score` (Arch-Router-style next-token scoring) or `colbert`
  # (rerank policy descriptions). See "Available classifiers" above.
  classifier: score

  # A model loaded by LocalAI that supports the Score gRPC primitive
  # (llama-cpp and vLLM ship implementations). Arch-Router-1.5B is the
  # canonical choice.
  classifier_model: arch-router-1.5b

  # Bounded LRU keyed on (case-folded, whitespace-trimmed) prompt - prompts
  # repeat in agent loops; the cache amortises the classifier round-trip
  # across them. 0 here means "use the default" (1024); the cache cannot be
  # disabled from YAML today.
  classifier_cache_size: 256

  # Softmax probability floor a label must clear to join the active label set.
  # 0 = use the package default (0.15). 0.40 is a better empirical
  # starting point on Arch-Router-1.5B - see the tuning note below.
  activation_threshold: 0.40

  # Used when no candidate covers the active label set, or the classifier
  # itself errors. Empty here = fail-fast with HTTP 500.
  fallback: qwen3-0.6b

  # The label vocabulary. Descriptions are fed verbatim into the
  # classifier's system prompt - short, action-oriented sentences work
  # best ("writing or debugging code", "small talk").
  policies:
    - label: code-generation
      description: writing, debugging, reading, or explaining code in any programming language
    - label: casual-chat
      description: small talk, greetings, jokes, or general conversation with no specific task
    - label: math-reasoning
      description: arithmetic, equations, percentage calculations, or step-by-step word problems

  # Routing table - order matters (smallest → largest). See "Score
  # classifier" above for the matching rule.
  candidates:
    - model: qwen3-0.6b
      labels: [casual-chat]
    - model: qwen_qwen3.5-2b
      labels: [code-generation, casual-chat, math-reasoning]

Tuning activation_threshold

The threshold is the single knob you’ll want to tune per (classifier-model, policy-set) pair. On Arch-Router-1.5B with the three-policy setup above, sweeping the threshold over a hand-labeled 30-prompt corpus produced:

The classifier’s argmax matches the dominant label 93% of the time on this corpus - what the threshold controls is how much secondary-label noise leaks into the active label set. Low thresholds push single-label queries to multi-label-capable (larger) candidates unnecessarily; 0.40 keeps the dominant label dominant without losing genuine compound activations.

Re-tune per (classifier-model, policy-set) pair. The /api/score endpoint (see below) is the convenient probe - it returns the raw length-normalized log-probabilities so you can sweep thresholds offline without driving real chat completions.

Embedding cache (L2)

Classification is the most expensive thing the middleware does. The score classifier already memo-caches verbatim repeats (case- and whitespace-folded prompt → decision); the embedding cache is the L2 tier that catches semantically similar prompts - “How do I exit vim?” and “i need to quit vim” can share a decision instead of running the classifier twice.

Pairs naturally with a larger / slower classifier model: the steady-state cost on cache hits collapses to one embedding round-trip plus a KNN search, both well under 100ms with nomic-embed-text-v1.5 + local-store.

Configuration

Add an embedding_cache: block to a router model:

router:
  classifier: score
  classifier_model: arch-router-1.5b
  policies: [...]
  candidates: [...]

  embedding_cache:
    embedding_model: nomic-embed-text-v1.5    # any loaded embedding model
    similarity_threshold: 0.80                # cosine sim floor for a hit (default 0.80)
    confidence_threshold: 0.60                # min top-label prob to cache a decision (default 0.60)
    # store_name: router-cache-smart-router   # optional override; defaults to "router-cache-<router>"

Omit the block entirely to disable. The cache adds two new failure modes (embedder unavailable, store unavailable) - both fall through to the inner classifier so routing keeps working.

How it works

For each request:

1. Embed the probe prompt via the configured embedding_model.
2. KNN top-1 against the per-router local-store collection.
3. If similarity ≥ similarity_threshold, return the cached decision (Cached=true, CacheSimilarity=<sim> in the decision log).
4. Miss → run the inner classifier. If decision.score >= confidence_threshold, insert (embedding, decision) into the store. Low-confidence decisions are deliberately skipped so they can’t poison future paraphrases.

The local-store collection is named router-cache-<router-model-name> by default - each router gets its own collection so two routers can’t cross-contaminate. Collections persist on disk (local-store is the canonical persistent vector backend), so the cache survives restarts.

Tuning notes

• Similarity threshold: 0.80 is the package default - re-tune per (embedding model, corpus). The histogram on the Routing tab shows where the cosine distribution actually sits; pick a threshold above the cross-intent cluster and below the paraphrase cluster.
• Confidence threshold: 0.60 corresponds roughly to “the classifier is committed to a top label.” Don’t lower this - caching unsure decisions propagates the uncertainty.
• Cache flush: invalidates automatically when the router YAML changes (the classifier cache is fingerprinted by yaml.Marshal), but the underlying local-store collection still holds the old payloads. Manual flush via local-store admin or by renaming store_name if you need a hard reset.
• Latency budget: an embedding round-trip (typically 30-80ms for small embedding models) plus KNN search (~5ms) is added to every miss on top of the classifier latency. Cache hits skip the classifier entirely. Break-even is around 7-10% hit rate; agent loops with repeated phrasing easily exceed this.

Admin page

The /app/middleware page has a Routing tab listing every router model’s classifier, policies, candidates, and fallback. The Events tab shows the decision log - one row per classified request with correlation ID, requested model, served model, classifier name, active labels, top-label score, and latency.

Routing decisions are stored in an in-process ring buffer (default capacity 5,000). The decision log is for audit and tuning - the canonical usage log lives in /api/usage and correlates by request ID.

REST surface

MCP tools

Mutating routing config - adding a candidate, changing the classifier model - is YAML-only today; reload with POST /models/reload to pick up edits without restarting.

Operational notes

• Reload after YAML edits. The router configs are loaded at startup and cached. POST /models/reload re-reads from disk; the next request rebuilds the classifier from the new config (the classifier cache is fingerprinted by yaml.Marshal(RouterConfig) so it invalidates automatically).
• Classifier latency on Arch-Router-1.5B Q4_K_M is ~500ms steady for 3 policies on Intel SYCL. The score primitive re-decodes the full prompt for every candidate today (the KV cache is cleared between candidates); the prompt-KV-sharing optimization is on the perf TODO list in backend/cpp/llama-cpp/grpc-server.cpp::Score. Until then, classifier_cache_size is the highest-leverage knob for repeat-query workloads (agent loops).
• Decision log size: 5,000-entry ring buffer per process. The log is in-process and not persisted - pair with the usage log for long-horizon audit.

Related features

• Cloud passthrough proxy - combine the router with proxy-* backends to send simple prompts to local models and complex ones to cloud providers.
• MITM proxy - apply the same PII filter to Claude Code, Codex CLI, and any HTTPS client without LocalAI holding their API keys.
• Authentication - admin role is required for mutating endpoints and the /app/middleware page; in no-auth single-user mode the synthetic local user has admin role automatically.

Cloud passthrough proxy

LocalAI can forward chat-completion and Anthropic Messages requests to an external provider instead of running them through the local gRPC backend pipeline. Configure a model with backend: cloud-proxy and a proxy.upstream_url, and LocalAI bypasses templating, MCP injection, and the local model loader entirely - the upstream sees the body the client sent (with only the top-level model field optionally rewritten).

The streaming PII filter still runs over the upstream’s SSE stream, so cloud egress remains subject to the same redaction rules a local model would apply.

When to use this

• Mix local and cloud models in the same LocalAI instance - clients hit one endpoint, LocalAI dispatches per model.
• Apply LocalAI’s auth, usage tracking, and PII redaction to cloud traffic before the body leaves the network.
• Use the intelligent router to send small or simple prompts to a local model and complex ones to Claude or GPT-4o.

How it works

1. Request hits LocalAI on /v1/chat/completions (OpenAI-shaped) or /v1/messages (Anthropic-shaped).
2. The standard auth and routing middleware runs.
3. Per-model PII redaction runs request-side as it would for any model.
4. The handler detects the cloud-proxy backend in passthrough mode and loads the cloud-proxy gRPC backend, which owns the outbound HTTP.
5. The backend POSTs the body to proxy.upstream_url with provider-aware authentication, then streams the SSE response back to core.
6. The streaming PII filter rewrites per-token text in flight; the upstream’s event names and metadata pass through unchanged.

Passthrough mode is wire-format-faithful - it does not translate request shapes between providers. A client posting an OpenAI-shaped body to an Anthropic upstream will get a confused upstream. Use the matching wire format, or switch to translate mode (below).

Configuration

The cloud-proxy backend has one knob - the provider it should authenticate against - and two modes:

proxy.provider selects the auth scheme and (in translate mode) the wire format. Supported values: openai, anthropic.

API keys are loaded from either an environment variable (api_key_env) or a file (api_key_file). The key never appears in the config file or the admin UI; pick whichever fits your secret-management setup.

OpenAI passthrough

name: gpt-4o-proxy
backend: cloud-proxy

# When set, replaces the client's "model" field before forwarding.
# Useful when the LocalAI alias differs from the upstream's canonical name.
proxy:
  mode: passthrough
  provider: openai
  upstream_url: https://api.openai.com/v1/chat/completions
  api_key_env: OPENAI_API_KEY
  upstream_model: gpt-4o
  request_timeout_seconds: 120

# PII filtering defaults to ON for cloud-proxy backends. Override by setting
# pii.enabled: false explicitly. Per-pattern action overrides go in
# pii.patterns; see the Middleware admin page or the Middleware feature doc.
pii:
  enabled: true

Then start LocalAI with the API key in the environment:

export OPENAI_API_KEY=sk-...
local-ai run

Clients hit http://localhost:8080/v1/chat/completions with "model": "gpt-4o-proxy" and the request lands on OpenAI’s API.

Anthropic passthrough

name: claude-sonnet-proxy
backend: cloud-proxy

proxy:
  mode: passthrough
  provider: anthropic
  upstream_url: https://api.anthropic.com/v1/messages
  api_key_env: ANTHROPIC_API_KEY
  upstream_model: claude-3-5-sonnet-20241022
  request_timeout_seconds: 300

pii:
  enabled: true
  # Block - not just mask - leaked credentials before they reach the upstream.
  patterns:
    - id: api_key_prefix
      action: block

Anthropic clients hit http://localhost:8080/v1/messages with "model": "claude-sonnet-proxy".

Other OpenAI-compatible providers

Most third-party providers (Together, Groq, DeepInfra, OpenRouter, …) speak the OpenAI chat-completions wire format. Use provider: openai with the provider’s URL and API key:

name: llama-3-70b-via-together
backend: cloud-proxy

proxy:
  mode: passthrough
  provider: openai
  upstream_url: https://api.together.xyz/v1/chat/completions
  api_key_env: TOGETHER_API_KEY
  upstream_model: meta-llama/Llama-3-70b-chat-hf

Translate mode

In translate mode the cloud-proxy backend converts LocalAI’s internal proto to the provider’s wire format. This lets a client speak one shape (e.g. OpenAI Chat Completions) against an upstream that expects another (e.g. Anthropic Messages).

name: claude-via-openai-clients
backend: cloud-proxy

proxy:
  mode: translate
  provider: anthropic
  upstream_url: https://api.anthropic.com/v1/messages
  api_key_env: ANTHROPIC_API_KEY
  upstream_model: claude-3-5-sonnet-20241022

Translate mode currently routes only pure-text completions - tool calls, image blocks, and per-request usage tokens are dropped through the internal Predict() signature. Use passthrough mode when your clients need the upstream’s full feature set.

Loading secrets from a file

api_key_file is an alternative to api_key_env when your secret manager mounts keys as files (e.g. Kubernetes secrets, Docker secrets, Vault Agent):

proxy:
  api_key_file: /run/secrets/openai_api_key

The file is read at backend load time and trimmed of surrounding whitespace. api_key_env and api_key_file are mutually exclusive.

Combining with the intelligent router

A router model can spread traffic across local and cloud candidates. The score classifier reads the policy descriptions and routes per request:

name: smart-router
router:
  classifier: score
  classifier_model: arch-router-1.5b
  fallback: qwen-3-7b-local
  activation_threshold: 0.40
  policies:
    - label: casual
      description: small talk, greetings, short answers
    - label: code
      description: writing or debugging code in any programming language
    - label: heavy-reasoning
      description: long-form analysis, complex math, multi-step reasoning
  candidates:
    - model: qwen-3-7b-local
      labels: [casual]
    - model: gpt-4o-proxy
      labels: [casual, code]
    - model: claude-sonnet-proxy
      labels: [casual, code, heavy-reasoning]

The router rewrites input.Model to the chosen candidate; per-model PII, ACLs, and the cloud-proxy fork all run against the resolved target.

See Middleware: PII filtering and intelligent routing for the full router and PII-filter reference.

Limitations

• Passthrough does no wire-shape translation. Use mode: translate (with the constraints documented above) or send requests that match the upstream’s format.
• No output-side PII for non-streaming responses. Streaming responses are filtered in flight; buffered responses pass through verbatim. Request-side PII covers both.
• No retry or backoff. Transient upstream failures bubble up to the client as 502 Bad Gateway.
• No request shape validation. If the upstream rejects the body, its error envelope is forwarded to the client unchanged.

Operational notes

• Cloud-proxy backends load like any other gRPC backend - they consume one process per loaded model and appear in the backend management view, but they hold no GPU memory.
• Usage stats and the trace log capture cloud-proxy requests like any other request. Token counts come from the upstream’s usage field when present.
• Set request_timeout_seconds defensively - a hung upstream otherwise ties up an HTTP handler until the client disconnects.

MITM proxy for Claude Code / Codex CLI

LocalAI can act as a local HTTPS proxy that redacts PII from your Claude Code, OpenAI Codex CLI, or any HTTPS client without holding their API keys. The proxy intercepts only the LLM API endpoints you allowlist (default: api.anthropic.com, api.openai.com); everything else - OAuth, telemetry, package fetches - passes through as a plain TCP tunnel.

Use this when:

• You want to use Claude Code with a Claude Pro/Max subscription but still apply the same PII redaction LocalAI applies to API-key traffic.
• You run Codex CLI on a corporate laptop and need an audit trail of prompts.
• You want LocalAI to enforce egress policies for AI traffic without becoming the API endpoint clients talk to.

The proxy is off by default. Operators opt in by setting --mitm-listen and distributing the generated CA cert.

How it works

1. The proxy generates a private CA on first start (persisted to disk).
2. Clients set HTTPS_PROXY=http://localai:port and add the CA to their trust store (e.g. NODE_EXTRA_CA_CERTS for Node-based CLIs like Claude Code and Codex).
3. The CLI sends CONNECT api.anthropic.com:443 to the proxy.
4. For allowlisted hosts, the proxy mints a per-host leaf cert signed by the CA, terminates TLS, parses the HTTP request, applies the global PII redactor on /v1/messages or /v1/chat/completions, and forwards to the real upstream over its own TLS connection.
5. The streaming SSE response runs through the same pii.StreamFilter the cloud-proxy backend uses.
6. For non-allowlisted hosts, the proxy is a plain CONNECT tunnel - no TLS termination, no inspection, no CA trust required.

The CLI authenticates with its own subscription / API key as it normally would. LocalAI never holds the credential - it just observes and rewrites the request body.

Quick start

Start LocalAI with the MITM listener:

local-ai run --mitm-listen :8443

The first start generates a CA at <data-path>/mitm-ca/{ca.crt,ca.key}. Restarting reloads the same CA so clients keep trusting it.

Download the public CA cert:

curl -O http://localhost:8080/api/middleware/proxy-ca.crt

Configure Claude Code to use the proxy and trust the cert:

export HTTPS_PROXY=http://localhost:8443
export NODE_EXTRA_CA_CERTS=$(pwd)/proxy-ca.crt
claude

Now any claude chat session that touches api.anthropic.com/v1/messages gets its prompts and tool inputs scanned by LocalAI’s PII filter, and any PII the model emits in its streaming response is masked before reaching your terminal. Events appear in the LocalAI middleware admin page under Filtering → Recent events.

The same works for Codex CLI - set HTTPS_PROXY and NODE_EXTRA_CA_CERTS and run codex.

Configuration

The proxy is enabled with two startup settings:

There is no global intercept-hosts flag. The hosts whose TLS is terminated and scanned are declared per model, in the model YAML mitm.hosts: block. Each model that names one or more hosts owns those hosts; everything not listed by any model tunnels through untouched. A cloud-proxy model that should intercept Anthropic traffic looks like:

name: claude
backend: cloud-proxy
mitm:
  hosts:
    - api.anthropic.com

Hostnames are case-insensitive. Add custom upstreams (e.g. an OpenAI-compatible third-party provider) by adding their hostname to a model’s mitm.hosts: list and ensuring their endpoint paths match /v1/chat/completions or /v1/messages. You can create these models from the Add Model UI.

What gets redacted

The MITM proxy runs the same PII detection as the regular request middleware. Detection is NER-based (a token-classification detector model), not a fixed regex list: the older pattern tier has been removed. See /features/middleware/ for how detector models, entity groups, and the mask / block actions are configured, and for the instance-wide default detector.

A block action returns HTTP 400 with error.type=pii_blocked to the client. The CLI sees the rejection and shows it as a request error.

Events are persisted via the same pii.EventStore the rest of LocalAI uses, so the /api/pii/events endpoint and the middleware admin page include MITM events alongside direct-API events.

Security notes

• The CA private key is the master credential. Anyone with read access to <data-path>/mitm-ca/ca.key can forge TLS for any host the proxy could intercept. The file is mode 0600; keep it that way.
• The proxy listener accepts plaintext HTTP CONNECT requests - bind it to localhost (--mitm-listen 127.0.0.1:8443) unless you’ve added auth in front of the listener. There is no built-in API-key check on this port.
• The MITM CA is separate from any TLS cert LocalAI’s main HTTP API uses. Installing the MITM CA grants trust only for traffic that flows through this proxy.
• The proxy does not pin upstream certificates; it trusts the system certificate store. If your machine’s trust store is compromised, the proxy is too.
• TLS termination negotiates HTTP/2 by default (ALPN h2) and falls back to HTTP/1.1 for clients that don’t speak h2. Modern CLIs (Claude Code, Codex) and the Anthropic / OpenAI APIs all use h2.

Limitations

• Only /v1/messages and /v1/chat/completions get redacted. Other paths on the same host (OAuth, model listing) are forwarded verbatim.
• No request-shape translation. The proxy assumes the request body matches the host’s wire format; cross-shape forwarding is the cloud proxy backend’s job, not the MITM’s.
• No CA rotation in the MVP. To rotate, delete ca.key and ca.crt and re-distribute the new cert to every client.
• Cert pinning kills MITM. Neither Claude Code nor Codex CLI pins certificates today, but a future SDK update could. If a CLI starts refusing the proxied handshake, that’s the signal.

Comparison with the cloud-proxy backend

LocalAI ships two cloud-related proxy modes; pick by who holds the credential:

For shared deployments where LocalAI owns the API key and clients are unsophisticated (curl, simple webapps), use the cloud-proxy backend. For “give my Claude Code a privacy filter” use cases, use the MITM proxy.