Paired clients: inference and add-ons

A paired client is any program or box that is not a person in a browser and that the owner paired with a LocalAI instance: a small always-on unit with its own front end, a script, a service. One pairing gives the client one key, and the key can do two things:

• Inference. The client calls the standard OpenAI-style API with its own key, for the models that the owner chose. See Inference with a client key.
• Add-on calls. The client talks to an add-on that the owner installed. The rest of this page is for people who write that add-on. It describes the routes of the client, the event stream and the host operations.

For add-on calls, the add-on does the work. LocalAI checks who the client is, what it may reach and how much it may send, and forwards the request.

Pair a client

The owner makes a one-time pairing code, and the client sends it to LocalAI. Pairing needs sign-in to be on. With sign-in off every caller is already the local owner, so a client credential would add nothing and the routes answer 409.

A code has 10 characters, for example 7KQ3M-X9PTD. It expires after 10 minutes by default and after one hour at most. It works once. LocalAI stores only its hash. An owner can hold 5 unused codes. The pairing route allows 10 tries a minute for each address. There is no limit for the whole instance, so nobody can block the pairing of other owners. A text that is not shaped like a code gets 400 and does not count as a try. A wrong, an expired and a used code get the same answer.

What a client key can do

A client key is a row of the API key table with the kind client. It is not a user key. It has the role of an ordinary user, even if an administrator made the code. It can reach only the routes that name one of the features of its list, and it is refused everywhere else, also on routes that need no special feature. A client key cannot start a session, create other keys or reach the add-on routes of the web UI.

A code without a list gets client_self. A key with only client_inference cannot read GET /api/client/self: add client_self when the client should learn its models and the version of the instance.

Which add-ons a client can reach

A key with client_addons reaches only the add-ons that the owner chose. The code carries the list in addons (at most 16 ids). The list is required when client_addons is in features, and it is refused when it is not, so every code that opens add-ons says which ones. Each id must be a well-formed add-on id. LocalAI does not check that the add-on is installed, because it can be installed after the pairing. The list is copied to the key when the client pairs. GET /api/client/pairing-codes, GET /api/client/keys and GET /api/client/self show it.

The rule is deny by default. A request to /api/client/addons/{addon}/... for an add-on that is not in the list of the key gets 403. The answer is the same when the add-on does not exist, so a client cannot learn which add-ons are installed. A key with client_addons and an empty list reaches nothing. The owner can change the list of a paired client with PUT /api/client/keys/{id}; the client does not have to pair again, and it cannot change its own list.

Client features are not user permissions. They do not appear in the user settings.

Inference with a client key

A key with client_inference calls these routes of the standard API, with Authorization: Bearer <client key>, and no others:

The list is fixed. Embeddings, voice activity detection, voice and face recognition, diarization, sound generation, audio transformations, MCP, the responses API, realtime calls and sessions, and every admin route are refused to a client key, also when the owner could use them. A route that is added to LocalAI later is not added to this list by itself: a test freezes it.

Which models a client can use

The key carries a list of model names, like the list of add-ons. The code carries it in models (at most 32 names). It is required when client_inference is in features, and it is refused when it is not. It is copied to the key when the client pairs, and the owner changes it with PUT /api/client/keys/{id}.

The rule is deny by default:

• A request must name a model, and the model must be in the list. The check runs in the handler, on the name that the handler resolved from the request, so the model that runs is the model that was checked. An earlier check reads the path, the query, a JSON body and a form field, and it refuses a request that it cannot read without doubt: a body that is not JSON, a urlencoded form or a multipart form (415), data after the JSON value, a model field twice or in another case, or a model given twice in the query (400). A request that names no model gets 400: a client key has no default model. A model that is not in the list gets 403.
• GET /v1/models names no model. It answers with the models of the list that the key can still use.
• For a realtime session, the list is checked for the pipeline that the client connects with (?model=). The stages of the pipeline are not checked one by one. A session.update that names another pipeline or another transcription model is refused, so the session stays on the model it was checked for.
• A model on the list must exist (it has a model configuration), and it must have no tools attached: no MCP servers, no automatic MCP, no agent settings, and the same for the models that an alias, a failover chain, a router (its candidates and its fallback) or a pipeline hands the request to. A model with tools would run them under the account of the owner. The owner gets 400 when a listed model does not fit. The check runs again on every request, so a model that gains tools after the pairing is refused with 403 and is dropped from GET /v1/models.

A client key never sees the saved voice profiles of the owner. A voice profile reference answers 404, also for a profile that is shared with the whole instance, and a voice that is a path is refused. A voice of the model itself works. A pipeline whose own configuration names a saved profile as its voice cannot start for a client.

The chat and transcription requests of a client key are recorded in the usage of the owner, with the id of the key. The other routes (speech, /tts, sound classification and realtime sessions) are not recorded in the usage yet. The user quota does not apply to a client key, and neither do the feature switches and the model allow-list of the owner account: a key works only while its owner is an active administrator, so a demoted owner stops its clients. There is no limit per key on the number of requests or on how many run at once; that is planned.

A realtime session ends when the key is lost

A realtime session lasts minutes. LocalAI asks every 5 seconds whether the key still stands, and it sends an error event (client_key_invalid) and closes the connection when the key is revoked, has expired or is paused, when the owner account is not active or is no longer an administrator, when the model of the session left the list of the key, or when that model gained tools. A change of the list therefore reaches an open session within about 5 seconds, and a new request at once. A check that fails (the store errors, or does not answer within 5 seconds) does not close the session at once: it closes after three failed checks in a row.

TLS and the pinned certificate

A pairing code and a client key cross the network, so a client should pair over TLS. LocalAI serves TLS when it is started with a certificate (--tls-cert and --tls-key) or with --tls-self-signed; see the CLI reference. It is off by default, and TLS 1.3 is the minimum.

With --tls-self-signed, LocalAI makes a certificate once, keeps it in the tls folder of the data path and logs its SHA-256 fingerprint at each start. The answers that make and list pairing codes carry that fingerprint (tlsFingerprint, 64 hex digits) and a short tlsDigest of the first 8 digits in two groups, for example AB12-CD34. The owner can read the digest to the person at the client, or put the fingerprint in the pairing offer. The client pins the certificate: it accepts only a certificate with that fingerprint, before it sends the code. A self-signed certificate is then as strong as any other for that client, since the names inside it do not matter.

If LocalAI runs behind a proxy that ends TLS, the answers have no fingerprint, because LocalAI does not know the certificate of the proxy.

The routes of an add-on

The add-on declares the routes in manifest.json, under capabilities.client. This needs featureLevel 7 and capabilities.service.

{
  "featureLevel": 7,
  "capabilities": {
    "service": true,
    "hostOperations": { "required": ["client.jobs.start", "client.jobs.get", "client.jobs.cancel"] },
    "client": {
      "routes": [
        { "id": "start",  "method": "POST", "path": "/jobs", "maxBodyBytes": 4096 },
        { "id": "job",    "method": "GET",  "path": "/jobs/{id}" },
        { "id": "events", "method": "GET",  "path": "/events", "stream": true }
      ]
    }
  }
}

At most 32 routes. Two routes that match the same request are refused.

A client calls /api/client/addons/{addon id}{path} with its key. LocalAI forwards the request only if the method and the path match a declared route:

• The path is matched as the client sent it. A percent escape, a backslash, a dot segment, an empty segment and a trailing slash match nothing.
• The body is read with the limit of the route, and it must be application/json. A larger body gets 413.
• A wrong method gets 405. An unknown route gets 404.
• A client has at most 8 calls and 2 streams at once, and 240 calls a minute.
• The request goes over the loopback connection of the add-on with the per-start bearer. The client never sees an address, and it cannot choose one.

The add-on gets a ClientRequest with the route id, the method, the path, the variables, up to 16 query values, the body, the Last-Event-ID and the client:

{ "operationId": "cl-9f2c...", "routeId": "job", "method": "GET", "path": "/jobs/ab12",
  "params": { "id": "ab12" }, "client": { "id": "<client key id>", "name": "kitchen" } }

The add-on never gets the key of the client. It gets no user id and no role.

The add-on answers with a status, a content type and a body. The status must be one of 200, 201, 202, 204, 400, 404, 409, 410, 413, 422 or 429. The content type must be application/json or text/plain; charset=utf-8. The body is at most 1 MiB. LocalAI passes no other header of the add-on. Another status or a failed call becomes a 502, 503 or 504.

The runtime protocol has two new routes, POST /client and POST /client/stream. An add-on built with the SDK of this repository implements contract.ClientRuntime. The reference add-on is in addons/internal/clientdemo.

The event stream

A route with "stream": true answers with text/event-stream. LocalAI:

• sends the headers at once and flushes every chunk, and turns off gzip for the path;
• writes a comment line (: hb) every 15 seconds when the stream is quiet, never in the middle of an event;
• ends the upstream stream when the client goes away;
• ends the stream after one hour, and when the client key is revoked, expired or paused (checked at each heartbeat).

The add-on keeps the outbox. A good outbox gives each event an id that only grows, also after a restart, and an expiry. When a client reconnects, it sends the last id it saw in Last-Event-ID, and the add-on sends what came after it. If events are gone, send one notice event so that the client asks the state of its jobs. The reference add-on uses the events job.progress, job.done, job.failed and notice, keeps at most 100 events for each client for one hour, and watches at most 8 jobs for each client.

Host operations for jobs

A job of a client runs a task that the owner saved (see Agent tasks and jobs). The add-on names the task and fills its parameters. It cannot send a prompt of its own.

The owner must grant the add-on the operations as for any host operation (for example the prefix client. in the host policy), and must hold the mcp_jobs feature.

The client is not an argument. LocalAI records each forwarded call under the owner and the client key. The add-on passes the operationId of the call to the host with each operation, and LocalAI finds the client in its own record. During a call that a client made, the add-on can use the client. operations and no others. A stream call lasts as long as the stream, so the add-on can ask for the state of a job while the client is connected. Each operation checks that the key is still paired.

A client sees only the jobs that were started for it. It may run 3 jobs at once. Ended jobs do not count against this limit.

Which tools a job can use

A job that is started for a client reads and reports only by default. A tool that writes or acts gets a plain refusal as its result, also when the agent of the task holds the grant.

The owner can lift this with the setting client_jobs_allow_all_tools (POST /api/settings). It is off by default. With it on, a client job may use the tools that write or act, within the grants of the agent of its task: the same grants an ordinary job of that task has. It never gets a grant that the agent does not hold, so a task without an agent still cannot write or act. The setting applies to jobs that are already running. It does not change the jobs that other people start.

Limits to know

• All limits are per process. The tries of the pairing route, the calls and streams of a client, and the rate of a client are counted in the memory of one LocalAI process. With several frontends, each one has its own count, so the limits multiply. The job quota of a client is shared (it is counted in the job store, under a lock that all frontends share), and so is the limit of 32 paired clients. POST /api/client/pair works on any frontend, while the routes of an add-on work only on the instance that owns the add-ons.
• A client stream uses one of the 64 long calls of the add-on. The same pool serves the connector polls of the add-on, so 32 clients with 2 streams each fill it. Keep the number of paired clients of such an add-on low, or lower MaxStreamsPerClient.
• Client calls are capped inside the declared concurrency. A call to a route that does not stream may run for up to a minute, so all clients together may have half of the short call slots of the add-on in use (at least one). A client call takes one of those slots, so the add-on never gets more calls at once than it declared, and the tool calls of chats keep the other half. A call past that limit gets a 503.
• Revoking a client ends its open streams at the next check, stops its running jobs and frees its place. The host cannot tell an add-on to forget a client. An add-on should remove the outbox of a client that has been silent for a long time; the reference add-on removes files untouched for a week, and looks for them at most once an hour, when a client asks for its outbox. The job records of the client stay until the normal retention removes them. A key that expires is not a revoke: the jobs it started end at their own time.
• A stream also ends when the owner account is disabled.
• Add-ons and the client routes run on the instance that owns the add-ons. A frontend of a distributed deployment that does not own them answers 503.
• The add-on updates its outbox while a stream is open or a route runs. A job that ends while the client is away shows up when the client reconnects.
• The owner trusts the add-on. A client is held to its own jobs by the host, and the add-on tells the host which operation it is serving by the operation id.