Telegram

The Telegram add-on binds a Telegram bot to one of your saved agents. A person who messages the bot talks to that agent, and the answer comes back in Telegram. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Slack

The Slack add-on binds a Slack app to one of your saved agents. People talk to the agent in a direct message with the app, or by mentioning the app in a channel. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Discord

The Discord add-on binds a Discord bot to one of your saved agents. People talk to the agent in a direct message with the bot, or by mentioning the bot in a channel or thread of a server. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Simple Office

The Simple Office add-on (ID office) lets a chat make, read and edit Word, Excel, PowerPoint and PDF files. It does not need a Sandbox. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

The Sandbox add-on can make the same files with Python and gives full control of the layout. The two add-ons work on their own. See Sandbox: run Python on your files.

Sandbox: run Python on your files

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

Sandbox is an add-on. It lets a chat run commands and code, for example to make a chart, convert a file or run a script. The commands run in an isolated sandbox with your files mounted. The Sandbox add-on is for Linux. In the current development builds, it is in the add-on library in the development channel, so it can change.

What you need

• An administrator who installs the add-on. See the add-on user guide.
• A host that can run an isolated backend. The add-on tries KVM (Cloud Hypervisor) first, then gVisor. KVM needs /dev/kvm. gVisor needs runsc and permission to create a user namespace. If neither works, no command runs, and the chat says so.
• The chats and their files must be on, because the sandbox works on your files.
• A user who has the addons feature, on a server with accounts.

Turn it on for a chat

1. Open the Tools menu of the chat.
2. Switch Sandbox on.

When it is on, the bash tool of the chat runs in the sandbox. Without it, a chat can run commands on the server only if an administrator allows it, and then there is no isolation. See Running commands.

How it works

• Each user and chat has its own sandbox. The model cannot name another user or chat.
• Your files are mounted under /workspace/home. Each command starts there.
• After each command, what changed is saved to your files as one change. You can undo it in the history of Files. If someone else changed the same file, their version stays and the sandbox version is saved as a copy.
• A sandbox is suspended after 10 minutes without use and removed after 24 hours.
• An answer is cut at 64 KiB. A command stops after 120 seconds by default, and after 28 minutes at most.

Network

A sandbox has no network device. HTTP and HTTPS go through a proxy that asks LocalAI for each connection. The Network row in the Tools menu has three choices: Off, Package registries and All public sites. The limit that an administrator sets is a ceiling. The default limit is Off, so pip install does not work until an administrator allows it.

When a command asks for a host that the chat does not allow but the limit allows, the chat shows a card. Choose Allow once, Allow for this chat or Deny. See Network of the Sandbox commands.

What is installed

The base profile has Python with pandas, matplotlib, openpyxl, python-docx, python-pptx, pypdf and Pillow, and tools such as jq, ripgrep, sqlite3, zip, git and curl. The media profile adds ffmpeg, sox and ImageMagick. The video profile adds a Rust toolchain and a video renderer. You pick the profile in the Tools menu, unless an administrator limits the choice. Programs cannot be installed with apt. See Sandbox profiles.

Example: ask “Make a bar chart of the numbers in sales.csv and save it as a PNG”. The model reads the file, runs a Python script, and saves the image in your files.

Troubleshooting

For the settings and the host backend, see Sandbox backends and settings.

Tables

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

Status

Tables is a first-party addon for LocalAI. This page describes the database lifecycle, the typed schema operations, and structured row queries and confirmed row changes that exist today, how a chat model uses them as tools, and the host side of managing databases from the UI: creating, renaming, sharing and deleting a database, and choosing the database of a conversation. A table, with the filter and sort of a view, exports as a private CSV file (see CSV export). Import and public file links are not available. The Tables pages (the native UI) are described in The Tables pages.

Tables is a separate executable under addons/tables. It is not linked into the LocalAI server, and nothing of it runs unless an administrator installs and enables the addon. It reuses the root Go module and needs CGO for SQLite.

go build ./addons/tables/...

Deployment, trust and limits

• One owner. A database is a SQLite file on the disk of the one LocalAI instance that owns the add-on. Other frontends forward to it; while the owner is down every Tables call fails with 503 on every frontend, and nothing is applied later. See Singleton owner and its limits.
• Trust. The Tables executable is trusted code that runs as the owner’s operating system user. Tables itself never learns who a caller is: the host gives it the database ID and the action, and the host decides the role and the approval.
• Storage and backup. The data is under the data directory of the add-on, and backups are made by the administrator through the recovery routes (see Backup, upgrade and restore). A copy of the data directory made while the add-on runs is not a consistent backup.
• Ownership mode. Named-user mode keeps databases private to their owners and shareable. Local-owner mode has one owner and no sharing, so the database page shows no Share button. The mode is stored with the data and is changed only while the host is stopped, with an explicit mapping (see Moving between the modes).
• CSV. An export is a private, temporary file with the formula policy described in Spreadsheet formulas: it protects a spreadsheet, not a program that trusts CSV.
• Updates. The host never upgrades Tables by itself. An administrator checks for an update and applies one named candidate (see Update detection and upgrade); the data is backed up first.
• Optional contributions. The package declares its UI pages and its tools. A host that has no UI still runs the tools, and a UI never gains a tool the manifest does not declare. The chat actions it declares are placed and drawn by the host; the page cannot add a button to the chat of its own.

What a database is

Each database is one SQLite file with its own metadata. The file is named by an opaque resource ID that LocalAI assigns. Tables does not choose the ID, does not store who owns the database and does not decide who may open it. LocalAI keeps owners and grants and checks them before every call.

A resource ID must match ^[a-z][a-z0-9-]{0,63}$. Any other value is refused before it reaches the file system. Symbolic links are never followed, in the data directory or as a database file.

Databases are private by default. A new database belongs to the person who created it and nobody else can list or open it until the owner shares it with a grant. Administrators who install the addon get no access to databases. Share with your team describes the role model and the sharing API that other kinds of resources use with the same rules.

Tools

The manifest declares the executable tools below. The one host-owned exception is addon__tables__databases-list, described in Database discovery. That name is reserved: a Tables manifest cannot redefine it.

Tool names use hyphens because LocalAI manifests accept only [a-z][a-z0-9-]* identifiers.

Databases of a saved agent

Tables opts in to per-agent resources. A saved agent that names Tables lists the databases it may use, each as Read or Read and change; with none listed it uses none, and the list tool shows it only those databases. Creating a database is not limited, but a new database is on no list until you add it to the agent. See Resources of an agent. This needs LocalAI with feature level 2 (see Feature level); an older LocalAI refuses this version of Tables when it installs it.

Database discovery

addon__tables__databases-list takes exactly {} and reads the host’s resource registry, not the Tables executable. It is available only through existing authorized actor-bound consumers when Tables is visible, enabled and running, and the backend supports host resource listing. The tool exists because the Tables manifest declares it under capabilities.resources.list (see Resource listing); a copy of Tables with an older manifest keeps it until the add-on updates. Existing chat opt-ins, hidden add-ons and agent allowlists still apply. Shared MCP sessions and consumers without a principal do not gain access.

The normal bounded data result contains only database IDs, optional display names and the caller’s role:

{"databases":[{"resourceId":"opaque-id","name":"Budget","role":"viewer"}],"truncated":false}

Only active databases owned by or shared with the caller are listed; installing Tables grants no data access. Credentials and grants are rechecked on each call. Permission, registry and owner/transport failures remain errors, not empty lists. A successful empty result has databases: [].

Results are ordered by resource ID and limited to 100 databases. There is no pagination, cursor, search or total. truncated: true means the list is incomplete; an omitted database is not evidence that it does not exist. Names may be absent or duplicated: use opaque IDs, inspect the chosen database with addon__tables__databases-inspect, then query its actual tables and columns. Ask for clarification when the choice is ambiguous; never create a database or change the chat’s selection as a discovery fallback. Inspect and query independently reauthorize the ID.

This projected host read does not expose resource management, sharing, selection changes or mutations as discovery tools.

In the chats (/api/chats), the agents and the triggers, a change made with these tools is applied at once: turning Tables on for the chat or the agent is the consent. LocalAI still previews the change, checks the person’s role and the revision the model read, and records the receipt, so a stale change is refused and a repeated operation applies once. Because nothing waits for a confirmation, undo is the safety net: Tables keeps a change set for every row and schema change a chat makes, and the chat shows an Undo button on it (see Undo of a chat change). On every other surface (the OpenAI-compatible API, MCP, an API key), changes normally return proposals for human confirmation. Authorized server-side chats can also apply database creation and schema changes with undo. Check the actual result kind: proposal is not completion; applied confirms a change. The model cannot manufacture approval or identity arguments. Sharing and whole-database deletion are not model tools.

Bundled discovery skill

Tables packages the tables-readonly skill for structured-data questions and table searches. Load it to learn the workflow: list existing databases when needed, disambiguate names, inspect tables and columns, then query relevant rows or aggregate totals. Database listing is bounded to 100 and has no pagination; row queries have their own cursor.

The skill needs no command runner. Existing installation, running-state, chat switch and agent-grant gates control its availability. Its read-only guidance does not revoke other tools granted to the chat. It never creates a database or changes conversation selection as a discovery fallback.

Model tools and approval cards

A chat can offer the tools of Tables to its model. The person turns it on per chat with the add-ons control in the chat header, which lists each addon that offers tools with a switch and shows how many are on. The /addons command of the composer turns all of them on or off, and it appears only when an enabled addon offers tools. The chat then sends metadata.addon_tools with each request, and the server decides what this person may call: the opt-in grants nothing by itself.

POST /v1/chat/completions
{"model": "...", "messages": [...], "metadata": {"addon_tools": "true"}}

When only some addons are on, the request adds metadata.addon_tool_ids, a comma-separated list of addon ids. The model then sees and can call only the tools of those addons; a call to any other addon is refused as an unknown tool. Without the list, every addon the person may use is offered.

{"metadata": {"addon_tools": "true", "addon_tool_ids": "tables,notes"}}

Names and schemas

A tool is named addon__<addonId>__<tool>, for example addon__tables__rows-insert. The name is valid for OpenAI function names and fits in 64 characters. The namespace is reserved: an MCP server of a model cannot publish a tool under it, and an MCP session never runs one.

The schema of each tool is in capabilities.toolDefinitions of the verified manifest (addons/tables/manifest.json). The host serves it from the manifest, not from the addon process, and rejects a manifest whose definitions declare a keyword outside the closed set, an identity or approval property (actor, userId, role, approved, confirm, digest, proposalId, operationId, tool and similar), more than 32 properties, or more than 8 KiB for one schema. A declared tool without a definition is not offered to a model: resource-exists is one.

Every tool that works on a database takes resourceId. The host takes it out of the arguments and binds it to the request. A model cannot name a database it has no grant for: the same denial answers every tool.

What a call does

On the OpenAI-compatible API and every other consumer, a tool call never applies a change. It stores a proposal that only the proposer’s own browser session can confirm. The tool adapter has no confirm, cancel or resolve operation, and no tool of any consumer maps to those routes. The model is told that nothing changed and that a person must confirm.

The chats are the one exception. Their adapter stores the proposal and confirms it in the same call, for the signed-in owner of the chat, with the credential of a chat turn. No HTTP request can carry that credential, so a key, an MCP session or a browser tab cannot use it. The result is {"kind":"applied","applied":{id,addonId,resourceId,action,count,undoable,changeId,undo,reason},"note":...}. undoable is true when Tables kept a change set: changeId names it and undo says in one line what an undo does. Otherwise reason says why not, for example that the data was too large to keep. The note tells the model to say plainly what changed and whether the person can undo it, and that only the person can: the model has no tool for it. A call the add-on refuses or finds stale leaves nothing waiting: the proposal is withdrawn and the model gets the error. A call whose outcome is not known is not repeated and not withdrawn: it is reported, and the operation ID resolves it.

These names do not exist and are refused with a typed error: confirm, approve, apply_proposal, yes, any sharing, grant, database deletion or retained-data purge name, and any name that is not defined by the manifest of an enabled addon. Arguments that carry identity, role, approval or operation fields are refused before they reach the owner, so a model cannot forge the actor, the role or an approval, and a repeated or oversized argument object is refused as well.

The approval card

Where a tool answers with a proposal (not in the chats, which apply the change at once), the host’s card is drawn inline. The card loads the stored proposal from the host by its ID. It shows the add-on, the database (the requested name for a creation), the operation, the exact count of effects, a bounded sample, the expiry and the state. The words of the model are never the preview. Confirm and Cancel are the host’s buttons: Confirm sends the digest that was displayed with the X-LocalAI-Confirm header from the person’s own session.

The card is disabled for an expired, blocked, applied, cancelled, rejected or unknown-outcome proposal and says why. It forgets what it loaded when the person logs out, another person signs in, the addons feature is withdrawn or the host stops answering for the proposal. The stored conversation keeps the proposal ID only: no rows, no digest, no sample.

GET /api/addons/proposals?state=pending lists the caller’s own pending proposals, newest first, at most 32. It is where a change proposed by an agent or a scheduled job waits: those callers can read and propose within their permissions and can never confirm, because a key has no browser session.

Consumers

The consumer inventory is executable. The tests scan the sources for code that discovers or executes tools and fail for a file that has no decision here, so a new consumer cannot ship without one. A routed consumer takes an immutable actor from the authenticated request and goes through the broker. A rejecting consumer never serves add-on tools: the namespace is refused or filtered there.

A shared model MCP session is a rejection on purpose. One session answers every user of a model and keeps no current-user state, so it could not carry an actor. A deployment that needs add-on tools for an agent uses the /api/addons routes with that agent’s own key.

Undo of a chat change

A change a chat, an agent or a trigger makes at once can be undone, within the limits below. Tables records the inverse of the change inside the database file, in the same transaction as the change, bound to the revision the change produced.

The rules:

• Stale. A change set is bound to the revision of the change. It can be undone only while the database is still at that revision, so an undo never overwrites later work. Any later change of any table makes it stale, and the answer says what changed after it (for example, “inserted 1 row into People”). One refinement keeps this useful: an undo is itself a revision with the state of the one before the change, so a run of changes unwinds from the newest to the oldest. A rename of the database does not move the revision, so its change set is stale only when the database was renamed again.
• Idempotent. An undo has an operation ID. The same ID with the same change is one undo and returns the same receipt; another ID for a change that is undone already is already_undone.
• Bounds. The inverse of one change is at most 4 MiB, and a database keeps at most 64 MiB of inverses, dropping the oldest first. A change whose inverse would be larger, such as dropping a large table, keeps none, and its result says so at apply time: undoable is false with the reason “cannot be undone: the data it changed is too large to keep”. Tables never claims an undo it cannot carry out.
• Retention. A change set can be undone for 7 days. After that the answer is expired and the stored data is dropped. This period is fixed in this version; no setting changes it.
• No redo. An undo is a recorded change of the database, but it has no change set of its own, so it cannot be undone in turn. Make the change again instead.
• Roles. The role of an undo is the role of the change: owners and editors undo a data change, only the owner undoes a schema change or the creation of a database.
• Not a tool. The undo is the person’s own action from a browser tab (see Undoing a change a chat made). An API key, an MCP session, an agent or a model cannot call it, and no tool name for it exists. This is a decision: the model is told the change ID and that the person can undo it, so it cannot silently undo its own work or anyone else’s.

Change sets live in the database file, in the table tables_changes. They are part of every backup and restore of the add-on’s data, keep their retention across a restore (the time is absolute), and are removed with the database when it is deleted or purged. Nothing else holds them.

Several frontends

The tool adapter keeps no runtime state, so it is stateless in the sense of the distributed-state rule. Each frontend builds the tool list from the apps list its owner answers for the caller, and sends every call, proposal and pending list through the same carrier as the other addon routes. A frontend that is not the owner never holds a package, a proposal or a policy record. The apps list carries the tool definitions of each enabled addon, at most 32 KiB per addon, so the model-facing schema is the one in the verified manifest of the installed generation. If the owner cannot answer, discovery fails and the chat request answers 503 rather than offering stale tools.

Admin lifecycle tools

The LocalAI Assistant catalog has no add-on lifecycle tool (install, enable, disable, upgrade, uninstall, library changes). This is deliberate. The Assistant is one process-wide server with no per-call principal, and its mutating tools rely on the model asking the person for confirmation in prose. Add-on lifecycle changes are administration with persistent effects on data and processes, so they are not exposed that way. Listing and status are read with GET /api/addons/apps and the addons-installed management read, which need a named administrator. A conversational lifecycle tool needs a per-call principal and a host approval card first; it is not part of this release. REST-to-MCP parity for /api/addons is intentionally partial for that reason, and TestToolHTTPRouteMappingComplete stays unchanged. The UI-only resource routes (create, rename, share, delete, selection) are likewise not tools: sharing and deletion are never tools, and the consumer matrix test scans the catalog for any /api/addons route.

How creation survives a crash

Creation has three steps, in this order:

1. LocalAI records a pending resource for the creator. A pending resource is invisible to everyone, its creator included.
2. The addon creates the database file. It builds the file under a temporary name and links it into place only when it is complete.
3. LocalAI commits the resource, which makes it usable.

The addon operation ID is derived from the reserved resource ID, so repeating a step is one operation. If LocalAI stops between steps, it settles every pending resource at the next start by calling resource-exists:

• No database: the pending record is dropped.
• Database present: the resource is activated for the creator.
• The addon is installed but cannot answer: LocalAI stops its startup with a recovery-required error. It never guesses.

Deleting a database follows the same rule from the other side. LocalAI removes the record of a deleting database without calling databases-delete only when resource-exists answers that the database is gone. If the addon cannot answer, the deletion stays pending and is retried.

A database file without a record cannot be reached by anyone. A creation whose outcome is uncertain keeps its reservation, and that reservation counts against the per-user limit of pending resources until its receipt is found, the person who asked stops waiting, or the next start. Stopping to wait drops the reservation only when resource-exists says there is no database. A pending chat proposal keeps its reservation, so a restart does not remove a database you can still confirm.

Managing databases from the UI

Creating, renaming, sharing and deleting a database are the person’s own actions in the Tables pages, through the routes described in UI-only resource management. They are not tools: a model never sees them and sharing and deletion are never proposals.

• databases-create and databases-rename are the same tools that chat uses, run by the person’s explicit request with no approval card.
• databases-delete is an admin tool. The manifest lists it in capabilities.adminTools and not in tools, so it has no model definition and the declared-tool rule refuses it on every other path. The addon runs it only under the host action delete and only for the revision the owner confirmed. It removes the file and its journal files and records the operation in the ledger; repeating the operation, also after a crash between the removal and the receipt, gives the same receipt.
• The manifest capabilities.resources maps create, rename, inspect and delete to these tools, with the noun database.

A database is shared whole, as viewer or editor, with named users picked by account. An owner and an editor see the same rows; only the owner changes the schema, shares, renames and deletes. Deleting asks the owner to type the name, removes the data and every grant, and never uninstalls the addon.

The database of a chat

The owner keeps which database each conversation works on, per person. A tool call that names no resourceId uses it, so a person can say “add a row” after choosing a database once. The model never sets it, and the owner authorizes it for the person on every call: a database whose grant was withdrawn or that was deleted is refused like an unknown one. With none selected the tool answers a select_resource result and runs nothing.

The Tables pages

Tables ships its own UI as a standalone ES module in the package. It is loaded by the host after the host was built, so installing or updating Tables changes the pages without a rebuild of LocalAI. The My add-ons page shows a Tables card.

What each role sees:

Hiding a control is never the control. The server checks the role of every call again, and a crafted write of a viewer, a schema change of an editor or a share of anybody who is not the owner is refused with the answer of an unknown database.

Editing rows

• Typed editors. Text, whole number, number, yes or no, and date and time (entered and shown in UTC). A value of the wrong type is refused in the form and nothing is sent. Integers beyond 2^53 are kept exact. An optional column has an Empty switch that sends null.
• Filters. A list of conditions over the table’s own columns, joined by all or any, with the operators the column type supports. They become the filter of rows-query; there is no free text and no expression. The row ID is filterable as _id.
• Paging. The add-on’s cursor goes forward and the page keeps the cursors it used to go back. A cursor that the add-on no longer accepts (the data changed) starts again at the first page and says so.
• Batches. Add rows takes up to 100 rows. A selection is changed or deleted in steps of at most 1000 row IDs, each step saved on its own with its own operation ID. If a later step fails, the page says how many rows were changed before it.
• Revisions. Every change sends the revision the page read. If another person changed the database, the save is refused as stale, the form keeps what you typed and offers Reload and apply again, which reads the revision and sends the same change once more under a new operation. An unknown outcome keeps the operation ID, so repeating is the same change and never applies twice.

A row editor sends only the fields you changed. A datetime keeps its stored fraction of a second (up to nine digits) when you edit another field, and two spellings of one instant, such as another UTC offset, are no change.

Row and schema changes made in these pages are the person’s own action. They need no confirmation card and are not proposals, and they still go through the host’s POST /api/addons/resources/{resourceId}/apply: the role of the action, the declared tool definition, the admission of the resource, the revision and the receipt (see Changing data and schema from the UI).

The database of a chat

Choosing a database in the Select database action stores it for you, the conversation and the add-on. A chat tool call that names no database uses it, and the chat sends the conversation ID in the X-LocalAI-Conversation-Id header only when the conversation has add-on tools enabled. Two people in the same conversation ID, or one person in two conversations, never share a selection. A database that is no longer shared with you is refused like an unknown one.

Package and build

The UI source is addons/tables/ui. node addons/tables/ui/build.mjs writes dist/ui/tables.js, dist/ui/tables.css and dist/ui/icon.png, which the manifest names under capabilities.ui (entry, css, icon, pages, navigation, mainPage, chatActions). The package declares no dependencies, so it has no lockfile: the build uses the bundler that the host UI already has installed (core/http/react-ui/node_modules) and needs nothing to download. node addons/tables/ui/test/run.mjs runs its unit tests. The module imports nothing: React, the components, the typed calls and the navigation arrive through the host SDK, and every class of the style sheet starts with tbl-. The UI is translated to the languages of the host; a missing key reads the English text.

Schema operations

Only the owner of a database changes its schema. LocalAI refuses the schema action for every other role before it calls the addon, and the addon refuses a schema tool that does not arrive as a schema action with an actor and the preview or apply scope, so a caller of the runtime transport cannot use a row action to change a schema.

Tables and columns have a label and a stable internal identifier. The label is what people see. It is 1 to 64 characters, with no control, format or bidirectional characters, no leading or trailing space, and it is unique in its scope ignoring case. The internal identifier (t_..., c_...) is random and never changes. Labels are never part of an SQL statement, so a rename touches only the catalog and no label can inject SQL.

A default is validated for its type and written through a strict literal encoder, never copied from the request. There is no type change, trigger, function, expression, computed column, view or raw SQL.

• A required column needs a default when the table has rows. On an empty table it needs none.
• A unique column is a unique index. A default that would repeat in more than one row of a unique column is refused. Several NULL values are allowed.
• Dropping a column rebuilds the table inside the same transaction: create the new table, copy the rows, drop the old one. A failure leaves the original.
• Limits: 50 tables per database, 64 columns per table.

Revisions, receipts and previews

Each database has a revision pair, schema and data. Every schema change raises schema by one. The preview returns the revision it was made against, and apply must send that revision back. If the database moved, apply is rejected with {"error":"stale","revision":{...}} and nothing changes. Both numbers must match, because the data a drop destroys depends on the rows.

The change, the new revision and the operation receipt commit in one SQLite transaction. After a crash the receipt is either in the database or the change did not happen. The same operation ID with the same request returns the same receipt and never applies twice. The same ID with another request is a conflict. An operation that was refused leaves no receipt, so its ID can be used again. If a reply is lost, LocalAI asks for the receipt, which the addon finds in the database even when its own ledger was not updated.

A preview runs the change in a transaction that it rolls back, so it accepts only what apply would accept. Its summary has this shape:

{"count": 1,
 "sample": [{"action": "drop column", "table": "Orders", "column": "Total", "rowsLost": 42}],
 "dataLoss": {"rows": 42, "columns": 1, "tables": 0}}

rowsLost of a column counts the rows that hold a value in it. For a table it counts all rows and columnsLost its columns. LocalAI shows count and sample in the confirmation card.

Row queries

rows-query and rows-aggregate run under the read action, so viewers can use them. The addon builds each statement from internal identifiers and binds every value as a parameter. A column or table label is only looked up in the catalog; it never becomes SQL text. There are no joins, raw SQL, expressions, functions or subqueries.

{"tool": "rows-query", "table": "Orders", "columns": ["Item", "Qty"],
 "filter": {"and": [{"column": "Qty", "op": "gte", "value": 2},
                    {"or": [{"column": "Item", "op": "starts-with", "value": "pen"},
                            {"column": "Item", "op": "is-null"}]}]},
 "sort": [{"column": "Qty", "desc": true}], "limit": 50}

• Rows come back as {"id": n, "values": [...]} in the order of the columns of the reply. id is the stable row ID of the table. It never changes and is never reused after a delete. Filters and sorts may name it with the label _id; no column may use that label. Omit columns to select all.
• Filter is a tree of and, or and conditions: at most 8 levels and 64 nodes. Operators are eq, ne, lt, lte, gt, gte, in (1 to 100 values), is-null, is-not-null, and for text columns contains, starts-with and ends-with. A boolean column supports eq, ne and the null tests only. A node is exactly one of and, or or a condition; unknown fields in the arguments are refused.
• Text matching. eq, ne and the order operators on text are case-sensitive and compare bytes. contains, starts-with and ends-with are ASCII case-insensitive and treat %, _ and \ in the value as plain characters. The value is at most 1024 bytes.
• NULL. A comparison (eq, ne, lt, …, in) never matches a NULL value, so ne does not return rows where the column is empty. Use is-null for those, for example or of ne and is-null. A comparison with a null value is refused. In a sort, NULL comes first ascending and last descending. Aggregates ignore NULL values; count without a column counts rows.
• Sort takes at most 4 columns. Rows with equal keys are ordered by row ID, so a page boundary is stable.
• Pages. limit is 1 to 500 and defaults to 50. When more rows match, hasMore is true and cursor is set; send it back with the same query to get the next page. A cursor is valid only for the same query and the same database revision; after any change the addon answers that the cursor is no longer valid and the caller starts again. There is no total count. Offsets beyond 100000 rows are refused, so add a filter. A page is also cut at 512 KiB of JSON, and hasMore is set.
• Values in JSON. text is a string, integer a JSON integer, real a finite number, boolean true or false, datetime an RFC 3339 string. Datetimes are stored in UTC with nine fractional digits and returned in UTC. Integers can be sent as a number or a decimal string; integers beyond +/- 2^53 are returned as strings so no precision is lost.
• Aggregates. aggregates holds 1 to 8 of {"fn": "count" | "sum" | "avg" | "min" | "max", "column": "..."}. sum and avg need an integer or real column; min and max work on integer, real, text and datetime. groupBy names one column; groups are ordered by the group value and limited to 500, with truncated set when more exist. A sum that overflows 64 bits is an error, not a wrapped number.
• Cancellation. A query is bound to the request context and to a per operation deadline; a cancelled or slow query is interrupted and returns an error. Each read sees one snapshot of one revision.

Row changes

rows-insert, rows-update and rows-delete are changes under the host write action: Editors and Owners may propose them and the proposer confirms them in the host card. The addon refuses a row tool that arrives under another action and a schema tool that arrives under the write action.

• Insert takes 1 to 100 rows; each row maps column labels to values. A column that is left out gets its default, or NULL. A required column without a default must be given. An explicit null for a required column is refused even when it has a default. The whole batch commits or none of it.
• Update and delete name their rows by exactly one of ids (distinct positive row IDs, all of which must exist) or filter. There is no way to touch every row without a filter that selects them, and no truncate. One operation changes at most 1000 rows; a broader filter is refused with a request to narrow it. An operation that matches no row is refused.
• Constraints. Type, required and unique rules are enforced by the database. A violation rejects the whole operation, names the column by its label and leaves the data unchanged.
• Preview. The preview runs the same code in a transaction that it rolls back. It reports the exact count, a sample of the first 10 rows (for update, the values before and after of the changed columns; for delete, the full rows), an affectedHash of the exact row ID set, and dataLoss.rows for a delete. A change that would break a unique rule fails at preview, so the card never offers it.
• Receipt. The payload holds count, the affected ids and affectedHash. The receipt commits in the same transaction as the rows and the data revision step. LocalAI reports success only from the receipt.

Revisions and staleness

The revision is conservative and per database. Every committed row change raises data by one in the same transaction; a schema change raises schema. Apply compares both numbers with the revision of the preview inside the transaction. If either moved, the change is rejected with {"error":"stale", ...}, and a new proposal and a new preview are required. Because the revision is unchanged, the set of rows a filter selects is the set that was previewed: a filter is never evaluated against rows that arrived after the preview. A write to any table of the database makes a pending row proposal stale, even when it touched unrelated rows. Two proposals made on the same revision cannot both apply: exactly one wins and the other is stale.

Row changes use the receipt rules above: the same operation ID with the same request replays one receipt, another request under the same ID is a conflict, and a rejected operation does not use up its ID.

CSV export

rows-export writes the rows a table, a filter and a sort select to a CSV file. It is a read: it runs under the host read action, so an Owner, an Editor and a Viewer may export what they can read, and it needs no approval card. A chat tool call and the Export CSV button of a table page run the same tool.

{"tool": "rows-export", "table": "Orders", "columns": ["_id", "Item", "Qty"],
 "filter": {"column": "Qty", "op": "gte", "value": 2},
 "sort": [{"column": "Qty", "desc": true}], "maxRows": 5000}

columns, filter and sort follow the rules of Row queries. columns may name _id to write the row ID as a column; empty exports every column in schema order. maxRows is optional. There is no limit and no cursor: an export is the whole result, not a page.

What the file holds

• One snapshot. The export reads the database in one read transaction, so every row comes from the same revision. A change that commits while the export runs is not in the file, and the export never blocks it. The file is written row by row through a small buffer; its memory use does not grow with the number of rows.
• RFC 4180 records with a header row of the column labels and CRLF line ends. A field is quoted when it holds a comma, a quote, a CR or LF, or starts or ends with a space or tab. A quote inside a field is doubled.
• UTF-8, with no byte order mark. Text is written as it is stored. A spreadsheet that guesses the encoding from the file may need to be told to read it as UTF-8.
• NULL is an empty field with no quotes. An empty text value is "". A reader that does not tell the two apart reads both as an empty value.
• Booleans are true or false. Integers are decimal and exact to 64 bits. Reals are the shortest decimal that reads back as the same number; very large or very small values use an exponent. Date-times are RFC 3339 in UTC.

Spreadsheet formulas

A spreadsheet runs a cell that starts with =, +, -, @, a tab or a carriage return as a formula. The export therefore writes a single quote before every text cell, and every column label, that starts with one of those characters, as the OWASP guidance recommends: =1+1 is written as '=1+1. A spreadsheet shows the quote as the text marker, not as part of the value.

Limits of this policy:

• Cells of integer, real, boolean and date-time columns are never prefixed. A negative number stays a number (-4).
• The file now holds '=1+1, not the stored =1+1. A program that reads the CSV back must remove that quote itself; the export does not mark which cells it changed. A text value that really starts with ' and a formula are therefore not told apart after the round trip.
• A text value such as -5 apples also gets the prefix.
• The policy protects the file against being run as a formula in a spreadsheet. It is not a defence for a program that treats CSV as trusted input.

Backup, upgrade and restore

The databases are SQLite files under the data directory of the add-on, with one ledger. The host backs them up with VACUUM INTO while the add-on is stopped, so the copy is consistent and holds no write-ahead log (see “Data recovery” in the add-ons reference). Backup, restore and purge are not tools and not part of the manifest: the administrator’s page uses the recovery routes, and no model can reach them.

Tables records the format of its data. store.FormatVersion is the format this build writes; it is kept in the format key of each database and, for the whole store, in the ledger. At start the executable calls Upgrade, which compares them, runs Migrate(ctx, fromVersion, toVersion) when the data is older and writes the progress to migration.json. Each database migrates in one transaction, so a step that fails leaves that database exactly as it was, and the store version is recorded last, so a run that stopped resumes. Data in a newer format than the executable reads stops the start with exit status 9 and is left as it is: the host then keeps the add-on unavailable instead of running it on data it cannot read. No step exists yet because the first format is the only one. The table of change sets (tables_changes) is created when a database first needs it and is ignored by a build that does not know it, so it needs no migration.

Limits

A result past a bound makes the tool answer {"exported": false, "error": "too-large"} and creates no file; add a filter to export less. A maxRows above the row bound is refused the same way, not clipped. A staged export that the owner did not fetch within ten minutes expires; the addon removes expired files when it handles the next export or fetch, and removes every staged file when it starts.

How the file reaches the person

There is no separate asynchronous export job in the addon. The CSV is written inside the read call, within the 25 second bound, because the owner needs the exact size before it moves a byte. A result that does not fit those bounds fails with too-large instead of running longer. The transfer from the addon to the owner’s private store is the background job: the read returns at once with the artifact in state transferring, and the page and the chat ask the host for its state until it is final. Cancelling the transfer, a revoked share and a stopped addon end it and remove the partial bytes.

The addon writes the CSV into its private directory and answers with one output description: a name (<table>.csv), the media type text/csv, the exact size and an opaque handle bound to the operation, the person and the database. LocalAI streams the bytes from the addon, checks the size and replaces the handle by its own artifact identifier. The page, the chat and the model get that identifier, the name, the size and the expiry. They never get the addon’s handle, a path or a URL. The transfer runs in the background; the artifact shows transferring, then complete, failed or cancelled, and an export that did not complete never becomes downloadable.

• GET /api/addons/artifacts/{id} is the transfer state, name, size and expiry. GET /api/addons/artifacts/{id}/download is the file, as an attachment with X-Content-Type-Options: nosniff and Cache-Control: private, no-store. Both need the person’s own session or API key.
• The file is private to the person who exported it. Sharing the database does not share a file. A person who is not the principal, an unknown identifier and an expired file all get the same 403.
• Access to the database is checked again when the transfer completes and at every download, and a revocation stops a download in progress. After a viewer’s share is withdrawn the viewer’s earlier files stop downloading. A file already downloaded cannot be recalled.
• A file expires one hour after it was made, and the person’s quota is 100 files. Expiry removes the file only. The database and any backup are not touched.
• A disabled addon does not serve downloads; enabling it again serves the files that have not expired.
• In single-owner mode there is one local owner and no separation between people, as for every other operation.

In the pages and in chat

A table page has an Export CSV button for every role. It exports the table with the filter and sort of the view, optionally with the row ID, and then shows the host’s file card: name, size, expiry and the download button. In chat, an export tool call is answered with the same card. The chat stores only the artifact identifier; after expiry the card says the file expired.

Limits

• One name per database, no uniqueness rule across databases.
• Rename needs the schema revision from the preview. A changed schema makes the rename rejected, and a new preview is required.
• The data directory holds the databases and a small operation ledger. Both are kept when the addon is disabled or uninstalled.

How to try it

The addon is not published in a catalog yet. Acceptance builds the package locally and installs it from a catalog on loopback; it never pushes, tags or signs. Build the executable, put it in a package with manifest.json from addons/tables, and install the package from a local source as described in Addon contracts. The suite sh scripts/addons/verify.sh tables-databases runs the store and the real executable under the real supervisor. The suite sh scripts/addons/verify.sh tables-schema covers the schema operations, revisions and receipts. The suite sh scripts/addons/verify.sh tables-rows covers queries, filters, aggregates and confirmed row changes with the real executable. The suite sh scripts/addons/verify.sh tables-tools covers the model tools, the consumer inventory, the pending proposal list and the approval card in a real browser against the real owner. The suite sh scripts/addons/verify.sh tables-csv covers the CSV export: the snapshot and encoding rules, the output handles, the private artifacts of the real owner and the download from the owner and two frontends. The suite sh scripts/addons/verify.sh tables-ui covers the pages: the node tests of the field codecs, the filter builder, the batch bounds and the build, the UI-intent change route against the real owner, and a browser spec with real sessions for the roles, private defaults, sharing and a concurrent revoke, destructive deletion, a stale tab after an upgrade or a disable, the database of a chat for two conversations and two people, keyboard, 400 pixels, both themes, clearing on sign-out and local-owner mode. The suite sh scripts/addons/verify.sh tables-acceptance is the end-to-end acceptance (see Acceptance). The suite sh scripts/addons/verify.sh tables-admin covers the UI-only management of databases with the real owner, real accounts and the real executable: private defaults, the owner, editor and viewer matrix, sharing and the user picker, deletion and its crash reconciliation, the database of a conversation, and the answers of a forwarding frontend.

Calendar

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

The Calendar add-on gives you a calendar inside LocalAI. Ask a chat or a saved agent “what is on Thursday?” or “add dinner with Sam at 8” and it reads or changes the calendar. There is no account to connect, no sync and no setup: install the add-on and use it.

What it keeps

There is one calendar for each user, shared by all of that user’s chats and agents. It is one SQLite file in the data directory of the add-on. An event has:

• a title, a start and an end, or a whole day (an all-day event);
• notes, a place and the attendees, each as free text;
• a time zone, as an IANA name such as Europe/Rome, and optionally a repeat.

A time is a local date and time with a zone name, the way a person says it. Set the default time zone once, on the Calendar page (it starts as the zone of the server). A new event uses it unless you or the agent name another one. An event keeps the zone it was written in when you change the default.

A repeat is daily, weekly, monthly or yearly, with an interval (“every 2 weeks”) and an optional last day. The local time stays the same across a change of daylight saving time. A monthly repeat on the 31st skips months that have no 31st. You can change or delete the whole series, or skip one occurrence.

There is no sync with another calendar, no invitation, no second calendar and no reminder that pushes a message. You can export the whole calendar as one ICS file from the page and import it elsewhere.

The page

Open Calendar in the add-ons. It has a month view, an agenda list of the next 60 days, a form to create or change an event, the default zone and the export button. Click a day to add an event, or an event to change, skip or delete it. The page speaks the language of the web UI.

What the agents can do

The reads run at once. The writes are action tools: they run at once, as the user, with no confirmation card, and an unattended run (a trigger or a job) needs the act.addons grant for them. The add-on ships a skill that tells the model when to use them.

In a messaging chat, the owner and the senders paired with the bot can read and change the calendar. Every other sender the bot lets in can read it but not change it (the tools.write rule, see Add-ons that keep the owner’s own data).

Limits

An event title has at most 200 characters, notes 4000, a place 200 and attendees 500. One user keeps at most 5000 events. A listing covers at most 400 days and returns at most 500 occurrences; the page says when a list was cut.

Contacts

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

The Contacts add-on gives you an address book inside LocalAI. Ask a chat or a saved agent “what is Marco’s number?” or “add Anna, anna@example.org” and it reads or changes the book. There is no account to connect, no sync and no setup: install the add-on and use it.

What it keeps

There is one address book for each user, shared by all of that user’s chats and agents. It is one SQLite file in the data directory of the add-on. A contact has:

• a name, and an organisation;
• phone numbers and email addresses, each with a free label such as mobile, home or work (up to 10 of each);
• a birthday, as YYYY-MM-DD, or --MM-DD when the year is not known;
• an address and notes, each as free text.

There is no sync with another service, no CardDAV, no groups, no photos and no merging of duplicates. Import of vCard files is not part of this version. You can export the whole book as one vCard file from the page.

The page

Open Contacts in the add-ons. It has a search box, the list of contacts, the detail of the one you pick, a form to add or change a contact, and the export button. The search matches every word you type against the name, organisation, phone numbers, email addresses, address and notes. It ignores case and accents, and a phone number matches however it is spaced. The page speaks the language of the web UI.

What the agents can do

The reads run at once. The writes are action tools: they run at once, as the user, with no confirmation card, and an unattended run (a trigger or a job) needs the act.addons grant for them. The add-on ships a skill that tells the model when to use them.

In a messaging chat, the owner and the senders paired with the bot can read and change the book. Every other sender the bot lets in can read it but not change it (the tools.write rule, see Add-ons that keep the owner’s own data). The book is private to its user: a tool call can never name another user’s book.

Limits

A name has at most 200 characters, an organisation 200, a label 40, a phone number 40, an email address 254, an address 500 and notes 4000. One user keeps at most 2000 contacts.

A vCard export is a vCard 3.0 file. A birthday without a year is written as --MM-DD, which some programs do not read.

Web search

The Web add-on (ID web) serves the chat tools web_search and web_fetch in place of the built-in ones. The built-in tools work with no add-on and no key. The add-on adds more. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Gmail

The Gmail add-on lets your saved agents read the mail of your Google account. If you allow it, they can also save a draft and send a draft that exists. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Everything the agent reads goes to the model of the chat. Use the label list of the connection to keep an agent to one label.

Mail

The Mail add-on lets your saved agents read a mailbox over IMAP and, if you allow it, write mail over SMTP. It works with any provider that offers both protocols. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Mail text goes to the model the chat uses.

Google Drive

The Google Drive add-on lets your saved agents read the files in the Drive folders you choose. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

GitHub

The GitHub add-on lets your saved agents read the repositories you choose: code, files, issues, pull requests and their diffs. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

The add-on does not listen for @mentions on GitHub. An agent works on GitHub only when you ask it in a chat.

Jellyfin

The Jellyfin add-on lets your chats and saved agents look into a Jellyfin media server. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Home Assistant

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

The Home Assistant add-on lets your chats and saved agents read your smart home: find entities, read a state, look at a history, list what is unavailable. If you allow it, they can also switch things, such as a light or a thermostat. You set the limits. The add-on never operates a lock, an alarm, a garage door or a siren, whatever you choose.

A chat can answer questions such as:

• “Is the office warmer than the bedroom, and was it warmer yesterday?”
• “Which devices are unavailable?”
• “Turn off the lights in the living room.” (a change, see Changes)

Set it up

1. Install the Home Assistant add-on from the add-on gallery and open its page.
2. In Home Assistant, open your profile (your name at the bottom of the sidebar), then the Security tab. Under Long-lived access tokens, choose Create token, name it (for example LocalAI) and copy it. Home Assistant shows it once. The token has the rights of the user that made it. If you can, make it with a user that sees only what you want to share.
3. On the add-on page, choose New Home Assistant connection. Paste the token and write the address you use to open Home Assistant in a browser, for example https://homeassistant.local:8123. Press Test: the row shows the Home Assistant version if the address and the token work.
4. Leave the write mode on Read only until you have read the limits on this page.

One connection is one home: one token and one address. For a second home, add a second connection. A chat uses the connections you enable for it in the Tools menu (see The connections of an add-on). With two or more enabled, the model names the connection in each call.

How the address is kept

The address is not a secret, so it is not stored with the token. The connection lists it as its one allowed resource, the same list the GitHub add-on uses for repositories. Every tool takes an instance argument. LocalAI compares it with the address on the connection before the add-on is called and refuses any other, so a model cannot send the token to another host. The add-on talks only to that address:

• The address is https. Plain http is accepted only for an address on your own network (a private IP address, a single-word name, or a name ending in .local, .lan, .home, .internal or .home.arpa).
• A reply that redirects is an error. The token is never sent to another host.
• The token goes in the Authorization header and nowhere else. It is not in a log, an error or a result.

A self-signed certificate

The add-on checks the certificate of Home Assistant. If yours is self-signed, the Test says the certificate is not trusted. Do not turn the check off. Pin the certificate instead: copy its SHA-256 fingerprint and add it to Pinned certificates in the limits below, as host:port=fingerprint.

openssl s_client -connect homeassistant.local:8123 </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

The add-on then accepts that exact certificate for that exact host:port, and nothing else. When the certificate is renewed, pin the new one.

What the tools do

Two more tools, get-policy and set-policy, belong to the add-on page. A model never sees them.

There is no tool that renders a template, fires an event or sets a state, because each can run arbitrary logic on your server.

Everything Home Assistant returns (names, states, attributes, logbook text) was written by devices, integrations and people. Every result is marked as untrusted data, and the add-on’s skill tells the model never to follow an instruction found in one. Results are bounded: a list is cut and says so, text is cut, an attribute value is cut, and a result above the size limit is refused.

The add-on removes these from every result: entity_picture, access_token, anything with token, password or secret in its name, a value that carries a token in an address, and the location of people and devices (latitude, longitude, gps_accuracy and similar). It never returns a camera snapshot address.

Limits you set

The limits are on the add-on page, under the connections. Everyone who opens the page sees them. Only an administrator can change them. They apply to every Home Assistant connection of the server.

Services a model may call

The allowlist holds domain.service entries, or domain.* for the services of a domain. A service that is not on it is refused. The default is small:

notify services are not on the list. Add one only if you want a model to send messages.

A script can do anything you wrote into it, and a switch can power a heater. Allow only what you would let a guest in your house do.

Never available

These cannot be allowed. The add-on refuses them when you save the limits, and again when a call is made, so a hand-edited file does not change it:

You can still let a chat read what a lock reports (see below). Reading is never operating.

Entities the tools never show or touch

Excluded entities are left out of every search, state, history, logbook and area count, and no service call can name them. You can exclude by:

• Domain. Default: lock, alarm_control_panel and camera. Remove one to let a chat read it. A lock you read is still never operated.
• Area. The entities of the areas you list. If Home Assistant cannot answer which entities an area holds, the tools refuse instead of guessing.
• Pattern. A regular expression matched against the whole entity ID, such as sensor\..*_bedroom.
• Label. Entities that carry the Home Assistant label you name (default sensitive). An older Home Assistant that does not know labels cannot check it. The result then says that the label exclusion was not applied.

How many things one call switches

A call names its entities one by one. The default is 5, the highest you can set is 20. Area, device, label and all targets are not accepted: the model has to name each entity, so you see each one in the confirmation.

Changes and confirmation

A connection has a write mode, which the host enforces before the add-on is called:

call-physical-service is always confirmed, even in the last mode. A proposal for it expires after 15 minutes, a quiet one after two hours. Changing the limits after a preview makes an old proposal stale: it is rejected and nothing is sent.

Home Assistant has no idempotency key. The add-on notes an operation before it sends it and never sends one twice. If the connection breaks after the call left, the result is “unknown”, and you decide what to do.

Install notes

• The add-on asks the host, through the workspace.admin operation, whether the person who saves the limits is an administrator. The shipped host policy (deploy/addons/host-policy.json) grants it. Without that grant, the limits can be read and not changed.
• The limits are kept in policy.json in the add-on’s data folder. A file that cannot be read, or that names a service that is never available, is refused as a whole: the tools stop working until an administrator saves the limits again. They do not fall back to something wider.
• The add-on runs one process for the server, so every frontend of a cluster reads the same limits.
• It is a Linux and macOS add-on, like the other tool add-ons.

Not verified without a real instance

The tests run against a fake Home Assistant that speaks the REST API. The fake answers the fixed templates of the add-on (areas and labels) by their shape, so a real instance can still differ: area and label questions use POST /api/template with fixed templates and no text from a model. If an area question fails on your Home Assistant, list-areas says so and the area exclusion refuses to guess.

PiKVM

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

The PiKVM add-on lets your chats and saved agents look at a PiKVM (or another device that runs the kvmd API) and the machine it controls: is it on, what is the device, is the keyboard connected. If you allow it, they can also press the power buttons of that machine and type on its keyboard. A person confirms those actions unless you choose otherwise for power actions and key presses (see Not asking for some actions and Allow all), and typing and forced power actions are off until an administrator switches them on.

A chat can answer questions such as:

• “Is the test box on?”
• “Power the test box on.” (a power action, see Power actions)
• “Press Escape and F2 to enter the BIOS.” (keys, see Typing and keys)

Set it up

1. Install the PiKVM add-on from the add-on gallery and open its page.
2. On the PiKVM, add a user for LocalAI: switch the filesystem to read-write with rw, run kvmd-htpasswd add localai, then ro. See the PiKVM authentication guide. A PiKVM has no per-user rights: every user can do everything. A dedicated user does not limit what the add-on can do, but you can remove it on its own. The limits of the add-on (below) are the only ones.
3. Get the fingerprint of the certificate of the PiKVM (see The certificate).
4. On the add-on page, choose New PiKVM connection. Type the user name, the password and the address, for example https://pikvm.local#sha256=<fingerprint>. Press Test: the row shows the model and the kvmd version if the address and the login work.
5. Leave the write mode on Read only until you have read the limits on this page.

One connection is one device. For a second device, add a second connection. A chat uses the connections you enable for it in the Tools menu (see The connections of an add-on). With two or more enabled, the model names the connection in each call.

The user and the password

The connection keeps one sealed credential. In this version it holds user:password: the host splits it at the first colon, so the user name has no colon and the password may have colons. The page has a field for each and joins them. This is a v1 format: when the host gets a credential kind with named fields (a user and a password), the add-on will move to it, and you will enter the pair again.

The add-on sends the user and the password in the X-KVMD-User and X-KVMD-Passwd headers, as the PiKVM API allows, and nowhere else. They are not in a log, an error or a result. The page trims nothing for you: a password with a space at its start or end is refused.

Two-factor login (2FA) is not supported in this version. With 2FA on, the PiKVM wants the current one-time code added to the password, and the code changes every 30 seconds. A device with 2FA on answers 403 and the Test says the login was refused. 2FA is one switch for the whole PiKVM (kvmd-totp), so the device cannot have it on for the browser and off for LocalAI.

How the address is kept

The address is not a secret, so it is not stored with the password. The connection lists it as its one allowed resource, the same list the GitHub, Jellyfin and Home Assistant add-ons use. Every tool takes a device argument. LocalAI compares it with the address on the connection before the add-on is called and refuses any other, so a model cannot send the password to another host. The add-on talks only to that address:

• The address is https. Plain http is accepted only for an address on your own network (a private IP address, a single-word name, or a name ending in .local, .lan, .home, .internal or .home.arpa), because the login travels in headers.
• A reply that redirects is an error. The password is never sent to another host.
• The add-on does not use a proxy of the environment.

The certificate

A PiKVM comes with a self-signed certificate. The add-on never turns verification off. Pin the certificate instead: copy its SHA-256 fingerprint and add it to the address as #sha256= and the 64 hex digits (colons are fine).

openssl s_client -connect pikvm.local:443 </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

The add-on then accepts that exact certificate for that exact address, and nothing else; a different certificate is refused before the password is sent. When you renew the certificate, edit the connection and pin the new one. Without a pin, a self-signed certificate is refused and the Test says the certificate is not trusted.

What the tools do

Two more tools, get-settings and set-settings, belong to the add-on page. A model never sees them.

Everything the device returns is marked as untrusted data, and the add-on’s skill tells the model never to follow an instruction found in a result. Results are bounded.

Changes and confirmation

All three write tools declare approval: always in the manifest. By default a person confirms every call: the mode Read only refuses them all, and Asks you first stores each as a proposal that waits for a confirmation card. For power-action and send-key you can stop being asked one by one, as described next, and Allow all stops it for all three tools (see Allow all). A proposal expires after 10 minutes (power) or 5 minutes (typing and keys).

Not asking for some actions

On the add-on page, the section Ask before acting has one choice for Power actions and one for Key presses: Ask each time (the default) or Don’t ask. There is no choice for typed text, because it can enter a password or a command; it waits for you unless the connection is on Allow all.

• The choice is yours alone. It is saved from your own browser session, it applies to all your PiKVM connections, and no chat, agent or model can change it.
• With Don’t ask, a chat can switch the machine off or reset it at once, with no card, and unsaved work on it is lost. If an administrator allowed the hard power actions, those run without asking too. Read the warning on the page before you choose it.
• It holds only for your own chats and for messaging senders you paired with the bot. Senders that are only on the allowlist, senders let in because the bot accepts anyone, roles, scheduled runs and sub-agents are still asked.
• A connection set to Read only runs nothing, whatever you choose. A connection set to Allow all runs these two tools without asking whatever you choose here.
• Every run without asking is written to the server log and to the latest decisions of the connection, with the tool, the connection and the result. The typed text and the keys are not written.
• The row of each connection lists which tools run without asking now.

Allow all

The write mode of a connection has a third choice, Allow all, for someone who wants the agent to act without asking. It is never the default and is never chosen for you: a new connection starts on Read only, and a connection keeps the mode it has.

With Allow all, a chat of yours runs power-action, send-key and type-text at once, with no card. Unsaved work on the machine is lost when it is switched off or reset, and typed text and key presses go to whatever has the focus, so a shell command or a password can be entered and none of it can be taken back. The page warns about it. If you want a confirmation, choose Asks you first. The per-tool choices of Ask before acting are not asked in this mode, and the row of the connection lists both tools that have a choice as running without asking.

These rules hold in every write mode, Allow all included:

• The switches of an administrator for each device stay in force. Typing and key presses need Allow typing and key presses, the forced power actions (off_hard, reset_hard, long_press) need Allow hard power actions, and the add-on refuses a call when the switch is off. Allow all does not turn a switch on.
• The same limits on text and keys, the stale-state check and the rule that nothing is sent twice.
• A run without you present, such as an agent started by a trigger, still leaves a proposal that waits for a person.

The add-on notes an operation before it sends it and never sends one twice. The PiKVM API has no idempotency key. If the connection breaks after the request left, the result is “unknown”, and you decide what to do. Read the power state to see what happened.

Power actions

power-action takes one of these:

The confirmation card shows the power state that was read just before, and what the action does. The preview reads the state again when you confirm. If the state changed, the action is rejected as stale and nothing is sent. An action that makes no sense is refused at once: on for a machine that is on, off for one that is off, a reset for one that is off. A busy device or one with ATX control turned off is refused.

Typing and keys

type-text and send-key are off until an administrator switches on Allow typing and key presses for the device on the add-on page. Both use that one switch.

• type-text types text through the HID print endpoint. The text is at most 200 characters by default; an administrator can set up to 1000 for a device. It has no control characters except newline and tab. The card shows the exact text, the number of characters and lines. The text is never written to a log or an error, and the result of the call does not repeat it. As defence in depth, the add-on refuses text that looks like a private key (a -----BEGIN header), a long unbroken blob of base64-like characters (80 or more), a well-known token shape (cloud access keys, source host tokens, SSH public keys) or an exported list of passwords (three or more lines such as password: ...). This is a heuristic and not a guarantee: the skill also tells the model never to type a secret the person did not dictate.
• send-key presses up to 8 keys or chords, in order. Allowed: the letters a to z, the digits 0 to 9, up, down, left, right, enter, escape, tab, f1 to f12, and the chords ctrl+alt+delete, ctrl+c and ctrl+z. Windows and Meta chords, modifiers alone and every other key are refused. Single keys use POST /api/hid/events/send_key, chords use POST /api/hid/events/send_shortcut.

Both refuse when the keyboard of the device is not connected to the target.

Limits you set

The switches are on the add-on page, under the connections. Everyone who opens the page sees them. Only an administrator can change them. They are kept for each device address, so a switch for one PiKVM does not open another:

The add-on asks the host, through the workspace.admin operation, whether the person who saves the switches is an administrator. The shipped host policy (deploy/addons/host-policy.json) grants it. Without that grant, the switches can be read and not changed. A settings file that cannot be read leaves every switch off. They are kept in settings.json in the add-on’s data folder; the add-on runs one process for the server.

What is not supported

• Screenshots and screen reading. The streamer snapshot is left out of this version. The screen of the target can show secrets, and a snapshot needs a way to hand an image back that this add-on does not have yet. Without it, ask the person what the screen shows.
• Mass storage. No mounting or uploading of disk images.
• GPIO, mouse, the HID reset, and the ATX click beyond what power-action covers.
• Changes to the users or the configuration of the device.
• Two-factor login. See above.
• Windows and Meta key chords.

Not verified without a real device

The tests run against a fake that speaks the parts of the kvmd API listed above, as the PiKVM API reference describes them. A real PiKVM can still differ. Check these on a device you can reach before you rely on the add-on:

1. The Test shows the model and the version (GET /api/info; the add-on reads hw.platform and system leniently).
2. get-power-state follows the real power LED.
3. A soft off and an on change a spare machine, and reset restarts it.
4. type-text types a short text with the layout you expect. The add-on does not set a keymap: the PiKVM default applies.
5. send-key presses escape and ctrl+alt+delete.
6. With a wrong pin, the connection is refused before the password is sent.

SSH

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

The SSH add-on lets your chats and saved agents run commands on a remote machine over SSH. By default a person confirms each command: the card shows the host, the user and the exact text before anything runs.

A chat can answer questions such as:

• “How full is the disk on the build server?”
• “Show the last 50 lines of the service log.”
• “Restart the worker.” (a change: it waits for your confirmation)

The add-on runs one non-interactive command per call. It has no terminal, no input and no environment variables chosen by the model. It has no interactive shell, no file transfer, no port forwarding, no agent forwarding and no jump hosts.

Set it up

1. Install the SSH add-on from the add-on gallery and open its page.
2. On the machine, make a user just for LocalAI, with only the rights the task needs. A command can change or delete anything that user may touch.
3. Make a key pair for that user and add the public key to its authorized_keys, or choose a password. A key with a passphrase is not supported.
4. On the add-on page, choose New SSH connection. Type the host (with the port if it is not 22), the user name and the key or the password.
5. Press Check host key. The page reads the key the machine presents and shows its fingerprint. Compare it with the one on the machine (ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub), tick that you trust it and save.
6. Press Test: the row shows the user and the system name if the login works.

One connection is one host. For a second machine, add a second connection. A chat uses the connections you enable for it in the Tools menu (see The connections of an add-on). With two or more enabled, the model names the connection in each call.

The secret

The connection keeps one sealed secret. It is either a private key in PEM or OpenSSH format, or user:password. For a password the host splits it at the first colon, so the user name has no colon and the password may have colons. This is a v1 format: when the host gets a credential kind with named fields, the add-on will move to it and you will enter the secret again.

For a key, the user name is the one in the address of the connection. The page asks for the secret again when you change the user or the host, because the saved one belongs to the old ones.

The secret is never in a result, an error, a log or a preview. The add-on also removes it from command output, in whole and in pieces. It refuses a command that holds the secret, so a model cannot ask the machine to echo it or write it to a file.

How the address is kept

The connection lists one allowed address: user@host:port#sha256=<host key fingerprint>. Every tool takes a host argument. LocalAI compares it with the address on the connection before the add-on is called and refuses any other, so a model cannot send the secret to another machine. The add-on dials only that address.

The host key

The add-on trusts a host key on first use, and you approve it when you save the connection. The page reads the key without sending any secret, shows the fingerprint, and the connection stores it as a pin (the SHA-256 of the key, as ssh-keygen -l shows it).

• The key is checked during the key exchange, before any password or key signature is sent. A machine that presents another key never sees the secret.
• If the key changes, every call is refused with a message that names both fingerprints. A changed key can mean the machine was reinstalled, or that someone is intercepting the connection.
• There is no tool or setting that accepts a new key for the model. Only you can trust a new key, by editing the connection on the page and checking the host key again.
• A connection without a pin never connects.

The mode

Each connection has a mode, set on the page by you and never by a model:

Choose Runs at once only for a host and a user where a wrong command does no harm. In that mode the model is told to say plainly which command ran. The mode is the host’s write mode of the connection (see Confirming a held write in the chat): the tool declares no approval of its own, so the connection decides. In a messaging chat, the usual rules for who may confirm apply.

Tools

run returns the exit code, standard output and standard error. Each stream is cut at 32 KiB, and a flag (stdoutTruncated, stderrTruncated) says when it was cut. A command that does not finish within the time limit gets its connection closed; the result says timedOut and holds the output so far, and the command may still run on the machine.

The limit is 45 seconds, not more, because the host stops a write call after 60 seconds and the login takes part of that.

The page also uses a third tool, scan-host-key, to read the host key for you. No model calls it.

Limits and safety

• One command at a time per connection. A second call is refused while one runs.
• Never twice. The add-on notes a command before it sends it, and a repeat of the same operation answers from that note. If the connection drops after the command started, the result is “unknown” and the add-on does not send it again.
• Output is data. Results are marked untrusted, and the skill tells the model never to follow instructions in them.
• Errors have their own words. The host cannot be reached, the login was refused, the host key changed, no host key is pinned, and the host would not run a command are different messages.
• The add-on reaches any address a connection lists, including addresses on your own network and on the LocalAI machine itself. The secret of the connection is the only limit on what a command can do there.

Studio

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

Studio is the part of the web UI where you make images, video, 3D objects, speech, sound and transformed audio, and find what you made before. It opens on a front page with a prompt box and your own work. Each type still has its own workspace page (Images, Video, 3D, TTS, Sound, Transform, Diarization) where a run happens.

Make something

Type what you want in the box under What do you want to make?, or pick a type first. The box suggests a type from your words (“Sounds like Video”) and never switches by itself. The suggestion is a short list of keywords in the browser. No model is called.

Generate opens the workspace for the chosen type with the prompt, the model, and the options that workspace has already filled in: size and count for Images, size for Video. The run itself happens on the workspace page, as before. The front page does not run anything.

3D, Transform and Diarization start from a file. Use the Start from list to pick one of your earlier results as the file, or add a file on the next page.

A type with no model

A type with no installed model is a dashed chip. Picking it shows one model from the gallery, its download size and memory need where the gallery knows them, the memory free now, and an Install button. Nothing installs until you press it, and the words you typed stay in the box while it installs. When the gallery gives no size, the note says so.

Your work

Your work lists the results this browser has a record of, newest first, with filters for each type and counts. Each tile shows a thumbnail (a waveform drawing for audio, one bar per speaker for diarization), the prompt, the model and the age. The star marks a favourite.

Results made from each other stack into one project tile. Open a tile to see the lineage.

Where the history is kept

The history lives in your browser, not on the server. Each workspace writes its own list when a result is produced: images, video, TTS, sound and audio transform in browser storage (up to 100 entries each), 3D in IndexedDB (up to 20 entries, with the model file), and diarization in browser storage with the file name, the model and the speaker count only. It never keeps the recording or the transcript. A prompt longer than 2000 characters is cut when it is stored. Favourites are a list of up to 500 ids.

Results made before this version have no link to the result they came from, so they appear as single tiles. The files themselves are on the server; if the server has cleaned its output folder, the tile shows the type icon instead of a picture.

Clear history removes all of these lists and the favourites from this browser after a confirmation. It does not delete files on the server.

Lineage

When a workspace is opened from a result (for example Animate on a picture), the new result records the id of the result it came from and how: take, animate, to 3D, variation, transform or who spoke. The lineage view draws those links as a board: prompts or files on the left, results next to them, the path through the selected result drawn heavier, and one dashed suggested next step chosen from the installed models.

The dock under the board acts on the selected result:

• Run as a new take opens the same workspace with the prompt, model and size filled in. It is disabled when the original input was not kept (3D, diarization).
• Branch from here opens a draft: choose the next step, edit the words, and Open in the workspace with the result as its starting point.

A step is listed but disabled, with the reason, when the destination page cannot start from that kind of result yet. Today a video cannot start a sound and a 3D object cannot start a clip.

Arrow keys walk the board, B branches and Esc closes the draft, then the details, then the view.

The workspace page

All seven workspaces share one layout, under a row of tabs, one per type:

• Compose card. Optional sources as dashed chips (a start image, reference images, an end image, avatar audio), or a drop area where the run cannot start without a file (a recording for Diarization, audio for Transform, a picture for 3D). Then the prompt, with starters while it is empty, a model chip, the essential options as chips (size and count, duration and frame rate, voice, mode), and an Advanced fold that names what is inside when it is closed. Below that: the memory the model needs and whether it fits, when the gallery knows it, and one button. When the button cannot run, the reason is next to it. With no model for the type, the install note from the front page shows in the card.
• Run area. While a request is out, a job card shows the model and options and the time that has passed. The server reports no phase and no percentage on these endpoints, so the bar is indeterminate. A failed run shows what the server said, says the prompt and settings are still there, and offers Try again. A finished run shows the result in a viewer for its type (picture grid, video player, waveform player, 3D viewer, spectrograms, or a timeline of speakers) with a toolbar:
  • Favourite, the same list the front page keeps.
  • Download.
  • Use in lists where the result can go next. A step is disabled, with the reason, when the destination cannot start from it, or when no model for it is installed it says so.
  • Re-run with edits puts the result’s values back in the form. The fields you then change are outlined and listed under Changed from this take, and the button reads Run again. It is disabled when the original input was not kept (3D, diarization).
  • Lineage opens the lineage view of the front page for this result.
• Recent results. A strip of the results of this type from the same history, with an All and a Favourites filter. Click a tile to show it above, and click it again to go back to the latest.

Diarization lists who spoke when as one lane per speaker, the talk time of each speaker, and the segments with their text when the model returned it. RTTM, SRT (only when there is text) and JSON are built in the browser from the result in hand.

What a workspace accepts from the front page

The workspaces read these query parameters, so a link of your own works too: prompt, model, size, n (count), from (the id of the result it starts from) and edge (how). For example /app/addons/studio/studio/video?prompt=Slow%20push-in&from=<id>&edge=animate.

Talk

The Talk add-on (ID talk) is a page for a spoken conversation with a model. It is separate from Studio. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Voice

The Voice add-on (ID voice) is the page for voice recognition, the voice library and text to speech. It is separate from the Face add-on. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Face

The Face add-on (ID face) is the page for face recognition. It is separate from the Voice add-on. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Fine-tuning

The Fine-tuning add-on (ID fine-tuning) is the Fine-Tune page and the chat tools for fine-tuning jobs. The API and the training backends are part of LocalAI. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

This feature is experimental and may change.

Quantization

The Quantization add-on (ID quantization) is the Quantize page and the chat tools for quantization jobs. The API and the quantization backend are part of LocalAI. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

This feature is experimental and may change.