Gmail

The Gmail add-on lets your saved agents read the mail of your Google account. If you allow it, they can also save a draft and send a draft that exists. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Everything the agent reads goes to the model of the chat. Use the label list of the connection to keep an agent to one label.

Mail

The Mail add-on lets your saved agents read a mailbox over IMAP and, if you allow it, write mail over SMTP. It works with any provider that offers both protocols. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Mail text goes to the model the chat uses.

Google Drive

The Google Drive add-on lets your saved agents read the files in the Drive folders you choose. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

GitHub

The GitHub add-on lets your saved agents read the repositories you choose: code, files, issues, pull requests and their diffs. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

The add-on does not listen for @mentions on GitHub. An agent works on GitHub only when you ask it in a chat.

Jellyfin

The Jellyfin add-on lets your chats and saved agents look into a Jellyfin media server. In the current development builds, it is in the development channel of the add-on library, so it can change.

At a glance

Home Assistant

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

The Home Assistant add-on lets your chats and saved agents read your smart home: find entities, read a state, look at a history, list what is unavailable. If you allow it, they can also switch things, such as a light or a thermostat. You set the limits. The add-on never operates a lock, an alarm, a garage door or a siren, whatever you choose.

A chat can answer questions such as:

• “Is the office warmer than the bedroom, and was it warmer yesterday?”
• “Which devices are unavailable?”
• “Turn off the lights in the living room.” (a change, see Changes)

Set it up

1. Install the Home Assistant add-on from the add-on gallery and open its page.
2. In Home Assistant, open your profile (your name at the bottom of the sidebar), then the Security tab. Under Long-lived access tokens, choose Create token, name it (for example LocalAI) and copy it. Home Assistant shows it once. The token has the rights of the user that made it. If you can, make it with a user that sees only what you want to share.
3. On the add-on page, choose New Home Assistant connection. Paste the token and write the address you use to open Home Assistant in a browser, for example https://homeassistant.local:8123. Press Test: the row shows the Home Assistant version if the address and the token work.
4. Leave the write mode on Read only until you have read the limits on this page.

One connection is one home: one token and one address. For a second home, add a second connection. A chat uses the connections you enable for it in the Tools menu (see The connections of an add-on). With two or more enabled, the model names the connection in each call.

How the address is kept

The address is not a secret, so it is not stored with the token. The connection lists it as its one allowed resource, the same list the GitHub add-on uses for repositories. Every tool takes an instance argument. LocalAI compares it with the address on the connection before the add-on is called and refuses any other, so a model cannot send the token to another host. The add-on talks only to that address:

• The address is https. Plain http is accepted only for an address on your own network (a private IP address, a single-word name, or a name ending in .local, .lan, .home, .internal or .home.arpa).
• A reply that redirects is an error. The token is never sent to another host.
• The token goes in the Authorization header and nowhere else. It is not in a log, an error or a result.

A self-signed certificate

The add-on checks the certificate of Home Assistant. If yours is self-signed, the Test says the certificate is not trusted. Do not turn the check off. Pin the certificate instead: copy its SHA-256 fingerprint and add it to Pinned certificates in the limits below, as host:port=fingerprint.

openssl s_client -connect homeassistant.local:8123 </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

The add-on then accepts that exact certificate for that exact host:port, and nothing else. When the certificate is renewed, pin the new one.

What the tools do

Two more tools, get-policy and set-policy, belong to the add-on page. A model never sees them.

There is no tool that renders a template, fires an event or sets a state, because each can run arbitrary logic on your server.

Everything Home Assistant returns (names, states, attributes, logbook text) was written by devices, integrations and people. Every result is marked as untrusted data, and the add-on’s skill tells the model never to follow an instruction found in one. Results are bounded: a list is cut and says so, text is cut, an attribute value is cut, and a result above the size limit is refused.

The add-on removes these from every result: entity_picture, access_token, anything with token, password or secret in its name, a value that carries a token in an address, and the location of people and devices (latitude, longitude, gps_accuracy and similar). It never returns a camera snapshot address.

Limits you set

The limits are on the add-on page, under the connections. Everyone who opens the page sees them. Only an administrator can change them. They apply to every Home Assistant connection of the server.

Services a model may call

The allowlist holds domain.service entries, or domain.* for the services of a domain. A service that is not on it is refused. The default is small:

notify services are not on the list. Add one only if you want a model to send messages.

A script can do anything you wrote into it, and a switch can power a heater. Allow only what you would let a guest in your house do.

Never available

These cannot be allowed. The add-on refuses them when you save the limits, and again when a call is made, so a hand-edited file does not change it:

You can still let a chat read what a lock reports (see below). Reading is never operating.

Entities the tools never show or touch

Excluded entities are left out of every search, state, history, logbook and area count, and no service call can name them. You can exclude by:

• Domain. Default: lock, alarm_control_panel and camera. Remove one to let a chat read it. A lock you read is still never operated.
• Area. The entities of the areas you list. If Home Assistant cannot answer which entities an area holds, the tools refuse instead of guessing.
• Pattern. A regular expression matched against the whole entity ID, such as sensor\..*_bedroom.
• Label. Entities that carry the Home Assistant label you name (default sensitive). An older Home Assistant that does not know labels cannot check it. The result then says that the label exclusion was not applied.

How many things one call switches

A call names its entities one by one. The default is 5, the highest you can set is 20. Area, device, label and all targets are not accepted: the model has to name each entity, so you see each one in the confirmation.

Changes and confirmation

A connection has a write mode, which the host enforces before the add-on is called:

call-physical-service is always confirmed, even in the last mode. A proposal for it expires after 15 minutes, a quiet one after two hours. Changing the limits after a preview makes an old proposal stale: it is rejected and nothing is sent.

Home Assistant has no idempotency key. The add-on notes an operation before it sends it and never sends one twice. If the connection breaks after the call left, the result is “unknown”, and you decide what to do.

Install notes

• The add-on asks the host, through the workspace.admin operation, whether the person who saves the limits is an administrator. The shipped host policy (deploy/addons/host-policy.json) grants it. Without that grant, the limits can be read and not changed.
• The limits are kept in policy.json in the add-on’s data folder. A file that cannot be read, or that names a service that is never available, is refused as a whole: the tools stop working until an administrator saves the limits again. They do not fall back to something wider.
• The add-on runs one process for the server, so every frontend of a cluster reads the same limits.
• It is a Linux and macOS add-on, like the other tool add-ons.

Not verified without a real instance

The tests run against a fake Home Assistant that speaks the REST API. The fake answers the fixed templates of the add-on (areas and labels) by their shape, so a real instance can still differ: area and label questions use POST /api/template with fixed templates and no text from a model. If an area question fails on your Home Assistant, list-areas says so and the area exclusion refuses to guess.

PiKVM

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

The PiKVM add-on lets your chats and saved agents look at a PiKVM (or another device that runs the kvmd API) and the machine it controls: is it on, what is the device, is the keyboard connected. If you allow it, they can also press the power buttons of that machine and type on its keyboard. A person confirms those actions unless you choose otherwise for power actions and key presses (see Not asking for some actions and Allow all), and typing and forced power actions are off until an administrator switches them on.

A chat can answer questions such as:

• “Is the test box on?”
• “Power the test box on.” (a power action, see Power actions)
• “Press Escape and F2 to enter the BIOS.” (keys, see Typing and keys)

Set it up

1. Install the PiKVM add-on from the add-on gallery and open its page.
2. On the PiKVM, add a user for LocalAI: switch the filesystem to read-write with rw, run kvmd-htpasswd add localai, then ro. See the PiKVM authentication guide. A PiKVM has no per-user rights: every user can do everything. A dedicated user does not limit what the add-on can do, but you can remove it on its own. The limits of the add-on (below) are the only ones.
3. Get the fingerprint of the certificate of the PiKVM (see The certificate).
4. On the add-on page, choose New PiKVM connection. Type the user name, the password and the address, for example https://pikvm.local#sha256=<fingerprint>. Press Test: the row shows the model and the kvmd version if the address and the login work.
5. Leave the write mode on Read only until you have read the limits on this page.

One connection is one device. For a second device, add a second connection. A chat uses the connections you enable for it in the Tools menu (see The connections of an add-on). With two or more enabled, the model names the connection in each call.

The user and the password

The connection keeps one sealed credential. In this version it holds user:password: the host splits it at the first colon, so the user name has no colon and the password may have colons. The page has a field for each and joins them. This is a v1 format: when the host gets a credential kind with named fields (a user and a password), the add-on will move to it, and you will enter the pair again.

The add-on sends the user and the password in the X-KVMD-User and X-KVMD-Passwd headers, as the PiKVM API allows, and nowhere else. They are not in a log, an error or a result. The page trims nothing for you: a password with a space at its start or end is refused.

Two-factor login (2FA) is not supported in this version. With 2FA on, the PiKVM wants the current one-time code added to the password, and the code changes every 30 seconds. A device with 2FA on answers 403 and the Test says the login was refused. 2FA is one switch for the whole PiKVM (kvmd-totp), so the device cannot have it on for the browser and off for LocalAI.

How the address is kept

The address is not a secret, so it is not stored with the password. The connection lists it as its one allowed resource, the same list the GitHub, Jellyfin and Home Assistant add-ons use. Every tool takes a device argument. LocalAI compares it with the address on the connection before the add-on is called and refuses any other, so a model cannot send the password to another host. The add-on talks only to that address:

• The address is https. Plain http is accepted only for an address on your own network (a private IP address, a single-word name, or a name ending in .local, .lan, .home, .internal or .home.arpa), because the login travels in headers.
• A reply that redirects is an error. The password is never sent to another host.
• The add-on does not use a proxy of the environment.

The certificate

A PiKVM comes with a self-signed certificate. The add-on never turns verification off. Pin the certificate instead: copy its SHA-256 fingerprint and add it to the address as #sha256= and the 64 hex digits (colons are fine).

openssl s_client -connect pikvm.local:443 </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

The add-on then accepts that exact certificate for that exact address, and nothing else; a different certificate is refused before the password is sent. When you renew the certificate, edit the connection and pin the new one. Without a pin, a self-signed certificate is refused and the Test says the certificate is not trusted.

What the tools do

Two more tools, get-settings and set-settings, belong to the add-on page. A model never sees them.

Everything the device returns is marked as untrusted data, and the add-on’s skill tells the model never to follow an instruction found in a result. Results are bounded.

Changes and confirmation

All three write tools declare approval: always in the manifest. By default a person confirms every call: the mode Read only refuses them all, and Asks you first stores each as a proposal that waits for a confirmation card. For power-action and send-key you can stop being asked one by one, as described next, and Allow all stops it for all three tools (see Allow all). A proposal expires after 10 minutes (power) or 5 minutes (typing and keys).

Not asking for some actions

On the add-on page, the section Ask before acting has one choice for Power actions and one for Key presses: Ask each time (the default) or Don’t ask. There is no choice for typed text, because it can enter a password or a command; it waits for you unless the connection is on Allow all.

• The choice is yours alone. It is saved from your own browser session, it applies to all your PiKVM connections, and no chat, agent or model can change it.
• With Don’t ask, a chat can switch the machine off or reset it at once, with no card, and unsaved work on it is lost. If an administrator allowed the hard power actions, those run without asking too. Read the warning on the page before you choose it.
• It holds only for your own chats and for messaging senders you paired with the bot. Senders that are only on the allowlist, senders let in because the bot accepts anyone, roles, scheduled runs and sub-agents are still asked.
• A connection set to Read only runs nothing, whatever you choose. A connection set to Allow all runs these two tools without asking whatever you choose here.
• Every run without asking is written to the server log and to the latest decisions of the connection, with the tool, the connection and the result. The typed text and the keys are not written.
• The row of each connection lists which tools run without asking now.

Allow all

The write mode of a connection has a third choice, Allow all, for someone who wants the agent to act without asking. It is never the default and is never chosen for you: a new connection starts on Read only, and a connection keeps the mode it has.

With Allow all, a chat of yours runs power-action, send-key and type-text at once, with no card. Unsaved work on the machine is lost when it is switched off or reset, and typed text and key presses go to whatever has the focus, so a shell command or a password can be entered and none of it can be taken back. The page warns about it. If you want a confirmation, choose Asks you first. The per-tool choices of Ask before acting are not asked in this mode, and the row of the connection lists both tools that have a choice as running without asking.

These rules hold in every write mode, Allow all included:

• The switches of an administrator for each device stay in force. Typing and key presses need Allow typing and key presses, the forced power actions (off_hard, reset_hard, long_press) need Allow hard power actions, and the add-on refuses a call when the switch is off. Allow all does not turn a switch on.
• The same limits on text and keys, the stale-state check and the rule that nothing is sent twice.
• A run without you present, such as an agent started by a trigger, still leaves a proposal that waits for a person.

The add-on notes an operation before it sends it and never sends one twice. The PiKVM API has no idempotency key. If the connection breaks after the request left, the result is “unknown”, and you decide what to do. Read the power state to see what happened.

Power actions

power-action takes one of these:

The confirmation card shows the power state that was read just before, and what the action does. The preview reads the state again when you confirm. If the state changed, the action is rejected as stale and nothing is sent. An action that makes no sense is refused at once: on for a machine that is on, off for one that is off, a reset for one that is off. A busy device or one with ATX control turned off is refused.

Typing and keys

type-text and send-key are off until an administrator switches on Allow typing and key presses for the device on the add-on page. Both use that one switch.

• type-text types text through the HID print endpoint. The text is at most 200 characters by default; an administrator can set up to 1000 for a device. It has no control characters except newline and tab. The card shows the exact text, the number of characters and lines. The text is never written to a log or an error, and the result of the call does not repeat it. As defence in depth, the add-on refuses text that looks like a private key (a -----BEGIN header), a long unbroken blob of base64-like characters (80 or more), a well-known token shape (cloud access keys, source host tokens, SSH public keys) or an exported list of passwords (three or more lines such as password: ...). This is a heuristic and not a guarantee: the skill also tells the model never to type a secret the person did not dictate.
• send-key presses up to 8 keys or chords, in order. Allowed: the letters a to z, the digits 0 to 9, up, down, left, right, enter, escape, tab, f1 to f12, and the chords ctrl+alt+delete, ctrl+c and ctrl+z. Windows and Meta chords, modifiers alone and every other key are refused. Single keys use POST /api/hid/events/send_key, chords use POST /api/hid/events/send_shortcut.

Both refuse when the keyboard of the device is not connected to the target.

Limits you set

The switches are on the add-on page, under the connections. Everyone who opens the page sees them. Only an administrator can change them. They are kept for each device address, so a switch for one PiKVM does not open another:

The add-on asks the host, through the workspace.admin operation, whether the person who saves the switches is an administrator. The shipped host policy (deploy/addons/host-policy.json) grants it. Without that grant, the switches can be read and not changed. A settings file that cannot be read leaves every switch off. They are kept in settings.json in the add-on’s data folder; the add-on runs one process for the server.

What is not supported

• Screenshots and screen reading. The streamer snapshot is left out of this version. The screen of the target can show secrets, and a snapshot needs a way to hand an image back that this add-on does not have yet. Without it, ask the person what the screen shows.
• Mass storage. No mounting or uploading of disk images.
• GPIO, mouse, the HID reset, and the ATX click beyond what power-action covers.
• Changes to the users or the configuration of the device.
• Two-factor login. See above.
• Windows and Meta key chords.

Not verified without a real device

The tests run against a fake that speaks the parts of the kvmd API listed above, as the PiKVM API reference describes them. A real PiKVM can still differ. Check these on a device you can reach before you rely on the add-on:

1. The Test shows the model and the version (GET /api/info; the add-on reads hw.platform and system leniently).
2. get-power-state follows the real power LED.
3. A soft off and an on change a spare machine, and reset restarts it.
4. type-text types a short text with the layout you expect. The add-on does not set a keymap: the PiKVM default applies.
5. send-key presses escape and ctrl+alt+delete.
6. With a wrong pin, the connection is refused before the password is sent.

SSH

In the current development builds, this add-on is in the development channel of the add-on library, so it can change.

At a glance

The SSH add-on lets your chats and saved agents run commands on a remote machine over SSH. By default a person confirms each command: the card shows the host, the user and the exact text before anything runs.

A chat can answer questions such as:

• “How full is the disk on the build server?”
• “Show the last 50 lines of the service log.”
• “Restart the worker.” (a change: it waits for your confirmation)

The add-on runs one non-interactive command per call. It has no terminal, no input and no environment variables chosen by the model. It has no interactive shell, no file transfer, no port forwarding, no agent forwarding and no jump hosts.

Set it up

1. Install the SSH add-on from the add-on gallery and open its page.
2. On the machine, make a user just for LocalAI, with only the rights the task needs. A command can change or delete anything that user may touch.
3. Make a key pair for that user and add the public key to its authorized_keys, or choose a password. A key with a passphrase is not supported.
4. On the add-on page, choose New SSH connection. Type the host (with the port if it is not 22), the user name and the key or the password.
5. Press Check host key. The page reads the key the machine presents and shows its fingerprint. Compare it with the one on the machine (ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub), tick that you trust it and save.
6. Press Test: the row shows the user and the system name if the login works.

One connection is one host. For a second machine, add a second connection. A chat uses the connections you enable for it in the Tools menu (see The connections of an add-on). With two or more enabled, the model names the connection in each call.

The secret

The connection keeps one sealed secret. It is either a private key in PEM or OpenSSH format, or user:password. For a password the host splits it at the first colon, so the user name has no colon and the password may have colons. This is a v1 format: when the host gets a credential kind with named fields, the add-on will move to it and you will enter the secret again.

For a key, the user name is the one in the address of the connection. The page asks for the secret again when you change the user or the host, because the saved one belongs to the old ones.

The secret is never in a result, an error, a log or a preview. The add-on also removes it from command output, in whole and in pieces. It refuses a command that holds the secret, so a model cannot ask the machine to echo it or write it to a file.

How the address is kept

The connection lists one allowed address: user@host:port#sha256=<host key fingerprint>. Every tool takes a host argument. LocalAI compares it with the address on the connection before the add-on is called and refuses any other, so a model cannot send the secret to another machine. The add-on dials only that address.

The host key

The add-on trusts a host key on first use, and you approve it when you save the connection. The page reads the key without sending any secret, shows the fingerprint, and the connection stores it as a pin (the SHA-256 of the key, as ssh-keygen -l shows it).

• The key is checked during the key exchange, before any password or key signature is sent. A machine that presents another key never sees the secret.
• If the key changes, every call is refused with a message that names both fingerprints. A changed key can mean the machine was reinstalled, or that someone is intercepting the connection.
• There is no tool or setting that accepts a new key for the model. Only you can trust a new key, by editing the connection on the page and checking the host key again.
• A connection without a pin never connects.

The mode

Each connection has a mode, set on the page by you and never by a model:

Choose Runs at once only for a host and a user where a wrong command does no harm. In that mode the model is told to say plainly which command ran. The mode is the host’s write mode of the connection (see Confirming a held write in the chat): the tool declares no approval of its own, so the connection decides. In a messaging chat, the usual rules for who may confirm apply.

Tools

run returns the exit code, standard output and standard error. Each stream is cut at 32 KiB, and a flag (stdoutTruncated, stderrTruncated) says when it was cut. A command that does not finish within the time limit gets its connection closed; the result says timedOut and holds the output so far, and the command may still run on the machine.

The limit is 45 seconds, not more, because the host stops a write call after 60 seconds and the login takes part of that.

The page also uses a third tool, scan-host-key, to read the host key for you. No model calls it.

Limits and safety

• One command at a time per connection. A second call is refused while one runs.
• Never twice. The add-on notes a command before it sends it, and a repeat of the same operation answers from that note. If the connection drops after the command started, the result is “unknown” and the add-on does not send it again.
• Output is data. Results are marked untrusted, and the skill tells the model never to follow instructions in them.
• Errors have their own words. The host cannot be reached, the login was refused, the host key changed, no host key is pinned, and the host would not run a command are different messages.
• The add-on reaches any address a connection lists, including addresses on your own network and on the LocalAI machine itself. The secret of the connection is the only limit on what a command can do there.