Focused addon testing

The addon verification scripts separate explicit dependency setup from offline checks. Run all commands from the repository root. Only the prerequisites suite is currently available. Unknown suites, empty selections, skipped required tests, and failed commands return a nonzero status.

Required tools

Provide these tools before running setup:

Go 1.26.x, with GOTOOLCHAIN=local. Other minor versions are rejected.
Node 22.12 or later within Node 22, with npm. The validated setup uses Node 22.22.0.
protoc 26.1. The general Makefile downloader uses a different version.
A native C compiler (cc, or the executable specified by CC) with CGO_ENABLED=1.
Python 3, POSIX shell tools, and the runtime libraries required by Chromium.

Setup and integration preparation require at least 20 GiB of free disk space before each major step. They do not install system packages or fetch Go, Node, or protoc automatically. Place locally provisioned executables in .cache/addons/bin, or put them on PATH.

For Linux x86-64, the validated tool archives have these SHA-256 checksums:

Archive	Source	SHA-256
`node-v22.22.0-linux-x64.tar.xz`	Node 22.22.0 distribution	`9aa8e9d2298ab68c600bd6fb86a6c13bce11a4eca1ba9b39d79fa021755d7c37`
`protoc-26.1-linux-x86_64.zip`	Protocol Buffers 26.1 release	`a7be2928c0454f132c599e25b79b7ad1b57663f2337d7f7e468a1d59b98ec1b0`

Validate archive checksums before extraction. These archives do not apply to other architectures.

Explicit setup

Setup downloads Go dependencies, installs the pinned protobuf plugins, and runs npm ci against the React lockfile. It then installs Chromium through the local Playwright CLI, unless you supply an existing browser.

sh scripts/addons/prepare.sh --prepare

The protobuf plugin pins match the Makefile:

google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af

Go resolves the second pin to module version v1.4.0. The check validates each plugin’s embedded module version and checksum, not its version label alone. Go downloads use module checksum authentication. npm uses the lockfile’s package integrity records.

The lockfile currently selects Playwright 1.58.2 and Chromium revision 1208 (Chrome 145.0.7632.6). The installed Playwright package selects the browser revision; the scripts never invoke npx. To provision only the browser after npm setup, run:

export PATH="$PWD/.cache/addons/bin:$PATH"
export PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/addons/browsers"
core/http/react-ui/node_modules/.bin/playwright install chromium

Keep PLAYWRIGHT_BROWSERS_PATH set to this worktree-local path for manual installation. The preparation and verification scripts set it automatically. A browser installed in Playwright’s default user cache does not satisfy this setup.

Alternatively, set PLAYWRIGHT_CHROMIUM_PATH to an absolute executable path before setup and verification. Setup then skips the browser download. Verification still requires a successful browser launch and rendered UI. Neither option installs OS libraries. If launch fails, inspect the reported library names and the browser executable with ldd on Linux. Provide a compatible runtime environment or report the missing libraries as a blocker; do not skip the browser probe.

Prepare generated inputs

After explicit setup, generate only the inputs needed by the focused tests:

sh scripts/addons/prepare.sh --integration

This command generates pkg/grpc/proto, builds the React distribution, and builds two auth-enabled test binaries:

tests/e2e/mock-backend/mock-backend
tests/e2e-ui/ui-test-server

It does not build inference backends or run a full repository build. It disables Go dependency downloads and requires the populated setup cache. Repeat this step when the proto, UI, or test-server sources change.

Check and verify

Inspect the toolchain, installed dependencies, and generated inputs without downloading or rebuilding them:

sh scripts/addons/prepare.sh --check

This read-only dependency check is not a browser or SQLite runtime test. Run the complete readiness gate separately:

sh scripts/addons/verify.sh prerequisites

Verification performs these checks:

Run isolated shell fixtures for missing tools, incompatible versions, failed children, invalid reports, and empty or skipped selections.
Inspect the real toolchain and generated inputs.
Compile and execute a SQLite transaction with CGO, Ginkgo v2, and Gomega.
Start the real UI test server with the mock backend on an isolated loopback port.
Wait for server health, launch Chromium, and require a rendered React root.

The SQLite probe uses go test -count=1 -tags auth and a machine-readable Ginkgo report. Verification rejects missing, empty, failed, skipped, or unexpected test cases. It records child commands, output, and status in .cache/addons/logs/verify-*.jsonl. Ginkgo reports use separate unique files in the same directory. Server processes stop after the probe, including when the browser check fails.

Checks and verification never install dependencies or browsers. They set GOTOOLCHAIN=local, GOPROXY=off, GOSUMDB=off, and GOFLAGS=-mod=readonly for verification builds. The npm dependency check uses --offline. Missing cached modules or browser files are failures, not permission to download them.

Cache layout and test inventory

The scripts keep caches and transient results under .cache/addons:

Path	Purpose
`bin`	Local tools and protobuf plugins; prepended to `PATH`
`go`, `go/pkg/mod`	`GOPATH` and `GOMODCACHE`
`go-build`	`GOCACHE`
`npm`	npm cache
`browsers`	Playwright browser installations
`tmp`	Setup, fixture, and verification scratch data
`logs`	Command results and Ginkgo reports
`downloads`	Optional archives from explicit local provisioning

Do not commit caches, generated distributions, or test binaries. Keep the command and required-case inventory in scripts/addons/suites.json synchronized with the runner. Add future suites only when their actual tests and commands exist. A successful prerequisite gate demonstrates readiness, not acceptance of addon functionality that has no tests yet.

Gateway verification

Run sh scripts/addons/verify.sh gateway for the owner carrier and real-auth HTTP suites. The inventory lists each report separately and requires their exact combined case count. Missing reports, skipped cases, empty reports, and unsuccessful child commands fail the gate. The tests use local certificate authorities, pinned mTLS peers, two receiving frontends, and a real SQLite identity database. No external identity service or addon port is exposed.

The disconnect fixture consumes the POST envelope before waiting for cancellation. Leaving that body unread prevents the HTTP/1 server from observing disconnect while its handler waits, which can strand test-server cleanup. Production owner handlers decode the bounded envelope before starting a relay.

Data recovery suite

sh scripts/addons/verify.sh recovery

The recovery suite checks backup, upgrade with a migration, restore and purge of add-on data. scripts/addons/recovery.sh runs, in this order: the node unit tests of the panel rules, the typed client and the locale keys; the Go specs of core/services/addons/recovery (every journal phase of an upgrade and of a restore, with failures injected through test-only seams, cancellation, a corrupt backup, too little storage, retention, grant resurrection and purge), the specs of the policy hold and restore filter, the specs of the Tables format migration, and tests/addons/recovery, which runs the real owner, real accounts and the real Tables executable over HTTP; and a Playwright spec against a UI test server. Set RECOVERY_SHOTS to a directory to keep the screenshots of the Playwright run. The suite is part of the foundation gate. Run sh scripts/addons/prepare.sh --integration first: the runner refuses a server older than the recovery code or a UI build older than the panel.

Foundation acceptance gate

The foundation gate is the end-to-end check of everything below the first add-on: the owner, the carrier, the lifecycle, the policy, the proposals, the file results, the Apps surface and the UI asset route. It is one integration fixture, not a set of fakes at the seams.

sh scripts/addons/verify.sh foundation-acceptance   # the end-to-end suite
sh scripts/addons/foundation_gate.sh                 # every foundation suite

Run sh scripts/addons/prepare.sh --integration first. The gate builds the server, the protocol fixture and the cluster command itself, so it never tests a stale binary.

The fixture

tests/addons/foundation/cluster starts three server processes of tests/e2e-ui: the owner and two frontends (front-a, front-b). Each has its own listener, data directory and carrier identity. They share one account database, and they speak the real pinned mTLS carrier. The frontends open no add-on storage and start no process. The add-on process is the protocol fixture (tests/addons/fixture/cmd), packaged three ways from a catalog served on loopback after the host was built:

Package	Declares
`found-ui`	a service, the tools `echo` and `export`, and a UI module
`found-tool`	a service and the same tools, no UI
`found-svc`	a service only: no tool and no UI

The fixture add-on reads fixture.json from its package. A mode changes how it fails: exit-start, never-ready, lose-reply (commit, then drop the reply) and orphan (ignore the closing of the startup channel). Its commits and every apply request are written to a durable log, so a spec counts side effects across restarts. Specs use real API keys and real browser sessions; the only route that is not product code is the owner’s loopback control route that registers a resource, because the fixture has no add-on that creates one.

What the gate proves

Case group	Proves
install, activate, alternate	An install and an enable through a frontend reach the owner, start one process per add-on under the owner only, and reads alternate through the owner and both frontends
host unchanged	The host binary and the shipped UI build have the same digest after the whole lifecycle
actor propagation	The add-on sees the actor of the credential; forged actor and authority headers change nothing, and a forged administrator cannot reach lifecycle
JSON, progress, files, assets	JSON and file relays and lifecycle progress work through a non-owner frontend, a private file downloads through another frontend, and UI assets are served by digest, immutable and only for declared files
denials	A missing feature, a stranger, a stale generation and an undeclared tool give the same typed denial on every node
exactly once	A confirmed change reaches the add-on once, even confirmed again through another frontend; a lost reply is `outcome_unknown` with the original operation ID and is settled by receipt lookup with no second apply
owner down	Every frontend answers 503 with the same body, starts nothing and creates no storage; a restart brings back the last good package and the retained data
no duplicate owner	A second owner on the same storage does not keep running, and each add-on keeps one process
orphans	A process that outlived its owner is stopped before the next one starts
failed upgrade and enable	A new generation that cannot start leaves the last good one running; killing the owner in the middle of an upgrade or an enable keeps the last good code, the retained data and a usable lifecycle
disable, remove, reinstall	Data and grants outlive the removal, and a stale generation and its assets are denied after an upgrade
sessions and keys	Ending a session or revoking a key on one frontend refuses it on every node
browser	A member sees the Apps list only and the server refuses every management request; Open works through a non-owner frontend; a proxy path prefix works for deep links; a session ended on one frontend loses the assets on the others
service layer	An upgrade that cannot start keeps the last good generation running and the data; a request canceled during the start puts the last good generation back, keeps the operation open and the same operation ID finishes it; one broken add-on does not stop the owner
regression	The existing model and backend gallery specs (`core/gallery`), the gallery operation specs (`core/services/galleryop`) and the process adapter specs (`pkg/model`) pass

The regression packages are the narrowest ones that run offline in under two minutes each and that cover the code the add-on lifecycle sits beside. The core/gallery model install specs run with FIXTURES=tests/fixtures. The only skips the gate accepts are the specs that state they need macOS.

One pkg/model spec, “includes the exit code and final stderr line for an unexpected exit”, replaces the process-wide logger to capture a warning. It is a test of the upstream package and the gate does not change it. Run together with the other specs of the package, goroutines of those specs can log while the logger is swapped and fail it at random. The gate therefore runs the package three times: a dry run that fixes the inventory, a run that skips that one spec, and a run that focuses on it alone. The case passes only when the dry run lists exactly one such spec, the second run passes every other spec (or skips it for macOS) and the third run passes that spec once. A spec that is not accounted for in exactly one of the runs fails the case. A real failure of the spec still fails the gate, with no retry.

Reading the result

foundation_gate.sh runs every suite in the gate set, in order, and prints the inventory: the passed count and the skipped count of each suite, and the total. It exits nonzero when a suite fails, reports a skipped case, or is missing from the set. A suite that is added to suites.json must be named in the script. The report of the parent review is this inventory plus the case list of foundation-acceptance in scripts/addons/suites.json.

Packaging suite

sh scripts/addons/verify.sh packaging checks the first-party OCI package, its release channels and its workflow. It runs scripts/addons/package_test.sh, which prints one JSON line for each case. It needs no registry, no Docker and no network.

Go specs of pkg/addons/packaging. They build real layouts and then rewrite one part of them to prove each rule: missing entrypoint, missing declared UI, missing runtime library, manifest or platform mismatch, unexpected layers or media types, digest mismatch, invalid permissions, traversal paths, a bundled React runtime and a partial platform set. They cover the event and channel table (empty, Unicode and long refs, latest, development, semver-like names and two refs that normalize alike). They publish to an in-process registry and check immutable tags, digest-identical retries, stale promotions and a failed promotion that leaves aliases unchanged. They rewrite a branch so that the commit an alias names is gone from the checkout, and check that only a push to the channel’s own ref takes it over, that development refuses, and that other git errors fail. The workflow policy has one mutated copy of the workflow for each rule, including the validate chain (publish must need validate, validate must not use continue-on-error, always() or failure()).
tests/addons/packaging. It builds the Tables executable, runs package.sh, publishes to a loopback distribution, signs the platform image with the fixture authority under the identity of the rendered catalog entry, installs it with the real installer and starts the executable under the real supervisor. A signature of another workflow identity, an owner policy that names the backend workflow and a wrong pinned digest are each refused.
Shell cases for package.sh, channel.sh and addonpkg policy, and static checks that the packaging scripts never push, tag, sign or log in and never name a backend registry.

The suite never signs, pushes or creates a tag against a real registry. The workflow itself (registry login, keyless signing, the verification command) is covered by the policy assertions and by review, not by a live run: running it needs the protected environment of the repository.

Tables acceptance

sh scripts/addons/verify.sh tables-acceptance is the end-to-end acceptance of the first-party Tables add-on. sh scripts/addons/tables_gate.sh runs every tables-* suite and then this one, in order, and prints the inventory like the foundation gate. It fails for a suite that is skipped, that leaves a process behind or that is added to suites.json and not named in the script.

What it runs, and what is real:

The cluster is the foundation cluster of tests/addons/foundation/cluster: one owner and two forwarding frontends, each a separate server process with its own listener, one shared account database and the real pinned mTLS carrier. The accounts are real: an administrator, the owner of a database, an editor, a viewer, a stranger and an account without the add-ons feature, each with a stored session and an API key.
The package is the real Tables. scripts/addons/package.sh builds the OCI layout of this platform (twice, for two versions). The suite checks every layer against its digest, joins the layers into the archive a catalog on loopback serves, and installs it through a frontend as an administrator does. Nothing is pushed, tagged or signed, and a receipt that says publishable is refused.
Calls use the public HTTP routes only. There is no route into the add-on that a user could not use, and no seam in the add-on: crashes are SIGKILL, a stopped process is SIGSTOP, an owner outage is a killed owner process.
Reads of the database by the test are read-only SQLite opens of the add-on’s own file, to count the receipts it committed.

The specs, by what they prove:

Area	Specs
Roles	Owner, editor, viewer, administrator, stranger and an account without the feature, over the browser module route, the REST routes, the typed proposal route and the chat tool consumer, each through a different node. Blocked consumers (`/v1/messages`, `/v1/responses`, `/v1/mcp/chat/completions`) fail with the typed `addon_tools_unsupported`.
Credentials	A paused key, a withdrawn feature, a lowered role (and the proposals it held), a revoked grant and a disabled account are denied on the next call on every node.
Authority	Forged actor and forwarding headers, authority, role and approval fields, and every way to confirm a proposal without the proposer’s own tab (a key with the right headers, the owner’s tab, an administrator, a missing header, a tool-call marker, a wrong digest).
Consistency	Schema and data staleness from two tabs, replay of an operation ID from every node, a conflicting request under the same ID, a burst from three nodes with exactly one receipt for each acknowledged ID, and a revocation that races a confirmation.
Crash	The add-on process killed in the middle of a batch and the owner killed and restarted in the middle of a batch: every acknowledged change survives, no change is applied twice, and a worker repeats only the same ID. A stopped and killed process leaves an unresolved confirmation that blocks other writes until the owner role resolves it.
Lifecycle	Disable, enable, uninstall and install: stale tabs and proposals of the older generation do nothing, and data and grants remain.
Files	CSV through a frontend, downloads from every node by the requester only, a revoked viewer, a disabled add-on.
Outage	With the owner down, a JSON read, the JSON relay, progress, status, file status, file download, an asset, the apps list, a proposal, the pending list, a confirmation, an apply and a chat tool call all fail closed (503 with the same typed answer from both frontends), and the frontends run nothing and create no storage.
Recovery	Backup, verify and restore by the administrator’s tab; a corrupt backup, a running add-on, a wrong confirmation, an unknown backup.
Upgrade	An explicit check, a redacted status for a member, a key refused, a stale or changed pin refused, one pinned upgrade, the data and grants kept, a backup made first, and a replay of the same operation from another node.
Browser	Approval cards from chat on one frontend and the result on the other, prefix deep links, keyboard, 400 pixels, both themes, the unavailable page while the owner is down and the reload after it is back, and nothing left after logout.

The evidence manifest

Each run writes .cache/addons/tables-acceptance-evidence.json: the commands the run executed, the case inventory with the state of every case, the native platform (operating system and architecture), the digests of both packaged versions (the archive the catalog serves and the image of the OCI layout), a statement that nothing was published, and, for a case that failed, the path of the retained trace (the logs of the three servers). A Go test (TestEvidenceManifest) checks it against the inventory of scripts/addons/suites.json: the platform is native, the digests are well formed and distinct, every case passed, and a failed case has a trace that exists.

Faults the real executable cannot show on its own

Three cases need a fault that the product executable does not have, and the product executable gets no flag or variable for them. The suite builds a second, test-only executable from addons/tables/cmd/tables-faults, packages it with the same scripts/addons/package.sh and installs it on one cluster. It serves exactly what the product serves. A spec arms a fault by creating a file in the fault directory of the add-on’s data:

crash-after-commit ends the process after the next change committed and before the answer is sent. The spec checks that the host reports the original operation ID as unknown from the frontend, that the receipt lookup finds it committed, and that the same operation again returns the same receipt: one row and one receipt in total.
hold-output keeps a stream of the add-on from being served, so a transfer stays in flight. The spec disables the add-on and checks that the artifact ends without being downloadable on any node, that no byte stays on disk and that a new export works.

The expiry spec needs a short artifact lifetime. The fixture’s cluster configuration has an artifactExpiryMillis field, which the test launcher passes to the owner as the programmatic AddonConfig.ArtifactExpiry option. No flag sets it and the default stays one hour. The spec checks that every node gives the same denial for the expired file and that the database and the backups are untouched.

What it does not prove

Other platforms. The evidence is of the platform it ran on. Advertising another platform needs a native run on that platform; the platform matrix keeps it disabled.
The free space seam is a storage bound that needs a seam the real add-on does not have. The recovery suite covers it with an in-process seam. The expiry and the withdrawal of an artifact are covered here (see below).
A needs-recovery upgrade needs a data format change that fails to migrate. The recovery suite covers it; this suite covers the successful upgrade and the restore.
Which interleaving a race takes. The race and the crash specs accept whatever order happens and check the invariants. The committed-and-lost-reply case is not left to chance: see the fault specs below.
A refused request is a typed 400. A request that the add-on refuses as invalid (an unknown table, an unknown field) is 400 invalid_request on every route: the runtime client types the add-on’s HTTP 400 and 422 as a refusal, and the host answers it the same way from the apply, read, proposal, chat and artifact paths, and from the owner and every frontend. The authorization specs pin the exact status.