https://localai.io/docs/reference/index.htmlReferenceHugoenThu, 30 Jul 2026 12:11:56 +0200https://localai.io/docs/reference/shell-completion/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/reference/shell-completion/index.htmlLocalAI provides shell completion support for bash, zsh, and fish shells. Once installed, tab completion works for all CLI commands, subcommands, and flags. Generating Completion Scripts Use the completion subcommand to generate a completion script for your shell: local-ai completion bash local-ai completion zsh local-ai completion fish Installation Bash Add the following to your ~/.bashrc:https://localai.io/docs/features/p2p/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/features/p2p/index.htmlLocalAI supports peer-to-peer (P2P) networking for distributed inference. The P2P API endpoints allow you to monitor connected worker and federated nodes, retrieve the P2P network token, and get cluster statistics. For an overview of distributed inference setup, see Distributed Inference. Endpoints List all P2P nodes Method: GET Endpoint: /api/p2p Returns all worker and federated nodes in the P2P network.https://localai.io/docs/reference/system-info/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/reference/system-info/index.htmlLocalAI provides endpoints to inspect the running instance, including available backends, loaded models, and version information. System Information Method: GET Endpoint: /system Returns available backends and currently loaded models. Response Field Type Description backends array List of available backend names (strings) loaded_models array List of currently loaded models loaded_models[].id string Model identifier Usage curl http://localhost:8080/system Example response { "backends": [ "llama-cpp", "huggingface", "diffusers", "whisper" ], "loaded_models": [ { "id": "my-llama-model" }, { "id": "whisper-1" } ] } Version Method: GET Endpoint: /version Returns the LocalAI version and build commit.https://localai.io/docs/model-compatibility/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/model-compatibility/index.htmlBesides llama based models, LocalAI is compatible also with other architectures. The table below lists all the backends, compatible models families and the associated repository. Note LocalAI will attempt to automatically load models which are not explicitly configured for a specific backend. You can specify the backend to use by configuring a model with a YAML file. See the advanced section for more details. All backends listed here can be installed on demand from the Backend Gallery. The exact set of acceleration variants published for each backend is defined in backend/index.yaml.https://localai.io/docs/reference/architecture/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/reference/architecture/index.htmlLocalAI is an API written in Go that serves as an OpenAI shim, enabling software already developed with OpenAI SDKs to seamlessly integrate with LocalAI. It can be effortlessly implemented as a substitute, even on consumer-grade hardware. This capability is achieved by employing various C++ backends, including ggml, to perform inference on LLMs using both CPU and, if desired, GPU. Internally LocalAI backends are just gRPC server, indeed you can specify and build your own gRPC server and extend LocalAI in runtime as well. It is possible to specify external gRPC server and/or binaries that LocalAI will manage internally.https://localai.io/docs/reference/cli-reference/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/reference/cli-reference/index.htmlComplete reference for all LocalAI command-line interface (CLI) parameters and environment variables. Note: All CLI flags can also be set via environment variables. Environment variables take precedence over CLI flags. See .env files for configuration file support. Global Flags Parameter Default Description Environment Variable -h, --help Show context-sensitive help --log-level info Set the level of logs to output [error,warn,info,debug,trace] $LOCALAI_LOG_LEVEL --debug false DEPRECATED - Use --log-level=debug instead. Enable debug logging $LOCALAI_DEBUG, $DEBUG Storage Flags Parameter Default Description Environment Variable --models-path BASEPATH/models Path containing models used for inferencing $LOCALAI_MODELS_PATH, $MODELS_PATH --data-path BASEPATH/data Path for persistent data (collectiondb, agent state, tasks, jobs). Separates mutable data from configuration $LOCALAI_DATA_PATH --generated-content-path TMPDIR/localai-UID/generated/content Location for assets generated by backends (e.g. stablediffusion, images, audio, videos). Defaults under the OS temp dir ($TMPDIR, falling back to /tmp), scoped to the current user’s UID so accounts sharing a host never collide. $LOCALAI_GENERATED_CONTENT_PATH, $GENERATED_CONTENT_PATH --upload-path TMPDIR/localai-UID/upload Path to store uploads from files API. Defaults under the OS temp dir ($TMPDIR, falling back to /tmp), scoped to the current user’s UID. $LOCALAI_UPLOAD_PATH, $UPLOAD_PATH --localai-config-dir BASEPATH/configuration Directory for dynamic loading of certain configuration files (currently runtime_settings.json, api_keys.json, and external_backends.json). See Runtime Settings for web-based configuration. $LOCALAI_CONFIG_DIR --localai-config-dir-poll-interval Time duration to poll the LocalAI Config Dir if your system has broken fsnotify events (example: 1m) $LOCALAI_CONFIG_DIR_POLL_INTERVAL --models-config-file YAML file containing a list of model backend configs (alias: --config-file) $LOCALAI_MODELS_CONFIG_FILE, $CONFIG_FILE Backend Flags Parameter Default Description Environment Variable --backends-path BASEPATH/backends Path containing backends used for inferencing $LOCALAI_BACKENDS_PATH, $BACKENDS_PATH --backends-system-path /var/lib/local-ai/backends Path containing system backends used for inferencing $LOCALAI_BACKENDS_SYSTEM_PATH, $BACKEND_SYSTEM_PATH --external-backends A list of external backends to load from gallery on boot $LOCALAI_EXTERNAL_BACKENDS, $EXTERNAL_BACKENDS --external-grpc-backends A list of external gRPC backends (format: BACKEND_NAME:URI) $LOCALAI_EXTERNAL_GRPC_BACKENDS, $EXTERNAL_GRPC_BACKENDS --backend-galleries JSON list of backend galleries $LOCALAI_BACKEND_GALLERIES, $BACKEND_GALLERIES --autoload-backend-galleries true Automatically load backend galleries on startup $LOCALAI_AUTOLOAD_BACKEND_GALLERIES, $AUTOLOAD_BACKEND_GALLERIES --max-active-backends 0 Maximum number of active backends (loaded models). When exceeded, the least recently used model is evicted. Set to 0 for unlimited, 1 for single-backend mode $LOCALAI_MAX_ACTIVE_BACKENDS, $MAX_ACTIVE_BACKENDS --single-active-backend false DEPRECATED - Use --max-active-backends=1 instead. Allow only one backend to be run at a time $LOCALAI_SINGLE_ACTIVE_BACKEND, $SINGLE_ACTIVE_BACKEND --preload-backend-only false Do not launch the API services, only the preloaded models/backends are started (useful for multi-node setups) $LOCALAI_PRELOAD_BACKEND_ONLY, $PRELOAD_BACKEND_ONLY --enable-watchdog-idle false Enable watchdog for stopping backends that are idle longer than the watchdog-idle-timeout $LOCALAI_WATCHDOG_IDLE, $WATCHDOG_IDLE --watchdog-idle-timeout 15m Threshold beyond which an idle backend should be stopped $LOCALAI_WATCHDOG_IDLE_TIMEOUT, $WATCHDOG_IDLE_TIMEOUT --enable-watchdog-busy false Enable watchdog for stopping backends that are busy longer than the watchdog-busy-timeout $LOCALAI_WATCHDOG_BUSY, $WATCHDOG_BUSY --watchdog-busy-timeout 5m Threshold beyond which a busy backend should be stopped $LOCALAI_WATCHDOG_BUSY_TIMEOUT, $WATCHDOG_BUSY_TIMEOUT --watchdog-interval 500ms Interval between watchdog checks (e.g., 500ms, 5s, 1m) $LOCALAI_WATCHDOG_INTERVAL, $WATCHDOG_INTERVAL --force-eviction-when-busy false Force eviction even when models have active API calls (default: false for safety). Warning: Enabling this can interrupt active requests $LOCALAI_FORCE_EVICTION_WHEN_BUSY, $FORCE_EVICTION_WHEN_BUSY --lru-eviction-max-retries 30 Maximum number of retries when waiting for busy models to become idle before eviction $LOCALAI_LRU_EVICTION_MAX_RETRIES, $LRU_EVICTION_MAX_RETRIES --lru-eviction-retry-interval 1s Interval between retries when waiting for busy models to become idle (e.g., 1s, 2s) $LOCALAI_LRU_EVICTION_RETRY_INTERVAL, $LRU_EVICTION_RETRY_INTERVAL --model-load-failure-cooldown 10s After a model load fails, refuse new load attempts for that model for this long (HTTP 503 + Retry-After) so a client polling a broken model doesn’t respawn a crashing backend every request. Doubles per consecutive failure up to 5m; reset on success. 0 disables $LOCALAI_MODEL_LOAD_FAILURE_COOLDOWN, $MODEL_LOAD_FAILURE_COOLDOWN For more information on VRAM management, see VRAM and Memory Management.https://localai.io/docs/reference/api-errors/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/reference/api-errors/index.htmlThis page documents the error responses returned by the LocalAI API. LocalAI supports multiple API formats (OpenAI, Anthropic, Open Responses), each with its own error structure. Note For backend and runtime failure messages (a model that will not load, could not load model, grpc service not ready, SIGILL, VRAM out of memory, the model-load cooldown 503), rather than API-envelope errors, see /docs/reference/runtime-errors/. Error Response Formats OpenAI-Compatible Format Most endpoints return errors using the OpenAI-compatible format:https://localai.io/docs/reference/binaries/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/reference/binaries/index.htmlLocalAI binaries are available for both Linux and MacOS platforms and can be executed directly from your command line. These binaries are continuously updated and hosted on our GitHub Releases page. This method also supports Windows users via the Windows Subsystem for Linux (WSL). macOS Download You can download the DMG and install the application: Note: the DMGs are not signed by Apple as quarantined. See https://github.com/mudler/LocalAI/issues/6268 for a workaround, fix is tracked here: https://github.com/mudler/LocalAI/issues/6244https://localai.io/docs/reference/runtime-errors/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/reference/runtime-errors/index.htmlThis page maps the runtime and backend error messages you actually see in the logs (or in an API response) to their likely cause and fix. It covers failures that happen while a model is loading or running, as opposed to API-envelope validation errors (bad request shape, unknown field, wrong content type), which are documented in /docs/reference/api-errors/. If you only have an HTTP 500 and no message, read How to read the real error first: the useful text is almost always in the server log, not in the HTTP body.https://localai.io/docs/reference/nvidia-l4t/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/reference/nvidia-l4t/index.htmlLocalAI can be run on Nvidia ARM64 devices, such as the Jetson Nano, Jetson Xavier NX, Jetson AGX Orin, and Nvidia DGX Spark. The following instructions will guide you through building and using the LocalAI container for Nvidia ARM64 devices. Platform Compatibility CUDA 12 L4T images: Compatible with Nvidia AGX Orin and similar platforms (Jetson Nano, Jetson Xavier NX, Jetson AGX Xavier) CUDA 13 L4T images: Compatible with Nvidia DGX Spark Prerequisites Docker engine installed (https://docs.docker.com/engine/install/ubuntu/) Nvidia container toolkit installed (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-ap) Pre-built Images Pre-built images are available on quay.io and dockerhub:https://localai.io/docs/reference/ai-coding-assistants/index.htmlMon, 01 Jan 0001 00:00:00 +0000https://localai.io/docs/reference/ai-coding-assistants/index.htmlThis document provides guidance for AI tools and developers using AI assistance when contributing to LocalAI. LocalAI follows the same guidelines as the Linux kernel project for AI-assisted contributions. See the upstream policy here: https://docs.kernel.org/process/coding-assistants.html. The rules below mirror that policy, adapted to LocalAI’s license and project layout. AI tools helping with LocalAI development should follow the standard project development process: CONTRIBUTING.md - development workflow, commit conventions, and PR guidelines AGENTS.md - the agent entry point with links to all detailed topic guides .agents/ai-coding-assistants.md - the full policy source of truth Licensing and Legal Requirements All contributions must comply with LocalAI’s licensing requirements: