Image Generation

(Generated with AnimagineXL)

LocalAI supports generating images with Stable diffusion, running on CPU using C++ and Python implementations.

Usage

OpenAI docs: https://platform.openai.com/docs/api-reference/images/create

To generate an image you can send a POST request to the /v1/images/generations endpoint with the instruction as the request body:

curl http://localhost:8080/v1/images/generations -H "Content-Type: application/json" -d '{
  "prompt": "A cute baby sea otter",
  "size": "256x256"
}'

Available additional parameters: mode, step, negative_prompt.

To set a negative prompt you can either pass a separate negative_prompt field or split the prompt with |, e.g. a cute baby sea otter|malformed. When both are present they are joined with a comma (negative_prompt first, then the |-suffixed tags), so a global negative prompt can live in negative_prompt while per-image negatives are appended after |.

curl http://localhost:8080/v1/images/generations -H "Content-Type: application/json" -d '{
  "prompt": "floating hair, portrait, ((loli)), ((one girl)), cute face, hidden hands, asymmetrical bangs, beautiful detailed eyes, eye shadow, hair ornament, ribbons, bowties, buttons, pleated skirt, (((masterpiece))), ((best quality)), colorful|((part of the head)), ((((mutated hands and fingers)))), deformed, blurry, bad anatomy, disfigured, poorly drawn face, mutation, mutated, extra limb, ugly, poorly drawn hands, missing limb, blurry, floating limbs, disconnected limbs, malformed hands, blur, out of focus, long neck, long body, Octane renderer, lowres, bad anatomy, bad hands, text",
  "size": "256x256"
}'

Backends

stablediffusion-ggml

This backend is based on stable-diffusion.cpp. Every model supported by that backend is supported indeed with LocalAI.

Setup

There are already several models in the gallery that are available to install and get up and running with this backend, you can for example run flux by searching it in the Model gallery (flux.1-dev-ggml) or start LocalAI with run:

local-ai run flux.1-dev-ggml

To use a custom model, you can follow these steps:

1. Create a model file stablediffusion.yaml in the models folder:

name: stablediffusion
backend: stablediffusion-ggml
parameters:
  model: gguf_model.gguf
step: 25
cfg_scale: 4.5
options:
- "clip_l_path:clip_l.safetensors"
- "clip_g_path:clip_g.safetensors"
- "t5xxl_path:t5xxl-Q5_0.gguf"
- "sampler:euler"

2. Download the required assets to the models repository
3. Start LocalAI

Memory and device placement options

When a model does not fit entirely in VRAM, the following options: control where weights and computation are placed. They map directly to the upstream stable-diffusion.cpp options.

The following convenience booleans are still accepted and are translated into the backend / params_backend specs above:

For example, to mmap the diffusion weights from disk while keeping the text encoder on the CPU:

options:
- "diffusion_model"
- "sampler:euler"
- "params_backend:diffusion=disk"
- "keep_clip_on_cpu:true"

VAE tiling

Sampling runs in a small latent space, but the final VAE decode expands it to full resolution and needs one large compute buffer. At 1024x1024 that buffer can exceed 8 GB, which is enough to fail on a GPU that has plenty of memory overall:

ggml_vulkan: Requested buffer size exceeds device buffer size limit: ErrorOutOfDeviceMemory
[ERROR] vae: failed to allocate the compute buffer
[ERROR] stable-diffusion.cpp - decode_first_stage failed for latent 1

Tiling decodes the latent in overlapping tiles, so peak memory scales with the tile instead of the image. It costs a little time and can leave faint seams, so it is off unless you ask for it.

options:
- "diffusion_model"
- "sampler:euler"
- "vae_tiling:true"
- "vae_tile_size:512"

Reach for this when the decode fails but sampling completed, which the backend log shows as sampling completed followed by a VAE allocation error. Two cases hit it: cards without the VRAM for a full-frame decode, and drivers that cap a single allocation regardless of free memory (Mesa RADV reports a 4 GiB maxMemoryAllocationSize, so a large decode fails there even with tens of GB free). Lowering the output resolution avoids it too, at the cost of the resolution.

ESRGAN image upscaling

The stablediffusion-ggml backend supports standalone ESRGAN upscaling through POST /v1/images/upscale. This is a separate, upscale-only model, not a diffusion pipeline. Image generation does not automatically upscale its results.

Install the gallery model:

local-ai models install realesrgan-x4plus-anime-6b

This uses the official RealESRGAN_x4plus_anime_6B.pth release, a six-block model for anime images with a native 4x scale. The PyTorch .pth checkpoint is supported directly by stable-diffusion.cpp; no GGUF conversion is needed. The upstream project is licensed under BSD-3-Clause.

For manual installation, download that checkpoint into your models directory and save the following as realesrgan-x4plus-anime-6b.yaml alongside it (the gallery installer configures these settings for you):

name: realesrgan-x4plus-anime-6b
backend: stablediffusion-ggml
parameters:
  model: RealESRGAN_x4plus_anime_6B.pth
known_usecases: [upscale]
options:
  - upscale_scale:4
  - upscale_tile_size:128

upscale_scale and upscale_tile_size are backend options, expressed as key:value strings in options, not root-level configuration fields. upscale_scale must be a positive integer. Keep known_usecases explicitly set to [upscale]: do not add image or diffusion_model options to this configuration. Options are forwarded unchanged; the backend rejects invalid or duplicate recognized settings.

The multipart fields model, image, and scale are required. The HTTP endpoint accepts only integer scales from 1 through 16, rejecting missing, malformed, zero, negative, or larger values with HTTP 400. This range is not a promise that each model supports every scale: for ESRGAN, the configured upscale_scale, the loaded model’s detected native scale, and the request’s scale must all match exactly. Mismatches fail rather than silently resizing or running multiple passes. For this model, always request scale=4.

curl http://localhost:8080/v1/images/upscale \
  -F model=realesrgan-x4plus-anime-6b \
  -F scale=4 \
  -F image=@input.png

The response follows the image-generation response format, with the output URL in data[0].url under /generated-images. A successful 4x upscale multiplies both width and height by four.

upscale_tile_size controls ESRGAN tiling in pixels: omit it or set it to 0 for stable-diffusion.cpp full-frame processing; set a positive value (such as the gallery entry’s 128) to enable tiling. Smaller tiles reduce working memory at the cost of more tile work. Full-frame processing can use much more memory. upscale_direct is a separate convolution-algorithm flag, not a tile control. Tile settings apply to ESRGAN, not the diffusion model’s separate VAE tiling settings.

In Studio, open an existing image, choose Upscale, select an installed upscaler, and submit. The scale is read-only and comes from the selected model’s configured scale. A successful result becomes an Upscale child node of the source image, leaving the original intact. Open that child to continue the workflow with the available image actions, including another upscale or image editing; each upscale is an explicit operation, not a post-generation default.

Unlike the Diffusers upscaling described below, this path uses ESRGAN inference and does not fall back to Lanczos resizing.

Distributed inference (RPC workers)

The stablediffusion-ggml backend can offload computation to remote ggml RPC workers, sharding a model that does not fit on a single machine. It reuses the same backend-agnostic rpc-server workers as the llama.cpp backend, so one worker pool can serve both.

Manual: point the model at running workers with the rpc_servers option:

options:
- "rpc_servers:192.168.1.10:50052,192.168.1.11:50052"

Start a worker on each remote machine the same way you would for llama.cpp:

local-ai worker llama-cpp-rpc --llama-cpp-args="--host 0.0.0.0 --port 50052"

Automatic (peer-to-peer): when LocalAI runs in p2p worker mode, discovered workers are published in the LLAMACPP_GRPC_SERVERS environment variable. The image-generation backend reads that variable automatically (when rpc_servers is not set), so the same local-ai worker p2p-llama-cpp-rpc workers used for text generation also accelerate image generation - no per-model configuration needed.

By default the RPC devices join the pool and participate in placement; combine with the backend / params_backend options above to pin specific components to them (e.g. backend:diffusion=rpc0).

Diffusers

Diffusers is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. LocalAI has a diffusers backend which allows image generation using the diffusers library.

(Generated with AnimagineXL)

Image upscaling

The Diffusers backend can upscale an uploaded image through POST /v1/images/upscale. The multipart scale field is required and must be an integer from 1 through 16 at the HTTP boundary; actual scaling behavior depends on the selected pipeline. Install the included Stable Diffusion x4 upscaler gallery model first:

local-ai models install stable-diffusion-x4-upscaler

Then send the model name, scale, and image as multipart form fields:

curl http://localhost:8080/v1/images/upscale \
  -F model=stable-diffusion-x4-upscaler \
  -F scale=4 \
  -F image=@input.png

The response uses the same format as image generation and returns the generated image under /generated-images. The diffusers backend uses a loaded StableDiffusionUpscalePipeline or StableDiffusionLatentUpscalePipeline when configured. Other diffusers pipelines fall back to Lanczos resizing. Upscale routes select models with known_usecases: [upscale], not ordinary image generation models. If you installed the x4 upscaler before it declared this usecase, update its model YAML to known_usecases: [upscale] or reinstall the gallery entry. Custom Diffusers configurations intended for upscaling should also explicitly declare the upscale usecase.

Model setup

The models will be downloaded the first time you use the backend from huggingface automatically.

Create a model configuration file in the models directory, for instance to use Linaqruf/animagine-xl with CPU:

name: animagine-xl
parameters:
  model: Linaqruf/animagine-xl
backend: diffusers

f16: false
diffusers:
  cuda: false # Enable for GPU usage (CUDA)
  scheduler_type: euler_a

Dependencies

This is an extra backend - in the container is already available and there is nothing to do for the setup. Do not use core images (ending with -core). If you are building manually, see the build instructions.

Model setup

The models will be downloaded the first time you use the backend from huggingface automatically.

Create a model configuration file in the models directory, for instance to use Linaqruf/animagine-xl with CPU:

name: animagine-xl
parameters:
  model: Linaqruf/animagine-xl
backend: diffusers
cuda: true
f16: true
diffusers:
  scheduler_type: euler_a

Local models

You can also use local models, or modify some parameters like clip_skip, scheduler_type, for instance:

name: stablediffusion
parameters:
  model: toonyou_beta6.safetensors
backend: diffusers
step: 30
f16: true
cuda: true
diffusers:
  pipeline_type: StableDiffusionPipeline
  enable_parameters: "negative_prompt,num_inference_steps,clip_skip"
  scheduler_type: "k_dpmpp_sde"
  clip_skip: 11

cfg_scale: 8

For an offline single-file checkpoint that needs its original Diffusers configuration, keep both files in the mounted models directory:

name: offline-stable-diffusion
parameters:
  model: model.safetensors
backend: diffusers
diffusers:
  pipeline_type: StableDiffusionPipeline
  original_config_file: /models/v1-inference.yaml

Configuration parameters

The following parameters are available in the configuration file:

There are available several types of schedulers:

Pipelines types available:

Advanced: Additional parameters

Additional arbitrary parameters can be specified in the option field in key/value separated by ::

name: animagine-xl
options:
- "cfg_scale:6"

Note: There is no complete parameter list. Any parameter can be passed arbitrarily and is passed to the model directly as argument to the pipeline. Different pipelines/implementations support different parameters.

The example above, will result in the following python code when generating images:

pipe(
    prompt="A cute baby sea otter", # Options passed via API
    size="256x256", # Options passed via API
    cfg_scale=6 # Additional parameter passed via configuration file
)

Usage

Text to Image

Use the image generation endpoint with the model name from the configuration file:

curl http://localhost:8080/v1/images/generations \
    -H "Content-Type: application/json" \
    -d '{
      "prompt": "<positive prompt>|<negative prompt>", 
      "model": "animagine-xl", 
      "step": 51,
      "size": "1024x1024" 
    }'

Image to Image

https://huggingface.co/docs/diffusers/using-diffusers/img2img

An example model (GPU):

name: stablediffusion-edit
parameters:
  model: nitrosocke/Ghibli-Diffusion
backend: diffusers
step: 25
cuda: true
f16: true
diffusers:
  pipeline_type: StableDiffusionImg2ImgPipeline
  enable_parameters: "negative_prompt,num_inference_steps,image"

IMAGE_PATH=/path/to/your/image
(echo -n '{"file": "'; base64 $IMAGE_PATH; echo '", "prompt": "a sky background","size": "512x512","model":"stablediffusion-edit"}') |
curl -H "Content-Type: application/json" -d @-  http://localhost:8080/v1/images/generations

🖼️ Flux kontext with stable-diffusion.cpp

LocalAI supports Flux Kontext and can be used to edit images via the API:

Install with:

local-ai run flux.1-kontext-dev

To test:

curl http://localhost:8080/v1/images/generations -H "Content-Type: application/json" -d '{
  "model": "flux.1-kontext-dev",
  "prompt": "change 'flux.cpp' to 'LocalAI'",
  "size": "256x256",
  "ref_images": [
  	"https://raw.githubusercontent.com/leejet/stable-diffusion.cpp/master/assets/flux/flux1-dev-q8_0.png"
  ]
}'

Depth to Image

https://huggingface.co/docs/diffusers/using-diffusers/depth2img

name: stablediffusion-depth
parameters:
  model: stabilityai/stable-diffusion-2-depth
backend: diffusers
step: 50
f16: true
cuda: true
diffusers:
  pipeline_type: StableDiffusionDepth2ImgPipeline
  enable_parameters: "negative_prompt,num_inference_steps,image"

cfg_scale: 6

(echo -n '{"file": "'; base64 ~/path/to/image.jpeg; echo '", "prompt": "a sky background","size": "512x512","model":"stablediffusion-depth"}') |
curl -H "Content-Type: application/json" -d @-  http://localhost:8080/v1/images/generations

img2vid

name: img2vid
parameters:
  model: stabilityai/stable-video-diffusion-img2vid
backend: diffusers
step: 25
f16: true
cuda: true
diffusers:
  pipeline_type: StableVideoDiffusionPipeline

(echo -n '{"file": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/svd/rocket.png?download=true","size": "512x512","model":"img2vid"}') |
curl -H "Content-Type: application/json" -X POST -d @- http://localhost:8080/v1/images/generations

txt2vid

name: txt2vid
parameters:
  model: damo-vilab/text-to-video-ms-1.7b
backend: diffusers
step: 25
f16: true
cuda: true
diffusers:
  pipeline_type: VideoDiffusionPipeline
  cuda: true

(echo -n '{"prompt": "spiderman surfing","size": "512x512","model":"txt2vid"}') |
curl -H "Content-Type: application/json" -X POST -d @- http://localhost:8080/v1/images/generations