Every model on your PC, behind one address.
Bonsai runs your llama.cpp models for any app that speaks the OpenAI, Anthropic or OpenRouter API. It loads a model when an app asks for it, swaps models when another one is needed, and unloads them when they have been idle.
One program for Windows, with a Linux build. Bring your own llama-server.
How swapping works
Your graphics card holds one big model at a time, but you have several. Bonsai keeps track of which one is loaded and changes it when a request names a different model, so every app can use every model through the same address.
Try it: send requests to different models and watch the card’s memory. This runs in your browser and only shows the order of events.
- chat-27beveryday chatloaded
- coder-14bcoding agentnot loaded
- vision-9breads imagesnot loaded
- embed-smalldocument searchnot loaded
- chat-27b is loaded and waiting.
What it does
- Swaps models for you
- Ask for any model by name. Bonsai finishes the replies already running, unloads what is in the way and loads the one you asked for, while the app sees loading progress. Models that sit idle unload on their own.
- Tells you what fits
- Before you load anything, Bonsai reads the model file and works out how much graphics memory the weights, the context and the working buffers need, using the same rules as llama.cpp. If it will not fit, it suggests the settings that make it fit.
- Finds and downloads models
- Search Hugging Face from the manager, see every quant with its size and whether it fits your card, and download it. Downloads resume if they stop.
- Chat, documents and tools
- A chat window for every model, with thinking shown separately. Drop files into a collection and ask about them with numbered sources. Connect MCP tool servers and approve each call.
- Fixes settings on the way in
- Apps send sampling settings in different shapes. Bonsai renames camelCase keys and unwraps nested options so llama-server actually uses them, and you can force or strip settings per model.
- Shows every request
- Each request is listed with tokens in and out, speed, time to first token and load wait. Open one to see what was sent and what came back. Live logs for every model, and Prometheus metrics.
Works with what you have
Bonsai is a gateway, not a new model runtime. It starts the llama-server builds you already use, with the arguments you choose, and puts one front door on them.
- OpenAI APIChat, completions, responses, embeddings and rerank at
/v1, streamed. - Anthropic APIMessages with streaming, tools, images and thinking, so Claude Code and Anthropic SDKs work with local models.
- OpenRouter APIPoint any OpenRouter app at
/api/v1. Fallback model lists, the reasoning object and usage reports are understood. - llama-swap configsGroups, macros, filters, aliases, peers and hooks read as they are.
- Any llama.cpp buildCUDA, ROCm/HIP, Vulkan or CPU. Vision projectors and draft models too.
- Your networkOther devices use the API directly. The manager can be shared behind a password.
Get started
Get bonsai.exe
Download it from GitHub and put it in a folder of its own. It is one file with nothing to install.
Start it
Run it. The manager opens at
http://127.0.0.1:8010and the API listens on port 8000.Add your models
Point Bonsai at the folders that hold your GGUF files, or find a model in the Catalogue. Already using llama-swap? Bonsai reads your config.yaml as it is.
Connect your apps
Use
http://your-pc:8000/v1for OpenAI apps,http://your-pc:8000for Anthropic apps, orhttp://your-pc:8000/api/v1for OpenRouter apps. The Apps page writes a ready config for OpenCode.
curl http://your-pc:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "chat-27b", "messages": [{"role": "user", "content": "Hello"}]}'