AI DEPLOYMENT GUIDE
General user guide for deploying and using the AI Deployment Tool
STRATISYSTEM AI DEPLOYMENT TOOL USER GUIDE
SteelDome Technologies · StratiSYSTEM OS
Deploy and operate self-hosted AI inference using deployai. This guide covers planning, installation, local inference, GPU acceleration, distributed CPU execution, the shared Open WebUI, knowledge ingestion, application integration, backup, recovery, and ongoing operations.
Audience: IT operators, SteelDome partners, and solution architects. Edition: 12 November 2026.
OPERATOR CONVENTIONS
Example convention Commands run in a root Bash shell. The executable is deployai. /ai-models is the general storage example; /stratistor/clustermounts/machines is the three-node CephFS example. Substitute your hostnames, private RPC IPs, certificates, and Knowledge Base IDs.
Keep a deployment record Retain the complete successful serve command. A later serve call reconciles the options you supply and their defaults; update is the workflow that reloads the saved deployment configuration.
AI CONCEPTS
WHAT THIS TOOL PROVIDES
deployai assembles local AI inference services from several established components. It stages trained GGUF models, builds the inference backend, configures systemd services, and connects multiple local model endpoints to one Open WebUI by default. Optional document retrieval, memory, web search, and an authenticated application API extend the service.
INFERENCE USES A MODEL THAT ALREADY EXISTS
A language model has learned patterns during training. Inference applies those learned weights to a prompt and generates an answer one token at a time. deployai prepares inference; it does not train or fine tune a model.
A QUESTION BECOMES A RESPONSE
- A user enters instructions and a question in Open WebUI, or an application submits a supported API request.
- The interface forms a prompt from the question, conversation history, and any selected document passages or web results.
- llama.cpp loads the GGUF weights and computes the response using CPU resources or a supported GPU backend.
- The interface presents the generated text. The user checks significant claims against the cited material or another authoritative source.
CHOOSE THE INFORMATION SOURCE FOR THE TASK
| Need | Approach | What it changes |
|---|---|---|
| General writing or explanation | Model chat | Uses the trained model and current conversation. |
| Answers from internal documents | RAG with a selected KB | Adds retrieved source text to the prompt. |
| External web information | Web search through SearXNG | Adds external search results and fetched content where supported. |
| User preferences across chats | Persistent memory | Makes saved user information available to later conversations. |
Quality expectation A fluent answer can still be wrong. Retrieval improves access to relevant evidence, but model choice, document quality, retrieval settings, and the question all affect the result.
AI TERMS IN PRACTICAL LANGUAGE
| Term | Meaning for the operator |
|---|---|
| LLM and parameters | A large language model generates text. Its parameters are learned numeric weights. Parameter count is a model characteristic, not a user limit. |
| Token and prompt | A token is a unit of text, often part of a word. A prompt contains the instructions and information presented to the model. |
| Context window and KV cache | Context is the token budget available for input and generation. The KV cache stores intermediate attention data; larger contexts consume additional memory. |
| Checkpoint and GGUF | A checkpoint contains model weights. GGUF is the model file format used by the main llama.cpp workflow. |
| Quantization and Q4_K_M | Quantization reduces weight precision to reduce storage and memory use. Q4_K_M is the quantization label used by the bundled GGUF profiles; quality and speed still require evaluation. |
| Dense model and MoE | A dense model uses its full parameter network for each token. A mixture of experts model activates selected experts. Fewer active parameters do not mean only those weights need storage. |
| CPU GPU and VRAM | The CPU uses system RAM. A GPU uses its device memory, or VRAM. Offload moves supported inference layers to a GPU; context and workspace also need memory. |
| CUDA HIP and ROCm | CUDA is the NVIDIA compute backend. HIP is the AMD backend built against the ROCm software environment. |
| Embedding and vector index | An embedding is a numeric representation of content. A vector index finds related passages for retrieval. |
| RAG and Knowledge Base | Retrieval augmented generation combines retrieved text with the prompt. A Knowledge Base is an indexed document collection in Open WebUI. |
| Tokens per second | A throughput measure. Prompt processing speed and output generation speed are different measurements. |
| API TLS OIDC and LDAP | An API lets applications call a service. TLS encrypts the connection. OIDC and LDAP integrate browser sign in with an external identity service. |
| RPC and coordinator | RPC sends inference work to a process on another host. The coordinator runs the front end inference server and assembles the response. |
ARCHITECTURE
HOW REQUESTS MOVE THROUGH THE STACK
Browser workflow
| Browser user HTTPS chat | → | Open WebUI Accounts, Knowledge Bases, memory and tools | → | Local data Chats, uploaded documents and memory |
Private model inference
| Open WebUI Connects to the selected model | → | llama.cpp CPU, CUDA or HIP inference | → | GGUF checkpoint Model weights on disk |
Application workflow
| Application HTTPS request with API key | → | Optional Go gateway TLS, authorization and rate limiting | → | llama.cpp Direct model inference |
External search when enabled
| Open WebUI Requests web search | → | SearXNG Queries external search engines | → | Search results Returned to the WebUI workflow |
Figure 1 The browser and application paths remain distinct. This single-model view shows one selection; in shared mode Open WebUI can route chats to several independently served llama.cpp model endpoints. The gateway forwards directly to its selected inference service.
BROWSER AND APPLICATION BEHAVIOR DIFFER
Open WebUI supplies the user experience, accounts, selected-model routing, document retrieval, and memory features. Shared mode registers multiple local OpenAI-compatible inference APIs into one model selector. The optional Go gateway adds TLS, a bearer API key, and rate limiting in front of selected llama.cpp endpoints; it does not automatically run WebUI retrieval, memory, or search tools.
Web traffic SearXNG queries external search services when web search is used. Model inference remains local, but a web search workflow can send query terms outside the host. Turn web search off where that behavior is unsuitable.
WHAT RUNS ON THE HOST
Administration and service management
| Administrator CLI and health checks | → | deployai Deploy, serve and maintain | → | systemd Starts, supervises and schedules services |
Managed workloads and operational data
| Managed workload | Run location | Associated data or action |
|---|---|---|
| llama.cpp and optional Go gateway | Host processes | GGUF checkpoints in model storage |
| Open WebUI and SearXNG | Selected Docker or Podman runtime | WebUI database, uploads and application state |
| RAG synchronization | systemd one-shot service and timer | Operator-scheduled knowledge ingestion |
| Backup operation | Explicit operator command | Backup files; not automatically scheduled |
Figure 2 deployai configures host processes, Docker or Podman containers, and systemd jobs. In default shared mode one Open WebUI container and database serve multiple active local model processes. Backups are an explicit operator action.
| Component | Runs as | Primary responsibility |
|---|---|---|
| llama.cpp | Host systemd service | Loads the selected GGUF and serves inference. |
| Open WebUI and SearXNG | Containers supervised by systemd | User access and web search. |
| Gateway and scheduled sync | Go executable and systemd units | Optional application access and knowledge ingestion. |
Topology boundary The tool does not install Kubernetes, a workload scheduler, a load balancer, or an HA browser application. Independent hosts and distributed CPU RPC solve different needs; neither adds automatic failover.
PLANNING
PREPARE THE HOST AND NETWORK
Choose the host role and storage before downloading a model. For a predictable first run, use one Linux x86 64 host, one model, a verified storage path, and an explicit CPU or GPU selection.
| Area | Required preparation |
|---|---|
| Administration | Use a root shell for installation and service work. The host needs systemd, working DNS, and the package tools used by the selected Enterprise Linux workflow. |
| Storage root | Create an existing real absolute directory. A symlink is rejected. Single host storage may be local; multi host execution requires a genuinely shared filesystem at the same path. |
| Memory and CPU | Allow for weights, context cache, inference workspace, the OS, Open WebUI, indexing, and other host workloads. Checkpoint size is not the RAM or VRAM requirement. |
| Downloads | Initial installation may reach GitHub, Hugging Face, Python packages, the chosen OCI registries, and selected GPU or OS repositories. Embedding models may download on first use. |
| Containers | Docker and Podman can coexist. Choose the runtime that will run the AI containers. Installation never deliberately removes either runtime or Ceph-managed containers. --container-install selected installs the chosen runtime if needed; off uses an existing installation; both requests both families and may fail on package conflicts. |
| GPU ownership | Confirm the GPU belongs to the inference environment. VFIO passthrough, vGPU host managers, and active AMD SR IOV virtual functions can block host driver changes. |
| Installation root | GPU preparation resolves /stratistor, /stratiserv, or /baseos. Set --install-root when several roots exist or the correct root is ambiguous. |
DEFAULT LISTENERS FOR THE WORKED QWEN PROFILE
| Service | Default address | Who should reach it |
|---|---|---|
| Open WebUI | HTTPS 0.0.0.0:18103 | Approved browser users on a restricted network. |
| llama.cpp and metrics | HTTP 127.0.0.1:8103 | Local components and protected monitoring access. |
| SearXNG | HTTP 127.0.0.1:18888 | Local search integration. |
| Optional gateway | HTTPS 127.0.0.1:18104 | Local applications until explicitly published. |
Firewall Plan and apply network access separately. The tool’s bind settings do not constitute a firewall policy. Keep the private inference API and distributed RPC traffic restricted to their intended peers.
CHOOSE A MODEL AND SIZE RESOURCES
Use list-models for the packaged profiles. Start with a smaller model to establish deployment and response quality, then compare larger candidates with representative questions and an identical benchmark procedure.
| Model ID | Approximate file | API port | UI port |
|---|---|---|---|
| qwen3-8b | 5.0 GB | 8101 | 18101 |
| qwen3-14b | 9.0 GB | 8102 | 18102 |
| qwen3-30b-a3b | 18.6 GB | 8103 | 18103 |
| qwen3-32b | 19.8 GB | 8104 | 18104 |
| llama3.1-8b | 4.9 GB | 8105 | 18105 |
| gemma3-12b | 7.3 GB | 8106 | 18106 |
| deepseek-r1-14b | 9.0 GB | 8107 | 18107 |
| custom-gguf | Depends on file | 8199 | 18199 |
Sizes are approximate profile labels. In default shared mode, an additional model keeps its own inference port but reuses the existing primary WebUI port and database; the listed UI ports are only per-model defaults or the first-model shared owner. Kimi K3 remains a separate approximately 1.56 TB CPU engine without this HTTP/WebUI workflow.
deployai list-models
deployai gpu-status
deployai plan --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --resource-profile auto
WHAT AUTOMATIC SIZING DOES
For deploy, serve, and preflight, auto sizing normally chooses physical core threads; shared hosts receive roughly two thirds of physical cores. It chooses context 2048 below 24 GiB RAM, 4096 from 24 GiB, and 8192 from 64 GiB. Explicit --threads and --ctx-size take precedence. These are heuristics rather than a model fit guarantee.
Shared host protection Auto isolation detects active Ceph and virtualization services and applies systemd controls plus container limits. Review the actual limits and measure storage or VM latency under load. A dedicated profile does not reserve the host for AI.
Port planning The optional gateway default port 18104 can collide with qwen3-32b per-model WebUI port. Each active model needs a distinct inference API port. Only the selected primary WebUI port is exposed in shared mode. A custom-gguf service has one fixed profile identity and port, so use one custom GGUF at a time unless extended.
QUICK START
INSTALL AND STAGE THE FIRST MODEL
This walkthrough uses CPU inference with Qwen3 30B A3B and Docker. Specify model and node count because raw defaults are kimi-k3 and three hosts. Use a smaller model if the host has less capacity. The first served model owns the shared WebUI by default; later models reuse it.
1 BUILD OR INSTALL THE EXECUTABLE
The attached source package contains go.mod and the Go source. Build it with Go 1.26.8 or a compatible newer toolchain. Build on a suitable build host if the serving host does not have Go. The source archive has no Makefile or prebuilt bin directory.
# From the extracted source directory
go build -trimpath -o ./deployai .
install -m 0755 ./deployai /usr/local/bin/deployai
deployai --help
2 CREATE AND INSPECT THE MODEL STORAGE PATH
install -d -m 0750 /ai-models
df -h /ai-models
findmnt -T /ai-models
Use an existing mounted data filesystem where appropriate. Creating a directory does not mount or provision storage. The checkpoint downloader checks free capacity against the resolved file size plus an 8 GiB margin; backups and future revisions need additional space.
3 CHECK READINESS AND DEPLOY
deployai preflight --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu none --gpu-install off
deployai deploy --nodes 1 --model qwen3-30b-a3b
--storage-root /ai-models --gpu none --gpu-install off
Expected result deploy prepares llama.cpp binaries, downloads a GGUF from a pinned Hugging Face revision, and validates its integrity by default. With --model-verify off it checks the pinned revision and expected size but skips the full checksum. The CLI smoke test runs unless disabled. deploy stages files; serve starts the API and browser services.
START SERVICES AND SIGN IN
4 SERVE THE MODEL AND BROWSER INTERFACE
deployai serve --nodes 1 --model qwen3-30b-a3b \
--mode independent --storage-root /ai-models \
--gpu none --gpu-install off \
--container-runtime docker --container-install selected \
--listen-address 127.0.0.1 --webui-bind 0.0.0.0 \
--webui-tls on --web-ui on --web-search on \
--rag on --memory on --metrics on
Use --container-runtime podman to select Podman; on a system where the selected runtime is already installed, use --container-install off. Docker and Podman may coexist safely. serve requires the checkpoint and matching backend binaries from deploy and then reconciles inference, search, the shared Open WebUI, and requested optional services.
5 VERIFY SERVICE HEALTH
deployai status --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models
deployai doctor --full
curl -fsS http://127.0.0.1:8103/health
curl --cacert \
/etc/stratistor-ai/open-webui/qwen3-30b-a3b/tls/server.crt \
https://127.0.0.1:18103/health
With one recorded deployment, doctor loads its saved configuration. status should be given an explicit model and node count. An inactive distributed or RPC unit is normal for an independent deployment; interpret status entries against the selected topology.
6 OPEN THE BROWSER INTERFACE
Browse to https://SERVER-IP:18103 for a first Qwen3 30B deployment, or to the pinned primary WebUI port for a multi-model installation. Trust the initial self-signed certificate through your approved procedure. Read the root-protected primary instance credential file; subsequent models reuse that account and do not create a second login.
cat /etc/stratistor-ai/open-webui/qwen3-30b-a3b/admin-credentials
Initial account The generated administrator email is admin@stratistor.local with a random password. Keep these details private, change the initial password through the application, and create named accounts for routine access. Public signup is disabled; new account permissions require administrator attention.
MODEL INTEGRITY
CHOOSE A CHECKPOINT VERIFICATION POLICY
GGUF verification confirms that a downloaded checkpoint matches the expected Hugging Face content. Full checksum verification is the safer default; a size-only mode is available for controlled labs where fast staging is more important than cryptographic integrity.
| Option | Result and practical use |
|---|---|
| --model-verify on | Default: use Hugging Face metadata and SHA-256 (when available), otherwise hf cache verify against the pinned revision. |
| --model-verify off | Obtain the pinned revision and expected size; check local size; skip full-file SHA-256 and hf cache verify. |
| --reverify-model | Force checking a prior checkpoint rather than accepting an existing marker; retain on for cryptographic verification. |
| --verify-jobs N | Kimi compatibility verification concurrency; not a switch to disable native GGUF hashing. |
PRACTICAL OPERATOR COMMANDS
# Fast size-only GGUF staging (not cryptographic verification)
./deployai deploy --nodes 1 --model qwen3-8b \
--storage-root /ai-models --gpu none --gpu-install off \
--model-verify off
Return to full integrity checking for a subsequent run
./deployai deploy --nodes 1 --model qwen3-8b
--storage-root /ai-models --gpu none --gpu-install off
--model-verify on --reverify-model
Receipt behavior Full checking records .gguf-verified; size-only checking records .gguf-size-validated. A matching prior full-verification receipt can be reused. Off does not mean offline: metadata is still requested from Hugging Face and missing/incorrect-sized files must be downloaded.
Performance and risk Full SHA-256 needs to read the entire GGUF, which can be costly on CephFS. Size-only avoids that read, but file length alone cannot detect silent corruption or prove authenticity. Keep on for first production validation and after unexpected storage or model issues.
EVERYDAY USE
USE AND VALIDATE THE BROWSER FEATURES
A healthy service is the starting point. Test the actual user workflow before handing it to a team. UI labels and permission options can vary with the Open WebUI image you deploy.
CHAT AND CONVERSATION CONTEXT
Select the served model and ask a simple question. Follow with a related question in the same chat, then start a new chat and compare behavior. Conversation history consumes context; long chats may leave less room for retrieved documents and generated output.
DOCUMENT QUESTIONS
Create or open a Knowledge Base in Workspace or the equivalent Knowledge area. Upload a short readable document, wait for processing, and attach or select the KB in a chat. Ask a question whose answer appears only in that document. Inspect the cited passage and verify that an unauthorized account cannot access a restricted collection.
WEB RESEARCH
Enable or select the web search tool for the conversation where available. Ask for a source backed response and inspect the resulting links. An enabled server setting allows the feature; it does not prove that every model or every prompt will choose a web tool. Use --web-search off for workflows without external search.
PERSISTENT MEMORY
With memory enabled and the user permitted to use it, save a harmless preference such as a preferred response format. Review the saved memory in the application, start a new chat, and check recall. Delete the fixture afterward. Background memory review runs according to the configured turn cadence; enabled settings alone do not prove useful recall.
| Acceptance test | Pass condition |
|---|---|
| Basic inference | A real request produces a coherent response without service errors. |
| Knowledge retrieval | A unique document fact is returned with the correct source. |
| Memory | A saved preference is available in a second authorized conversation. |
| Permissions | Named users see only the collections and features they should. |
| Web search | A requested search returns usable external results and sources. |
COMMON USE CASES
CPU AND SHARED HOST RECIPES
A SMALL SINGLE HOST PILOT
Use a smaller profile to verify installation and usability before committing a large host. Both commands must use the same model and backend.
deployai deploy --nodes 1 --model qwen3-8b \
--storage-root /ai-models --gpu none --gpu-install off
deployai serve --nodes 1 --model qwen3-8b \
--storage-root /ai-models --gpu none --gpu-install off \
--container-runtime docker --container-install selected \
--web-ui on --webui-tls on --web-search off
Verify For a fresh first deployment, open https://SERVER-IP:18101. If a shared WebUI already exists, open its retained primary URL instead and select qwen3-8b from the model list. A second model adds an inference service, not another browser database.
A PODMAN HOST SHARED WITH STORAGE OR VIRTUALIZATION
Keep existing Docker or Podman installations intact and choose the one intended for AI containers. Use --container-install off when your selected runtime is already present. Force the shared resource profile if service detection does not capture the host role; the following serve recipe assumes the checkpoint and binaries are staged.
deployai serve --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu none --gpu-install off \
--container-runtime podman --container-install off \
--resource-profile shared --resource-isolation on \
--threads 24 --ctx-size 8192 \
--listen-address 127.0.0.1 --webui-bind 0.0.0.0 \
--webui-tls on --web-ui on --web-search on \
--rag on --memory on
Verify Confirm the proposed context fits the host, inspect service resource properties, and compare storage or VM latency before and during inference. The 24 thread and 8192 context values are examples, not universal sizing targets.
A SERVICE FOR LOCAL APPLICATIONS ONLY
For a fresh inference service without Open WebUI, use --web-ui off --web-search off --rag off --memory off --api-gateway off on serve. This leaves the local llama.cpp API. For the tool’s authenticated TLS gateway, keep the TLS enabled WebUI configuration required by the implementation.
NVIDIA AND AMD ACCELERATION RECIPES
GPU acceleration applies to independent GGUF deployments. It requires compatible hardware and a working compute stack. The same --gpu value must be used when building, staging, and serving the model.
NVIDIA WITH A PREPARED CUDA ENVIRONMENT
deployai preflight --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu nvidia --gpu-install off
deployai deploy --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu nvidia --gpu-install off
deployai serve --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu nvidia --gpu-install off \
--container-runtime podman --container-install off \
--gpu-layers -1 --webui-tls on
AMD WITH A PREPARED ROCM AND HIP ENVIRONMENT
deployai preflight --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu amd --gpu-install off
deployai deploy --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu amd --gpu-install off
deployai serve --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu amd --gpu-install off \
--container-runtime podman --container-install off \
--gpu-layers -1 --webui-tls on
Use your installed runtime; these examples assume Podman is already present. Auto offload requests the backend’s maximum layer offload, represented as 999. It does not calculate a guaranteed VRAM fit. Reduce --gpu-layers or context when required and inspect the inference journal to confirm actual offload.
When prerequisites are missing deploy with --gpu-install on may install permitted prerequisites. A newly installed kernel driver stops the run and requires a reboot and verification before rerun. On a custom kernel, missing AMDGPU is refused unless the supported kernel path is satisfied. Resolve passthrough, vGPU host, MOK, and signing conflicts through the platform owner; do not bypass them.
STAGE A CUSTOM GGUF MODEL
Choose custom-gguf when a required model is not in the packaged list. You need a Hugging Face repository and one compatible GGUF file at its root. The CLI validates a single basename, so nested paths and a set of split GGUF shards are outside this workflow.
CHECK THE SOURCE BEFORE DEPLOYING
- Confirm that the selected file is intended for text inference with your llama.cpp version and that its model license and access requirements fit your use.
- Record the repository, exact filename, quantization, expected size, and access credentials if the repository requires them. A custom model’s extra modalities are not automatically deployed by this tool.
- Allow RAM or VRAM beyond the checkpoint size for context and workspace. Test a small context and a real request before increasing it.
DEPLOY AND SERVE THE SELECTED FILE
Replace ORG/REPO and Model-Q4_K_M.gguf below with real values. Include the same custom source options on both commands and on explicit operational calls that require model lookup.
deployai deploy --nodes 1 --model custom-gguf \
--hf-repo ORG/REPO --hf-file Model-Q4_K_M.gguf \
--storage-root /ai-models --gpu none --gpu-install off
deployai serve --nodes 1 --model custom-gguf \
--hf-repo ORG/REPO --hf-file Model-Q4_K_M.gguf \
--storage-root /ai-models --gpu none --gpu-install off \
--container-runtime podman --container-install off \
--webui-tls on --web-search off
Expected result The custom profile uses private API port 8199. On a fresh system its per-model default browser port is 18199; in shared mode it registers with the existing primary WebUI instead. Its checkpoint directory includes a repository/file hash. Verify the served model with a real inference request.
RUN SEVERAL BUILT IN PROFILES DELIBERATELY
Built-in profiles have distinct inference ports and service identities. In shared mode, active locally served profiles appear in one primary WebUI model selector and share its chats, credentials, and RAG state. Each model still consumes its own CPU, RAM or VRAM. The custom-gguf profile shares one service name and port 8199 across sources; different hashed checkpoint directories do not create simultaneous custom endpoints.
CONTAINER OPERATIONS
RUN DOCKER AND PODMAN SAFELY TOGETHER
Docker Engine and Podman can be installed side by side on a StratiSYSTEM host. Ceph may depend on Podman, so deployai never intentionally removes Podman, Docker, runc, or containerd to make the selected runtime install succeed. Containers already managed by Ceph are outside the AI deployment scope.
| Flag or condition | Expected behavior |
|---|---|
| --container-runtime docker | Use Docker for managed Open WebUI and SearXNG containers. |
| --container-runtime podman | Use Podman for managed AI containers. |
| --container-install off | Do not install a runtime; require the selected one to be present and functional. |
| --container-install selected | Attempt to install only the chosen runtime, preserving existing runtime packages. |
| --container-install both | Attempt installation of both families, without removing either one. |
| DNF conflict / podman-docker alias | Fail safely with a useful error rather than using forced package erasure; resolve real package conflicts separately. |
RECOMMENDED COMMANDS ON AN ESTABLISHED CEPH HOST
# Docker already installed: select it, skip package changes
--container-runtime docker --container-install off
Docker not installed: request Docker installation only
--container-runtime docker --container-install selected
Confirm both engines and protect Ceph services
docker info >/dev/null && podman info >/dev/null
systemctl status ceph.target --no-pager
Caution A DNF dependency conflict is a legitimate failure, not permission to remove Podman or Ceph components. Do not use --allowerasing as a shortcut. If podman-docker supplies the docker CLI alias, installation may require explicit administrator remediation. AI runtime selection does not change Ceph deployment.
UNIFIED WEBUI
ONE INTERFACE WITH SELECTABLE MODELS
Shared mode is now the default. Each GGUF model runs in its own llama-server systemd service with a distinct private API port. A single Open WebUI container connects to the active local inference APIs and presents those models in one dropdown. Chats, login accounts, RAG Knowledge Bases, memory, TLS, and settings belong to the primary WebUI.
One browser interface
| Browser users One HTTPS sign-in | → | Primary Open WebUI One model selector, chats, RAG and memory |
Active local inference connections
| Selectable model | Local endpoint | Operation |
|---|---|---|
| Llama 3.1 8B | HTTP API port 8105 | Separate llama.cpp service |
| Qwen3 0.6B (custom GGUF) | HTTP API port 8199 | Separate llama.cpp service |
| Other supported GGUF model | Its own available local API port | Separate model service identity |
All models share the primary WebUI database and accounts; their inference services and memory consumption remain separate.
Figure 5 One Open WebUI selects among separate local inference services. Model weights may reside on shared CephFS; the browser database remains on the coordinator and is not automatically clustered.
| Configuration | Meaning |
|---|---|
| --webui-mode shared | Default: reuse or create one primary WebUI and manage its local OpenAI-compatible connections. |
| --webui-mode per-model | Legacy behavior: create a separate WebUI for each model. |
| --webui-primary-model llama3.1-8b | On first shared-mode serve, pin an already deployed Llama WebUI as primary. |
| shared-owner marker | /etc/stratistor-ai/open-webui/shared-owner records the selected primary identity. |
Migration An existing active WebUI is preferred. Primary credentials, TLS material, and database are retained; secondary legacy WebUI databases are not merged, deleted, or automatically stopped. After the primary owner is pinned, switching it is refused rather than risking a silent data replacement.
ADD A SMALL SECOND MODEL TO YOUR EXISTING UI
Example environment: Llama 3.1 8B is already served across stratistor1-1 through stratistor1-3, and its shared WebUI is https://stratistor1-1:18105. For a lightweight test, stage one custom Qwen3 0.6B Q4_K_M checkpoint on the coordinator only. Model selection is independent of whether Llama uses distributed RPC.
1 STAGE THE SECOND MODEL (DO NOT REDEPLOY LLAMA)
./deployai deploy \
--nodes 1 --model custom-gguf \
--hf-repo unsloth/Qwen3-0.6B-GGUF \
--hf-file Qwen3-0.6B-Q4_K_M.gguf \
--mode independent \
--storage-root /stratistor/clustermounts/machines \
--install-root /stratistor --gpu none --gpu-install off \
--threads 4 --ctx-size 4096 --build-jobs 4 \
--resource-profile shared --resource-isolation auto
2 SERVE IT AND REGISTER WITH THE PINNED WEBUI
./deployai serve \
--nodes 1 --model custom-gguf \
--hf-repo unsloth/Qwen3-0.6B-GGUF \
--hf-file Qwen3-0.6B-Q4_K_M.gguf \
--mode independent \
--storage-root /stratistor/clustermounts/machines \
--install-root /stratistor --gpu none --gpu-install off \
--threads 4 --ctx-size 4096 \
--resource-profile shared --resource-isolation auto \
--container-runtime docker --container-install off \
--web-ui on --webui-mode shared \
--webui-primary-model llama3.1-8b \
--web-search on --rag on --memory on --webui-tls on
3 Open https://stratistor1-1:18105 and select the second model. The existing Llama inference API remains on 8105; the custom model serves on 8199. Model labels may be prefixed by their deployai service identity. The primary WebUI may restart briefly when its connection list changes.
VERIFY AND OPERATE SHARED-MODEL SELECTION
CHECK THE OWNER, SERVICES, AND REGISTERED ENDPOINTS
cat /etc/stratistor-ai/open-webui/shared-owner
systemctl is-active stratistor-open-webui-llama3.1-8b.service
systemctl is-active stratistor-llama-custom-gguf.service
# For distributed Llama, the coordinator service is stratistor-dist-llama3.1-8b
ss -lntp | grep -E ":(8105|8199|18105)\b"
grep ^OPENAI_API_BASE_URLS= \
/etc/stratistor-ai/open-webui/llama3.1-8b/environment
The expected connection list contains the active local model APIs (for example http://127.0.0.1:8105/v1 and http://127.0.0.1:8199/v1). Only active local systemd inference services are discovered; it does not automatically combine independent models running exclusively on other hosts.
SAFE LIFECYCLE PRACTICES
| Operation | What to expect |
|---|---|
| Serve a new model | Registers the new active local API; primary WebUI settings are reconciled and the UI may restart. |
| Stop one secondary model | Stops that model; shared WebUI remains available for other models. |
| Backup/restore | Run under the primary model identity to capture shared UI state consistently. |
| RAG or admin credentials | Use the primary WebUI endpoint and existing root-protected credentials. |
| Legacy per-model UI | Old secondary containers and data remain intact until an intentional migration/retirement. |
Security The managed connection list is written into the primary environment file (including OPENAI_API_BASE_URLS). Back up any hand-added OpenAI connections before enabling shared mode, because deployai manages those values. API endpoints should remain loopback-only or reachable only through appropriate private controls.
SPECIALIZED DEPLOYMENT
USE THE SPECIALIZED KIMI ENGINE
The kimi-k3 profile follows a separate CPU C engine compatibility workflow with an approximately 1.56 TB checkpoint. It is intended for a carefully planned high capacity environment. It does not supply the normal llama.cpp HTTP server, Open WebUI, RAG, or gateway workflow.
PLAN STORAGE AND MEMORY BEFORE STAGING
Review the compatibility engine requirements and the memory preset that fits the host. The preset changes the engine’s working memory behavior; it does not make the complete checkpoint download disappear. Leave capacity for verification, staging, and the other host workloads. Use the standard GGUF workflow when a browser chat service is the goal.
deployai preflight --nodes 1 --model kimi-k3 \
--storage-root /ai-models --gpu none
deployai deploy --nodes 1 --model kimi-k3 \
--storage-root /ai-models --gpu none --preset auto
deployai test --nodes 1 --model kimi-k3 \
--storage-root /ai-models --gpu none --preset auto
deployai status --nodes 1 --model kimi-k3 \
--storage-root /ai-models --gpu none
The compatibility worker preserves the specialized download and integrity pipeline. The Go preflight checks host and storage before handing the action to that worker; it is not a substitute for checking the full engine capacity requirements.
| Setting or command | Meaning for Kimi |
|---|---|
| --preset | auto, server, max, workstation, desktop, laptop, or ultra. The compatibility engine evaluates the selected memory policy. |
| --verify-jobs and --download-workers | Passed to the compatibility worker. These controls do not tune the native single GGUF download loop. |
| serve and stop | Rejected for this profile by the Go parser. Use the engine specific operational workflow. |
| Distributed mode or GPU flags | Distributed mode and explicit NVIDIA or AMD inference are rejected. Kimi uses independent CPU execution. |
| Generic update and backups | Do not treat the GGUF updater or operator state archive as a full Kimi checkpoint lifecycle. Retain its model material and recovery procedure separately. |
MULTIPLE HOSTS
CHOOSE THE RIGHT MULTI HOST TOPOLOGY
| Topology | Behavior | Choose it when |
|---|---|---|
| Independent single host | One model server uses one host. UI and optional integrations are local. | You need the simplest pilot or a dedicated team endpoint. |
| Independent two or three hosts | Each host runs a separate model service. The tool fans out operations over SSH and verifies shared model storage. | You need separate endpoints on several hosts and can manage their access individually. |
| Distributed CPU RPC | One coordinator splits inference work among local and remote CPU RPC workers. | You are evaluating a model across combined host resources and can accept the added network and operational requirements. |
PREPARE MULTI HOST ACCESS
- Use one to three unique, resolvable hostnames.
--nodes2 and--nodes3 expand to stratistor1-1, stratistor1-2, and stratistor1-3; use an explicit comma separated list for other hosts. - Provide noninteractive root SSH and SCP access from the initiating host. Verify host keys and connectivity before running an operation.
- Mount the same real shared filesystem at the same absolute storage root on every selected host. The tool writes a sentinel and verifies that remote hosts can read the same bytes.
- Keep all hosts on a compatible llama.cpp revision. The distributed workflow is CPU only and requires explicit
--gpunone plus private IPv4 worker addresses.
INDEPENDENT HOST EXAMPLE
deployai deploy --nodes ai01,ai02 --mode independent \
--model qwen3-8b --storage-root /shared/models \
--gpu none --gpu-install off
deployai serve --nodes ai01,ai02 --mode independent \
--model qwen3-8b --storage-root /shared/models \
--gpu none --gpu-install off \
--container-runtime podman --container-install off
Advanced settings For custom TLS, OIDC, or LDAP on independent remote nodes, configure and validate each host locally. The multi host SSH path does not forward every optional security setting. Independent endpoints have separate local UI state; they do not become an HA application.
DEPLOY DISTRIBUTED CPU INFERENCE
Distributed request path
| Browser or API client Connects to coordinator | → | Coordinator Open WebUI and llama-server | → | CPU RPC workers Collaborative inference across nodes |
CPU RPC and model storage placement
| Component | Location | Network or storage requirement |
|---|---|---|
| Local CPU RPC | Coordinator host | Loopback address 127.0.0.1 |
| CPU RPC worker 2 | Second host | Restricted private IPv4 connection |
| CPU RPC worker 3 (optional) | Third host | Restricted private IPv4 connection |
| Shared GGUF storage | Mounted on every participating host | Same absolute mounted storage path |
Figure 3 A distributed deployment has one coordinator and two or three CPU RPC processes in total. The shared storage path holds the model; it does not turn the WebUI into a clustered database.
Run from the first listed host, ai01. Here ai02 uses 10.10.25.12 and ai03 uses 10.10.25.13. The default RPC port for qwen3-30b-a3b is 48103, calculated as 40000 plus the API port.
deployai deploy --nodes ai01,ai02,ai03 --mode distributed \
--model qwen3-30b-a3b --storage-root /shared/models \
--gpu none --rpc-address-2 10.10.25.12 \
--rpc-address-3 10.10.25.13
deployai serve --nodes ai01,ai02,ai03 --mode distributed \
--model qwen3-30b-a3b --storage-root /shared/models \
--gpu none --rpc-address-2 10.10.25.12 \
--rpc-address-3 10.10.25.13 \
--container-runtime podman --container-install off
Operate Validate a real coordinator API request. The standalone distributed test command is not implemented; generic update keeps the distributed backend binaries. Stop and inspect services locally on each host. Multi host distributed status and stop are not complete cluster operations in this release. Benchmark before expecting a speed benefit.
RAG WORKFLOWS
UNDERSTAND KNOWLEDGE INGESTION AND RETRIEVAL
Document ingestion — incremental adds/updates; mirror may delete source-absent documents
| Source folder Local regular files | → | RAG sync client Authenticated Open WebUI API | → | Knowledge Base Extraction, indexing and embeddings |
Question answering — retrieval does not train the model
| User question Selects a Knowledge Base | → | KB retrieval Finds relevant passages | → | llama.cpp Receives question and retrieved context | → | Response User verifies citations and content |
Figure 4 Sync prepares the Knowledge Base. Retrieval selects relevant passages at question time. The model checkpoint is unchanged by both operations.
PREPARE A USEFUL SOURCE COLLECTION
Use documents with readable text, clear titles, current content, and a deliberate access policy. Begin with a small collection of Markdown or plain text files. Supported document extraction depends on the deployed Open WebUI image; a file uploaded successfully still needs successful indexing and a retrieval test.
Scope --rag on configures document retrieval in the UI. It does not select a source folder, create a KB, or schedule ingestion. The rag-sync command performs one synchronization; --rag-sync on on serve creates a timer for repeated runs.
CREATE A KNOWLEDGE BASE AND RUN SYNC
1 CREATE THE COLLECTION AND CREDENTIAL
In the primary shared Open WebUI, create a Knowledge Base and copy its ID from the collection URL or details. Give the syncing account permission to the collection. If account API keys are unavailable, rerun the complete working primary serve configuration with --webui-api-keys on; then check administrator permissions before generating the account key. An inference gateway key cannot authenticate this operation.
install -d -m 0700 /etc/stratistor-ai
install -d -m 0750 /ai-models/company-docs
umask 077
read -r -s -p 'Open WebUI account API key: ' rag_api_key
printf '\n'
printf '%s\n' "$rag_api_key" > /etc/stratistor-ai/rag-sync-key
unset rag_api_key
chmod 0600 /etc/stratistor-ai/rag-sync-key
Place the approved source documents in /ai-models/company-docs. Replace YOUR-KB-ID below with the real collection ID. Keep this nonsecret value available for the scheduled workflow.
2 PREVIEW THE INCREMENTAL DIFFERENCE
deployai rag-sync --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models \
--rag-source /ai-models/company-docs \
--rag-knowledge-id YOUR-KB-ID \
--rag-token-file /etc/stratistor-ai/rag-sync-key \
--rag-sync-mode incremental --webui-tls on --dry-run
3 APPLY AND VERIFY
Repeat the command without --dry-run. Incremental sync reports added, updated, unchanged, and source absent counts. It uploads changed files and waits for indexing before cleaning the superseded revisions of those changed files. Attach the KB to a chat and verify a unique fact with its source.
Authentication behavior A real one shot sync can enable Open WebUI API key support and restart Open WebUI if required. A dry run does not perform that settings change; a reported authentication failure means the preview did not establish a remote diff. The key must belong to an authorized account and be stored in a protected regular file.
CHOOSE INCREMENTAL OR MIRROR SYNC
| Source or collection event | Incremental default | Mirror opt in |
|---|---|---|
| New source file | Uploads and indexes. | Uploads and indexes. |
| Changed source file | Indexes the replacement, then cleans its superseded revision. | Reconciles the changed source content. |
| Unchanged file | Skips unchanged content. | Reconciles according to the sync client. |
| File absent from source | Keeps the indexed KB document. | Can delete the corresponding KB document. |
| Separately managed KB content | Preserves source absent content. | Can reconcile it away if absent from the source. |
| Implementation | Native Go checksum diff and upload. | Isolated oikb Python client. |
INCREMENTAL INPUT RULES
- The source must be a local absolute directory. Hidden files and hidden directories are skipped. Visible symlinks and special files cause failure; targets are not followed.
- Each regular file must be no larger than 50 MiB. A source with no eligible files is rejected. Extraction and indexing still depend on Open WebUI support for the file type.
- Incremental synchronization needs the Open WebUI 0.9.6 or newer knowledge sync API described by this release. Verify compatibility with the specific image you deploy.
WHEN MIRROR IS APPROPRIATE
Use a dedicated KB whose entire document lifecycle is owned by the source folder. Take an operator backup before the first destructive reconciliation and test additions, updates, and removals in a disposable collection. Select --rag-sync-mode mirror explicitly only after confirming the source mount and contents.
Legacy flags --rag-sync-prune on selects mirror unless --rag-sync-mode is explicitly supplied. Existing saved records that lack a mode can retain the older mirror behavior. Always state the intended mode when changing a scheduled configuration.
Preview limit Incremental dry run attempts a real remote diff and can fall back to local inventory with a warning. In this source version, mirror dry run logs the intended oikb command without executing a full remote preview. Treat it as a command preview, not proof of which documents would be deleted.
SCHEDULE AND OPERATE KNOWLEDGE SYNC
After a successful one shot sync, add scheduling to the complete working serve configuration. This example keeps Podman, CPU inference, and the standard UI features. Preserve any gateway, identity, TLS, and sizing options from your deployment record.
deployai serve --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu none --gpu-install off \
--container-runtime podman --container-install off \
--listen-address 127.0.0.1 --webui-bind 0.0.0.0 \
--webui-tls on --web-ui on --web-search on \
--rag on --memory on --rag-sync on \
--rag-source /ai-models/company-docs \
--rag-knowledge-id YOUR-KB-ID \
--rag-token-file /etc/stratistor-ai/rag-sync-key \
--rag-sync-mode incremental --rag-sync-minutes 30
CONFIRM THE JOB AND FIRST UNATTENDED RUN
systemctl list-timers --all 'stratistor-ai-ragsync-*'
systemctl start stratistor-ai-ragsync-qwen3-30b-a3b.service
journalctl -u stratistor-ai-ragsync-qwen3-30b-a3b.service \
-n 60 --no-pager
systemctl cat stratistor-ai-ragsync-qwen3-30b-a3b.timer
The timer uses OnBootSec=5min and OnUnitInactiveSec=30min for this recipe. The interval is relative to completion of the previous run, not a wall clock every half hour. The job is a oneshot service; inactive after successful completion is normal. Check its last result and journal, not only whether it is currently active.
PAUSE AND CHANGE SCHEDULING
# Pause now; this does not change the persisted deployment options
systemctl stop stratistor-ai-ragsync-qwen3-30b-a3b.timer
For a durable configuration change, rerun the full serve command with --rag-sync off or with the revised interval or mode. Confirm the timer’s enabled state and next activation afterward. Stop an already running sync separately or wait for it to finish before maintenance; stopping a timer alone does not cancel its active service.
Avoid collisions Do not intentionally overlap manual and scheduled sync runs. Keep the source filesystem mounted and readable, monitor credential rotation, and verify retrieval after document changes. See the release behavior page for the incremental doctor client check.
APPLICATION ACCESS
EXPOSE AN AUTHENTICATED APPLICATION API
Enable the gateway when software needs a controlled inference endpoint. The default bind is loopback. For remote clients, choose an approved host address, a certificate covering the name clients use, and a separate network access policy.
deployai serve --nodes 1 --model qwen3-30b-a3b \
--storage-root /ai-models --gpu none --gpu-install off \
--container-runtime podman --container-install off \
--web-ui on --webui-tls on \
--api-gateway on --gateway-bind 127.0.0.1 \
--gateway-port 18104 --gateway-rpm 60
The tool creates a persistent random key unless --gateway-key-file points to an existing protected key. It reuses the managed WebUI certificate and private key. A supplied gateway key must be at least 32 characters; protect it with mode 0600.
| Endpoint | Method | Behavior |
|---|---|---|
| /health | GET | Gateway liveness without a key. Does not prove inference health. |
| /v1/models | GET | Authenticated model discovery. |
| /v1/chat/completions | POST | Authenticated chat completion proxy. |
| /v1/completions | POST | Authenticated completion proxy. |
| /v1/embeddings | POST | Proxy only; the selected model and backend must support the requested embedding behavior. |
Limits Rate limiting is per source IP in one minute windows. Clients behind one NAT or proxy may share a limit. The handler caps requests at 4 MiB, rejects unlisted routes or methods, and returns 401 for a missing key, 403 for an invalid key, and 429 when rate limited.
RAG applications This gateway reaches llama.cpp directly. An application requiring Open WebUI Knowledge Base retrieval or user memory needs a separately designed authenticated Open WebUI integration. The gateway key and WebUI account API key are not interchangeable.
TEST API ACCESS AND CONFIGURE HTTPS
A LOCAL AUTHENTICATED REQUEST
The following Bash example loads the key into a temporary curl configuration through process substitution, keeping the expanded bearer value out of the curl argument list. Do not enable shell tracing while handling credentials.
api_key=$(cat /etc/stratistor-ai/api-gateway/qwen3-30b-a3b/api-key)
curl --cacert \
/etc/stratistor-ai/open-webui/qwen3-30b-a3b/tls/server.crt \
--config <(printf 'header = "Authorization: Bearer %s"\n' "$api_key") \
https://127.0.0.1:18104/v1/models
unset api_key
Use the returned model ID for a completion request. First prove authenticated model discovery, then submit a short /v1/chat/completions request from the actual client environment. A successful /health response alone only confirms the gateway listener.
SELF SIGNED AND ORGANIZATION ISSUED CERTIFICATES
HTTPS is provided directly by Open WebUI’s Uvicorn server; no extra reverse proxy is installed. Default generated certificates contain localhost, hostnames, local addresses, and requested SANs. Trust the certificate on the client and use a matching DNS name or address.
# Add to the complete working serve command for a self signed identity
--webui-tls on --webui-tls-san ai.example.com,192.168.59.101
Or supply a real matching PEM pair
--webui-tls on --webui-tls-cert /etc/pki/ai/fullchain.pem
--webui-tls-key /etc/pki/ai/private-key.pem
The supplied files must be regular files and form a valid current certificate and key pair. They are copied into the private managed TLS directory. If local RAG sync uses HTTPS, the certificate also needs to support the loopback name or address used by that client, normally 127.0.0.1. Confirm both browser and sync trust after replacement.
Renewal Generated certificates are checked during reconciliation and renewed when near expiry or their identity set changes. There is no separate renewal daemon. Keep running your planned maintenance process; renew supplied certificates through your organization’s certificate process and reconcile the full deployment settings.
Network rule Leave --listen-address 127.0.0.1 in place. Managed health checks and the gateway upstream expect the local inference address. Publish only the approved HTTPS listener and restrict source networks.
IDENTITY
INTEGRATE COMPANY IDENTITY
Local accounts are adequate for a pilot. Use OIDC or LDAP when the team needs centralized sign in. Prepare the external service, trust chain, account approval process, and a tested local administrator recovery login before changing an active deployment.
OIDC SIGN IN
Register a client with your identity provider. For the example public URL below, the documented callback is https://ai.example.com:18103/oauth/oidc/callback. Confirm the callback against the exact Open WebUI image and provider configuration you deploy. Store the client secret in a root protected file.
# Append these options to the full working serve command
--identity oidc --identity-url https://ai.example.com:18103 \
--oidc-client-id YOUR-CLIENT-ID \
--oidc-discovery-url https://id.example.com/.well-known/openid-configuration \
--oidc-secret-file /etc/stratistor-ai/oidc-client-secret \
--oidc-provider-name 'Company SSO' \
--webui-tls on --webui-tls-san ai.example.com
LDAP OR ACTIVE DIRECTORY SIGN IN
Use an LDAPS endpoint whose server certificate is trusted by the application environment. The tool enables certificate validation. Create a directory bind identity with the necessary read permissions and keep its password in a protected file.
# Alternative identity options for the full serve command
--identity ldap --ldap-host dc.example.com --ldap-port 636 \
--ldap-search-base 'dc=example,dc=com' \
--ldap-bind-dn 'cn=ai-bind,ou=service,dc=example,dc=com' \
--ldap-password-file /etc/stratistor-ai/ldap-bind-secret \
--ldap-tls on --webui-tls on
VALIDATE AS REAL USERS
- Sign in with a representative user, confirm the expected account role, and test collection permissions.
- Test sign out, a rejected or unapproved account, and the local administrator recovery login.
- Document provider URLs, client registration, secret renewal, and ownership. OIDC or LDAP integration does not by itself define document authorization.
Scope The tool configures Open WebUI integration; it does not deploy an identity provider or promise automatic enterprise group mapping. New user approval and permission behavior require validation in the deployed application.
ROUTINE OPERATIONS
OBSERVE HEALTH AND THROUGHPUT
Use a simple baseline: service state, a real inference request, browser sign in, a known document question, and resource measurements. Compare later changes against that baseline.
RUN DIAGNOSTICS
# With one recorded deployment
deployai doctor --full
deployai benchmark --benchmark-runs 5 --benchmark-tokens 256
deployai metrics
Service and log checks for the worked model
systemctl status stratistor-llama-qwen3-30b-a3b --no-pager
journalctl -u stratistor-llama-qwen3-30b-a3b -n 80 --no-pager
doctor checks service state, checkpoint presence, inference health, search responses, browser health, optional gateway behavior, and configured features. --full adds a real 32 token generation. Configuration checks for RAG and memory do not prove retrieval or recall. The gateway check rejects anonymous access but does not validate your real client key.
BENCHMARK WITH A REPEATABLE METHOD
benchmark performs a warm up followed by the requested number of identical completion requests. It reports prompt processing and generation speed, elapsed time, and median, mean, minimum, and maximum output throughput. Record model revision, backend, context, threads, layer offload, resource limits, and concurrent workloads. The server is configured with one parallel slot; this test is not a multiuser capacity or latency SLA test.
COLLECT TELEMETRY
When the llama.cpp binary supports it, --metrics on enables /metrics on the same inference listener. No Prometheus or Grafana service is installed. Use an existing local collector or a protected connection; the metrics listener inherits --listen-address and is not independently secured.
curl -fsS http://127.0.0.1:8103/metrics
systemctl show stratistor-llama-qwen3-30b-a3b \
-p CPUQuotaPerSecUSec -p MemoryHigh -p MemoryMax
MAINTAIN SERVICES WITHOUT LOSING STATE
RECORD CONFIGURATION BEFORE CHANGING IT
Save the successful serve command and the matching deployment registry entry. serve reconciles supplied values and parser defaults; it is not a general partial settings merge. Omitting --api-gateway on or --rag-sync on can stop those optional services, and omitting identity or sizing options can replace that configuration.
STOP THE SELECTED INFERENCE AND UI SERVICES
# Pause optional scheduled ingestion and gateway if enabled
systemctl stop stratistor-ai-ragsync-qwen3-30b-a3b.timer
systemctl stop stratistor-ai-gateway-qwen3-30b-a3b.service
deployai stop --nodes 1 --model qwen3-30b-a3b
--storage-root /ai-models
Only run optional unit commands when those units exist. deployai stop disables the selected model inference and RPC units, but retains the shared WebUI when --webui-mode shared is in effect. A per-model legacy UI follows the legacy stop behavior. Shared SearXNG is not automatically stopped; verify dependent services before maintenance.
RESTART A CONFIGURED INDEPENDENT DEPLOYMENT
systemctl enable --now stratistor-llama-qwen3-30b-a3b.service
systemctl enable --now stratistor-open-webui-qwen3-30b-a3b.service
deployai doctor --full
Restore optional gateway and timer services when appropriate. For an inference setting change, rerun the complete recorded serve command. A secondary model must not stop or replace the primary shared WebUI; do not restart Ceph-managed Podman or storage services as part of AI maintenance.
MULTIPLE INSTALLED MODELS
With several installed models, identify the intended inference deployment explicitly on status, stop, and other model-scoped calls. In shared mode, model endpoints are discovered from active local systemd units. Run WebUI operator-state backup/restore under the pinned primary model; a secondary model backup is refused to protect consistent UI state. Store the shared owner identity with the site record.
Keep the executable available Scheduled sync and the gateway use a stable installed helper under /usr/local/libexec/stratistor-ai. Replacing the CLI in another directory does not automatically update every managed helper; reconcile the full serve configuration when rolling out a new deployai executable.
LIVE CONFIGURATION
INCREASE CONTEXT WITHOUT REDEPLOYING WEIGHTS
The message “request (8860 tokens) exceeds the available context size (4096 tokens)” means the assembled request, including chat history and any RAG or search passages, is too large for the active inference window. For Llama 3.1 8B start with --ctx-size 16384 and monitor memory. A bigger context increases KV-cache RAM requirements.
RERUN SERVE, NOT DEPLOY, FOR THE EXISTING CLUSTER
./deployai serve \
--nodes stratistor1-1,stratistor1-2,stratistor1-3 \
--model llama3.1-8b --mode distributed \
--storage-root /stratistor/clustermounts/machines \
--install-root /stratistor --gpu none --gpu-install off \
--threads 8 --ctx-size 16384 \
--resource-profile shared --resource-isolation auto \
--rpc-address-2 "$RPC2" --rpc-address-3 "$RPC3" \
--container-runtime docker --container-install off \
--web-ui on --webui-mode shared \
--web-search on --rag on --memory on --webui-tls on
Use the complete previously successful serve command and change only the intended parameter; the example assumes RPC2/RPC3 contain the correct private worker IPv4 addresses. serve reconciles services in place, without rebuilding or re-downloading model weights. An inference-service restart can interrupt active chats.
CONFIRM THE SETTING APPLIED
systemctl cat stratistor-dist-llama3.1-8b.service | grep -E -- "ctx-size| -c "
ps -eo pid,args | grep "[l]lama-server"
curl -fsS http://127.0.0.1:8105/health
If the prompt remains too long Trim chat history and reduce RAG passages or retrieved documents. A 16K window is a starting recommendation, not a resource guarantee. Re-run a real request after changing the setting.
BACKUP AND RECOVERY
BACK UP OPERATOR STATE
A recoverable deployment needs two sets of assets: application state and the software or model material needed to run it. The built in backup captures operator state and intentionally excludes model weights.
| Captured when present | Retain separately |
|---|---|
| Open WebUI database, chats, uploaded documents, vector index, and memory data. | GGUF or Kimi checkpoints, revision and verification metadata, and any required Hugging Face access credentials. |
| Per model UI configuration, managed TLS material, session secrets, and deployment registry. | OS, kernel, GPU stack, llama.cpp binaries or a reproducible build record, and container images or pinned image references. |
| Selected AI units, managed gateway configuration and key, SearXNG settings, and RAG timer definitions. | External identity provider configuration, original document source folders, and externally managed secrets outside the captured paths. |
| Default RAG key and configured RAG key files inside /etc/stratistor-ai. | Embedding model download caches, which may need to be downloaded again. Keep an offline cache copy if recovery has no internet access. |
CREATE AND VALIDATE AN ARCHIVE
deployai backup
ls -lht /var/backups/stratistor-ai/
Set the exact archive path reported by backup
BACKUP=/var/backups/stratistor-ai/EXACT-operator-state-ARCHIVE.tar.gz
tar -tzf "$BACKUP" >/dev/null
sha256sum "$BACKUP"
A running RAG sync blocks backup. In shared mode perform a WebUI state backup using the pinned primary model. On an approved backup, the tool pauses the timer, quiesces WebUI for a consistent database snapshot, archives state, restarts the UI, and checks readiness. Model weights remain excluded. Use an explicit primary model when multiple deployments exist.
Retention Archives contain credentials and personal or company content. Copy them to an approved off host destination, preserve restrictive permissions, and apply the organization’s encryption and retention controls. The tool has no automatic backup schedule or retention pruning command.
VALIDATE AND EXECUTE A RESTORE
Restore writes the saved state back to managed paths and interrupts affected services. Test on an isolated target with the same model identity and expected paths before relying on it for disaster recovery. Preserve the current state and the exact archive you plan to restore.
1 VALIDATE THE ARCHIVE WITHOUT APPLYING IT
deployai restore --model qwen3-30b-a3b --nodes 1 \
--storage-root /ai-models --restore-file "$BACKUP" --dry-run
Dry run checks archive structure, allowed paths, embedded manifest, model identity, and safe symlinks. It does not prove the database is compatible with a different image or that every external dependency is available. Match the original WebUI data root and custom RAG key path when applicable.
2 PERFORM THE PLANNED RESTORE
deployai restore --model qwen3-30b-a3b --nodes 1 \
--storage-root /ai-models --restore-file "$BACKUP"
The tool takes a safeguard backup, records active service state, stops affected services, restores validated data, and resumes previously active dependencies. It stages the previous UI trees while extracting. A failed extraction or restart still requires operator review; the full environment is not an atomic transaction.
3 VERIFY APPLICATION AND SERVICE BEHAVIOR
- Confirm inference and HTTPS readiness, then sign in using an appropriate restored account.
- Inspect conversation history, a known document answer, source permissions, and memory behavior.
- Check the gateway with the intended key and the scheduled sync timer with its intended source and mode.
- Confirm restored credentials remain valid with external providers. A historical key or certificate may have been revoked or expired.
New host recovery Prepare the OS, runtime, matching model weights, binaries, and storage mounts before the restore. Retain matching container image versions; application schema migrations may prevent recovery by image downgrade alone. Restore does not reinstall a complete server or reconstruct remote RPC workers.
MAINTENANCE
UPDATE THE DEPLOYED SOFTWARE
update discovers saved GGUF deployments and preserves their recorded runtime, paths, ports, and service options. Without --model it processes all discovered deployments. Plan a maintenance window for changes that restart inference or the UI.
deployai update --dry-run
deployai backup
deployai update --model qwen3-30b-a3b
deployai doctor --full
| Choice | Effect |
|---|---|
| update | Checks the configured UI and search images, updates compatible independent llama.cpp software, reconciles services, and updates an enabled mirror sync client. |
| --update-llama off | Skips the llama.cpp build update while continuing UI and search update handling. |
| --update-models on | Explicitly permits checkpoint revision replacement. New content is staged and verified; allow additional storage and download time. |
| --model ID | Scopes the discovered deployment selection. It does not isolate shared backend binaries or the shared search service. |
| --dry-run | Lists the deployments and intended checks. It does not fetch an exhaustive version diff or prove an upstream update will be compatible. |
WHAT THE UPDATER PROTECTS AND WHAT TO VERIFY
Independent backend updates build in staging and validate capabilities before promotion; earlier binaries are retained for rollback attempts. Checkpoint updates respect --model-verify on|off. Shared WebUI state and its selected primary owner are preserved; changing registered API connections may restart the primary WebUI container. Validate inference and chat access afterward.
Open WebUI defaults to a moving main tag and SearXNG to latest. Select a tested Open WebUI image reference through --webui-image on serve if controlled versioning is required. update pulls the configured UI image without needing --update-webui; that Boolean flag controls image refresh during ordinary setup.
Separate maintenance The generic updater does not update the deployai executable, OS, custom kernel, GPU drivers, signed DKMS, DOCA, or Ceph. It keeps distributed RPC backend binaries pending a coordinated rollout. Back up each model’s state when several deployments are in scope.
TROUBLESHOOTING
TROUBLESHOOT BY THE FAILING LAYER
| Symptom | First checks and corrective action |
|---|---|
| Missing llama-server or backend | Run deploy with the same model, GPU choice, and mode. Check /opt/llama.cpp backend paths and the deployment log. |
| GPU preflight stops | Read the exact ownership or prerequisite error. Verify device binding, vGPU host manager, /dev/kfd, CUDA or HIP, kernel build tree, MOK, and Secure Boot as relevant. |
| Browser cannot connect | Check the model UI port, HTTPS scheme, bind address, systemd unit, network policy, and certificate trust. Inspect the WebUI journal before restarting. |
| UI works but inference fails | Check the private inference /health, model file, RAM or VRAM, context, and inference journal. Keep the managed loopback address available. |
| Request tokens exceed context | Rerun the complete existing serve command with a larger explicit --ctx-size (e.g. 16384); verify active llama-server process args, monitor KV-cache memory, and trim RAG/history. |
| Secondary model missing in WebUI | Verify shared-owner, secondary inference systemd service, private API /health, and OPENAI_API_BASE_URLS in the primary environment. Run serve for that model to refresh local connections. |
| Docker install fails while Podman exists | Preserve Ceph Podman/runc; resolve a genuine DNF package conflict or podman-docker alias manually. Never use forced package removals as an AI installer workaround. |
| RAG sync returns 401 or 403 | Use the real WebUI account key, check API key support and KB permission, and confirm the token file. A gateway key is the wrong credential. |
| RAG upload succeeds but answer misses it | Wait for indexing, attach the correct KB to the chat, check extraction and permissions, then use a unique document fact as the test. |
| Knowledge sync rejects source | Check visible symlinks or special files, the 50 MiB incremental limit, empty eligible inventory, source access, and the actual mount. |
| Gateway 401 403 429 or 502 | Check missing key, invalid key, shared source IP rate limit, or unavailable inference respectively. Test /v1/models with the correct key. |
| Runtime shared library error after update | Inspect the updater log and backend library bundle. The updater validates and attempts runtime recovery; preserve the previous binary directory while investigating. |
| Slow inference or out of memory | Review weights, context, offload, CPU quotas, memory ceilings, indexing load, and concurrent host workloads. Benchmark one change at a time. |
| Distributed worker unavailable | Check private IPv4, default RPC port, firewall, exact host order, shared filesystem proof, compatible RPC builds, and local worker journals. |
journalctl -u stratistor-open-webui-qwen3-30b-a3b -n 80 --no-pager
ss -lntp
ls -lt /var/log/stratistor-ai-go/
VERSION SPECIFIC BEHAVIOR TO RECOGNIZE
Use these behaviors when interpreting deployai 1.3.4 with the 9 November 2026 patches. A successful size check is not a cryptographic hash check. WebUI ownership and selected inference service are independent, and the shared interface is not a high-availability cluster.
READINESS AND DRY RUN SCOPE
preflight avoids driver or compute package installation, but the AMD path can still reconcile the stable ROCm repository file even with --gpu-install off. Use plan and gpu-status for initial inspection when repository changes must be avoided. Diagnostic and benchmark requests can contact services even when --dry-run is present; it is not a global network or inference suppression switch.
SCHEDULED INCREMENTAL SYNC AND DOCTOR
The doctor implementation checks for the oikb binary whenever scheduled RAG is enabled, including native incremental mode that does not require it. A missing client diagnostic can therefore be unrelated to an incremental job’s ability to run. Verify the sync job journal, the KB contents, and retrieval before treating this isolated check as ingestion failure.
EXTERNAL SEARCH AND HEALTH CHECKS
When search is enabled, doctor checks the local SearXNG systemd unit as well as a live query. If --searxng-url selects an external service, an inactive local unit can still appear as a failure. Interpret it against the external endpoint test. Health checks generally use loopback; custom bind designs must preserve the required local connectivity.
SINGLE MODEL DISCOVERY
doctor, benchmark, metrics, backup, and restore can load a single saved deployment when --model is omitted. update has its own all deployment discovery. status and stop require an explicit intended model for reliable selection. Always provide full target details for one shot rag-sync; its required sync options are validated before operational discovery.
SHARED SERVICES AND DISTRIBUTED CONTROL
SearXNG and backend build paths are shared. Open WebUI is also shared by default on each coordinator: the primary database and TLS/credentials are reused, while each active model has its own API. A model-scoped stop does not stop the shared UI. Multi-host distributed status, stop, and standalone test remain limited; use local systemd checks when necessary.
Operational acceptance Service health, source reported test results, and a plan are separate forms of evidence. Complete the real account, retrieval, memory, GPU, and recovery tests needed for your installation before production acceptance.
PRODUCTION READINESS
ACCEPT AND HAND OVER A DEPLOYMENT
Use this checklist to agree on the intended service and demonstrate that it works. Record the evidence, owner, and date in the site’s deployment record.
| Area | Acceptance evidence |
|---|---|
| Model and capacity | Chosen model and revision; representative answer quality; acceptable prompt and generation throughput; tested context and concurrent use. |
| Host coexistence | Observed CPU, RAM, GPU, and container limits; acceptable storage and VM latency while inference and ingestion are active. |
| Network and TLS | Private inference and RPC scope; approved browser and gateway clients; valid certificate identity, trust, and renewal ownership. |
| Users and identity | Named administrators; tested local recovery login; expected approval and collection access; validated OIDC or LDAP if used. |
| Knowledge | Approved source path and owner; known KB ID; declared incremental or mirror policy; real indexing and retrieval acceptance test. |
| Credentials | Protected WebUI, RAG, gateway, and identity secrets; rotation and recovery procedures; approved custody of backup archives. |
| Maintenance | Complete serve command and registry record; pinned or recorded image choices; update procedure; service dependencies and rollback assets. |
| Recovery | Off host operator backup; separately retained weights, binaries or build record, secrets, and offline caches where needed; successful isolated restore. |
| Monitoring | Real inference and HTTPS checks; timer job results; local metrics collection; disk free space and backup completion review. |
| Ownership | Named operational owner; support route; maintenance window; agreed recovery time and recovery point objectives. |
A PRACTICAL HANDOVER TEST
Ask a new authorized user to sign in, select the model, retrieve a fact from a permitted KB, and report the source. Have the operator show a successful backup and a recent sync result, then explain how to pause the service and restore it. This connects infrastructure checks with the service the team actually needs.
Keep current Store changes to the serve command, secret paths, model revision, and image choice with the site record. Rerun the relevant acceptance tests after changes to the model, identity, document extraction, network path, or recovery environment.
COMMAND REFERENCE
SELECT THE RIGHT COMMAND
Syntax is deployai COMMAND [OPTIONS]. Place options after the command. With arguments but no explicit command, the parser defaults to deploy; use a named command in runbooks. With no arguments, the program shows help.
| Command | Purpose and operational effect |
|---|---|
| list-models | Lists built in GGUF profiles, the Kimi compatibility profile, and the custom option. |
| gpu-status | Reads PCI GPU inventory, loaded modules, and available NVIDIA or AMD runtime signals. |
| plan | Calculates CPU and context recommendations and displays hardware information. Does not establish model fit or apply quotas. |
| preflight | Validates host, storage, and selected GPU readiness without driver or toolkit installation. See the AMD repository side effect note. |
| deploy | Builds the chosen backend, downloads a pinned checkpoint, checks full integrity by default (or size only with --model-verify off), and normally runs a CLI smoke test. |
| serve | Starts/reconciles the selected model inference service and default shared Open WebUI (or per-model UI if requested), search, gateway, and sync services. |
| test | Runs a standalone model CLI smoke test. Uses a fixed 2048 context in the GGUF test path; does not validate the online UI. |
| status | Reports selected model units and model path. Inactive unused topology units can be normal. Explicitly select the model. |
| stop | Stops the selected model inference/RPC services; preserves the pinned shared WebUI when --webui-mode shared. Model data is retained. |
| doctor | Checks configured services and endpoints. --full issues real inference. User retrieval and memory still need separate tests. |
| benchmark | Runs one warm up and repeated fixed inference generations against the local serving API. |
| metrics | Prints the local llama.cpp Prometheus response to standard output, with normal tool logging also present. |
MAINTENANCE COMMANDS AND HELP
| Command | Purpose and operational effect |
|---|---|
| backup | Captures operator state, briefly quiescing WebUI and a scheduled sync timer. Excludes model weights. |
| restore | Validates and applies a selected operator archive. --restore-file is required; --dry-run validates without applying. |
| update | Discovers deployed GGUF configurations, checks software updates, and reconciles service configuration. |
| rag-sync | Synchronizes one source directory to an existing Knowledge Base using the specified WebUI account key. |
| rpc-worker | Advanced managed worker setup action used by distributed deployment. Not the normal operator entry point. |
| gateway-runtime | Internal systemd runtime. It consumes --gateway-config PATH, not the ordinary deployment option set. |
| mcp-stdio | Internal native SearXNG MCP transport. It accepts --searxng-url for its runtime; standard deployment logging is suppressed to preserve the protocol. |
HELP FORMS
deployai --help
deployai help all
deployai --help-all
deployai help gpu
deployai help rag
deployai help api
deployai help identity
deployai help update
deployai help distributed
The patched deployment parser registers 88 documented flags, including --model-verify, --webui-mode, and --webui-primary-model. -h and --help are top-level help shortcuts; --gateway-config is an internal runtime argument. Use deployai help all to confirm support in the executable you installed.
BOOLEAN FLAGS AND ON OR OFF SETTINGS
A Boolean flag is enabled by writing the flag alone, for example --verbose or --dry-run. Use --verbose=false to explicitly clear one. String toggles require a value, for example --web-search off. A bare --web-search is invalid. Quoted paths are accepted; bind addresses must be literal IP addresses rather than DNS names.
Preview with care Use plan for sizing, update --dry-run for deployment discovery, restore --dry-run for archive validation, and incremental rag-sync --dry-run for a remote diff attempt. Review command specific behavior rather than assuming one dry run contract for every command.
PARAMETER REFERENCE
DEPLOYMENT AND MODEL SELECTION
All 88 registered deployment flags are organized across the eight parameter-reference pages. Defaults are raw source-code defaults before environment values, auto sizing, or saved deployment discovery. Unset means no parser value.
| Option | Default | Use and constraints |
|---|---|---|
| --model | kimi-k3 | Profile ID. Select explicitly for a new GGUF deployment; list-models shows choices. |
| --mode | independent | independent or distributed. Distributed requires two or three hosts and explicit CPU selection. |
| --nodes | 3 | 1 for the local host; 2 or 3 expands to the built in stratistor1 hostnames. Otherwise use one to three unique comma separated hostnames. |
| --storage-root | Unset | Existing real absolute directory for checkpoints. Required for deploy, serve, preflight, and multi host work. |
| --shared-root | Unset | Legacy alias for --storage-root. Prefer the current name and do not supply conflicting aliases. |
| --install-root | auto | auto, /stratistor, /stratiserv, or /baseos. Resolves platform and MOK material for GPU preparation. |
| --hf-repo | Unset | ORG/REPO Hugging Face source for custom-gguf only. Required together with --hf-file. |
| --hf-file | Unset | Single basename ending .gguf for custom-gguf only; no directory path. The custom service identity and port are shared. |
INFERENCE SIZING AND STAGING
| Option | Default | Use and constraints |
|---|---|---|
| --threads | 64 raw | CPU inference threads, 1 to 1024. Auto sizing can replace the raw default. An explicit CLI value wins. |
| --ctx-size | 4096 raw | Context tokens, 512 to 131072. Auto sizing can replace the raw default; actual model and memory limits still apply. |
| --sizing | auto | auto or off. Auto applies thread and context heuristics during deploy, serve, preflight, and plan. |
| --resource-profile | auto | auto, shared, or dedicated. Selects sizing and isolation policy; auto detects active Ceph or virtualization services. |
| --resource-isolation | auto | auto, on, or off. Auto adds systemd and OCI resource ceilings on detected shared hosts. |
| --build-jobs | 16 | Parallel compiler jobs, 1 to 256. Controls building, not inference request concurrency. |
| --tokens | 2 | CLI smoke test output tokens, 1 to 128. Does not set the browser chat output limit. |
| --verify-jobs | 8 | Compatibility checksum workers for Kimi; native GGUF hashing is not controlled by this flag. For GGUF integrity use --model-verify. |
| --model-verify | on | on performs the standard pinned-Hub SHA-256 or hf cache verification. off checks pinned revision and file size only; no cryptographic validation. GGUF workflow; default on. |
| --download-workers | 16 | Compatibility download worker count passed to Kimi. Native single GGUF download does not use this flag for its transfer concurrency. |
| --preset | auto | Kimi only: auto, server, max, workstation, desktop, laptop, or ultra. No effect on normal GGUF serving. |
Override rule To force a thread or context value, supply the CLI flag explicitly. Values inherited from environment variables do not set the parser’s explicit override markers and can be replaced by automatic sizing.
GPU REASONING AND RPC
| Option | Default | Use and constraints |
|---|---|---|
| --gpu | auto | auto, none, nvidia, or amd. Auto selects a detected vendor, CPU if none, and errors if both vendors are present. |
| --gpu-install | on | on or off. Allows permitted missing GPU prerequisite installation. Off does not turn all deployment package installation off. |
| --gpu-layers | -1 | -1 for maximum requested offload, or 0 to 999 layers. Does not guarantee VRAM fit. Positive layers are invalid with CPU selection. |
| --nvidia-driver-flavor | open | open or proprietary. Chooses the NVIDIA driver path if installation is needed; it is not permission to replace a conflicting host stack. |
| --qwen-thinking | off | on or off. Off asks compatible Qwen3 server builds to disable reasoning output through supported template or reasoning flags. |
| --rpc-port | 0 | 0 derives 40000 plus the model API port. Use the same valid TCP port on all RPC peers; distributed default is recommended. |
| --rpc-address-2 | Unset | Private IPv4 address for the second listed distributed worker host. Required for distributed multi host actions. |
| --rpc-address-3 | Unset | Private IPv4 address for the third listed worker. Required when selecting three distributed hosts. |
Distributed ports Use the derived RPC port in this release unless you have validated all peer startup commands. Remote worker setup does not propagate a nondefault --rpc-port consistently.
RUNTIME BROWSER AND FEATURES
| Option | Default | Use and constraints |
|---|---|---|
| --container-runtime | docker | docker or podman. Both may coexist; selects the runtime used by managed AI containers. Does not request uninstall of the other runtime. |
| --container-install | both | both, selected, or off. selected attempts install only the chosen runtime; off uses what exists. Existing Podman/Docker/Ceph packages are protected; real RPM conflicts fail without forced replacement. |
| --web-ui | on | on or off. In shared mode (default), this controls one primary Open WebUI for all active local model inference services. |
| --webui-mode | shared | shared (default): one primary UI across active local models. per-model: legacy one UI per model. |
| --webui-primary-model | Unset | Optional existing primary model ID during first shared-mode serve; owner is pinned and cannot be changed implicitly afterward. |
| --web-search | on | on or off. Configures SearXNG integration. A UI disabled fresh deployment should also set this off unless an MCP adapter is intentionally configured. |
| --rag | on | on or off. Configures Open WebUI document retrieval; does not automatically select, create, or sync a KB. |
| --memory | on | on or off. Configures user memory capabilities and review settings. Recall requires real application testing. |
| --memory-review-turns | 10 | Memory background review cadence, 1 to 1000 conversation turns. |
| --listen-address | 127.0.0.1 | Literal inference bind IP. Metrics inherit this address. Keep loopback for the managed health and gateway workflows. |
| --webui-bind | 0.0.0.0 | Literal UI bind IP. All IPv4 interfaces by default; constrain network access separately. |
| --webui-port | 0 | 0 derives API port plus 10000. For an override, choose a free valid TCP port and reuse it in diagnostics and sync. |
| --webui-image | main image | Default ghcr.io/open-webui/open-webui:main. Accepts an OCI image reference; use a tested reference for controlled updates. |
DATA SEARCH TLS AND API KEYS
| Option | Default | Use and constraints |
|---|---|---|
| --webui-data-root | /var/lib/stratistor-ai/open-webui | Persistent state root. Default shared mode preserves the chosen primary model data directory; secondary models do not create replacement databases. |
| --webui-tls | on | on or off. Native HTTPS in the UI container. The gateway and external identity require on. |
| --webui-tls-san | Unset | Comma separated extra DNS names or IPs for generated certificates. Include the name clients actually use. |
| --webui-tls-cert | Unset | Absolute existing regular PEM certificate file, supplied together with --webui-tls-key. The pair must match and be current. |
| --webui-tls-key | Unset | Absolute matching regular PEM private key. Material is copied to the managed TLS folder; source paths are not mounted directly. |
| --webui-tls-renew-days | 30 | Generated certificate renewal threshold, 1 to 365 days. Evaluated on reconciliation; no standalone renewal daemon. |
| --searxng-port | 18888 | Local SearXNG loopback port, 1024 to 65535. Search is shared, so coordinate changes across models. |
| --searxng-url | Unset | External SearXNG HTTP or HTTPS base URL, replacing local setup for that configuration. |
| --webui-api-keys | auto | auto, on, or off. Auto preserves the existing setting; requested real sync enables API key support when needed. Cannot be off with sync. |
KNOWLEDGE SYNC AND GATEWAY
| Option | Default | Use and constraints |
|---|---|---|
| --rag-sync | off | on or off. On in serve creates a scheduled ingestion timer. Requires RAG and WebUI plus source, KB, and token settings. |
| --rag-source | Unset | Absolute local source directory. Incremental mode rejects visible symlinks, special files, empty eligible content, and files above 50 MiB. |
| --rag-knowledge-id | Unset | Existing Open WebUI collection ID. The syncing account needs permission; the tool does not create the collection. |
| --rag-token-file | Unset | Absolute protected regular file containing the real Open WebUI account API key. Use mode 0600. |
| --rag-sync-mode | incremental | incremental adds and updates while preserving absent documents; mirror also reconciles deletions. Explicit mode overrides the legacy alias. |
| --rag-sync-prune | off | Legacy on or off alias. on selects mirror unless an explicit --rag-sync-mode wins. |
| --rag-sync-minutes | 30 | Minimum 5 minutes. Interval is measured from the previous oneshot job becoming inactive; boot delay is five minutes. |
| --api-gateway | off | on or off. Optional authenticated HTTPS inference proxy; requires TLS enabled WebUI. |
| --gateway-bind | 127.0.0.1 | Literal gateway listener IP. Change only with an approved remote client and network design. |
| --gateway-port | 18104 | Gateway TCP port, 1024 to 65535. Avoid collision with another model UI port. |
| --gateway-rpm | 60 | Per source IP request limit per minute, 1 to 100000. NAT or proxy clients can share the limit. |
| --gateway-key-file | Unset | Optional absolute protected existing key file. If omitted, creates a managed key. Runtime requires at least 32 characters. |
IDENTITY INTEGRATION
| Option | Default | Use and constraints |
|---|---|---|
| --identity | local | local, oidc, or ldap. Configures browser authentication; document roles and authorization still require validation. |
| --identity-url | Unset | External HTTPS UI URL. Required for OIDC redirect configuration, including a nonstandard port when used. |
| --oidc-client-id | Unset | OIDC client registration identifier. Required for oidc. |
| --oidc-discovery-url | Unset | HTTPS OpenID configuration discovery URL. Required for oidc. |
| --oidc-secret-file | Unset | Absolute protected file holding the OIDC client secret. Required for oidc; keep mode 0600. |
| --oidc-provider-name | StratiSYSTEM SSO | Label shown for the external login provider. |
| --ldap-host | Unset | Directory server hostname. Required for ldap. |
| --ldap-port | 636 | Directory server TCP port, 1 to 65535; normally LDAPS 636. |
| --ldap-search-base | Unset | Directory base DN to search. Required for ldap. |
| --ldap-bind-dn | Unset | Directory bind principal DN. Required for ldap. |
| --ldap-password-file | Unset | Absolute protected file containing the directory bind password. Required for ldap. |
| --ldap-tls | on | on or off. Enables TLS for LDAP and certificate validation. Use a trusted directory certificate. |
DIAGNOSTICS UPDATES AND CONTROL FLAGS
| Option | Default | Use and constraints |
|---|---|---|
| --full | false | Boolean. doctor performs real inference in addition to endpoint and configuration checks. |
| --metrics | on | on or off. Requests the llama.cpp metrics endpoint when supported; shares the inference listener. |
| --benchmark-runs | 3 | Measured generation runs, 1 to 30, following a warm up. |
| --benchmark-tokens | 128 | Requested output tokens per measured run, 16 to 4096. |
| --restore-file | Unset | Absolute operator archive path for restore. Use --dry-run first for structural validation. |
| --update-llama | on | on or off for update. Distributed backend binary updates are deliberately skipped pending coordinated rollout. |
| --update-models | off | on or off for update. Explicit on permits new checkpoint revisions and can consume significant extra space. |
| --update-source | false | Boolean. During deploy, fetches and fast forwards llama.cpp source before building; also passed to the Kimi compatibility worker. |
| --update-webui | false | Boolean. During normal setup, pulls the selected UI image. update checks that image independently. |
| --force-build | false | Boolean. Ignores a matching build receipt and rebuilds during deploy. Does not bypass GPU safety checks. |
| --force-test | false | Boolean. Repeats the deploy smoke test instead of reusing its prior matching receipt. |
| --reverify-model | false | Boolean. Rechecks checkpoint content even when a prior verification marker is reusable. |
| --no-test | false | Boolean. Skips the automatic post deploy CLI model test. Validate real inference separately. |
| --verbose | false | Boolean. Shows detailed external command and build output; detailed logs are also written to the log directory. |
| --dry-run | false | Boolean. Command specific preview; restore validates, update lists, incremental sync attempts a diff. Does not universally suppress all service requests. |
ENVIRONMENT REFERENCE
UNDERSTAND DEFAULTS AND PRECEDENCE
Environment variables supply initial values for selected flags. Explicit CLI values then replace them. Automatic sizing follows parsing and can replace thread or context values unless those particular flags appeared on the command line. Operational discovery can instead load an entire saved deployment.
| Situation | Effective behavior |
|---|---|
| New deploy or serve | Uses parsed CLI and environment settings, then applicable automatic sizing. The successful serve configuration is saved. |
| Existing serve rerun | Does not merge every omitted option from the registry. Reuse the complete recorded command. |
| One saved deployment and no explicit model | doctor, benchmark, metrics, backup, and restore can use the saved configuration. Some command specific controls are then applied. |
| Explicit operational model | Do not assume every saved port, TLS setting, or disabled feature was merged. Supply relevant matching options. |
| update | Discovers recorded deployments and applies update controls. Without an explicit model it processes all discovered records. |
| No dedicated environment mapping | Use the documented CLI option or recorded configuration. Do not invent an environment name from a flag. |
CORE ENVIRONMENT MAPPINGS
| Environment variable | Equivalent option |
|---|---|
| KIMI_MODEL_ID | --model |
| STRATISTOR_AI_MODE | --mode |
| STRATISTOR_AI_NODES | --nodes |
| STRATISTOR_AI_STORAGE_ROOT | --storage-root |
| KIMI_SHARED_ROOT | Fallback storage root when the current variable is unset |
| GGUF_CUSTOM_REPO and GGUF_CUSTOM_FILE | --hf-repo and --hf-file |
| STRATISYSTEM_INSTALL_ROOT | --install-root |
| KIMI_THREADS and KIMI_BUILD_JOBS | --threads and --build-jobs |
| STRATISTOR_AI_CONTEXT_SIZE | --ctx-size |
| KIMI_TOKENS and KIMI_PRESET | --tokens and --preset |
| STRATISTOR_VERIFY_JOBS and STRATISTOR_DOWNLOAD_WORKERS | --verify-jobs and --download-workers |
| STRATISTOR_AI_MODEL_VERIFY | --model-verify (on|off, default on) |
ENVIRONMENT VARIABLE MAPPING
| Environment variable | Equivalent option |
|---|---|
| STRATISTOR_AI_GPU | --gpu |
| STRATISTOR_AI_GPU_INSTALL | --gpu-install |
| STRATISTOR_NVIDIA_DRIVER_FLAVOR | --nvidia-driver-flavor |
| STRATISTOR_AI_GPU_LAYERS | --gpu-layers |
| STRATISTOR_AI_CONTAINER_RUNTIME | --container-runtime |
| STRATISTOR_AI_CONTAINER_INSTALL | --container-install |
| STRATISTOR_WEBUI_MODE and STRATISTOR_WEBUI_PRIMARY_MODEL | --webui-mode and --webui-primary-model |
| STRATISTOR_WEB_UI and STRATISTOR_WEB_SEARCH | --web-ui and --web-search |
| STRATISTOR_AI_RAG and STRATISTOR_AI_MEMORY | --rag and --memory |
| GGUF_LISTEN_ADDRESS | --listen-address |
| STRATISTOR_WEBUI_BIND and STRATISTOR_WEBUI_PORT | --webui-bind and --webui-port |
| STRATISTOR_WEBUI_IMAGE | --webui-image |
| STRATISTOR_WEBUI_DATA_ROOT | --webui-data-root |
| STRATISTOR_WEBUI_TLS and STRATISTOR_WEBUI_TLS_SAN | --webui-tls and --webui-tls-san |
| STRATISTOR_WEBUI_TLS_CERT and STRATISTOR_WEBUI_TLS_KEY | --webui-tls-cert and --webui-tls-key |
| STRATISTOR_WEBUI_TLS_RENEW_DAYS | --webui-tls-renew-days |
| STRATISTOR_SEARXNG_PORT and STRATISTOR_SEARXNG_URL | --searxng-port and --searxng-url |
| STRATISTOR_MEMORY_REVIEW_TURNS | --memory-review-turns |
| STRATISTOR_QWEN_THINKING | --qwen-thinking |
| STRATISTOR_RPC_PORT | --rpc-port |
| STRATISTOR_RPC_IP_2 and STRATISTOR_RPC_IP_3 | --rpc-address-2 and --rpc-address-3 |
| STRATISTOR_AI_UPDATE_LLAMA and STRATISTOR_AI_UPDATE_MODELS | --update-llama and --update-models |
Secrets Use the protected file options for identity, RAG, and gateway credentials. The general parser does not expose direct environment mappings for every newer operational feature. Avoid recording secret values in shell history or deployment command text.
OPERATOR REFERENCE
PATHS SERVICES AND SITE RECORD
| Path or unit pattern | Contents or responsibility |
|---|---|
| /ai-models/ai/models/MODEL/ | Checkpoint and verification metadata for the example storage root. custom-gguf adds a repository and file hash subdirectory. |
| .gguf-verified / .gguf-size-validated (within model directory) | Verification receipt after full Hugging Face checksum validation / size-only receipt when checksums are intentionally disabled. |
| /opt/llama.cpp/ | Shared source checkout and backend builds: build, build-cuda, build-hip, and build-rpc. |
| /var/lib/stratistor-ai/open-webui/MODEL/ | Persistent WebUI state for the pinned primary model (shared mode); standalone state directories remain preserved for legacy per-model instances. |
| /etc/stratistor-ai/open-webui/MODEL/ | Environment, admin-credentials and TLS for the pinned primary shared WebUI; secondary model deployments reuse them. |
| /etc/stratistor-ai/open-webui/shared-owner | Pinned primary model identity for the shared WebUI; retain it with backups and deployment records. |
| /etc/stratistor-ai/deployments/MODEL.json | Recorded configuration used for deployment discovery and updates. |
| /etc/stratistor-ai/api-gateway/MODEL/ | Gateway configuration and default generated API key. |
| /var/backups/stratistor-ai/ | Protected operator archives and pre update UI snapshots. |
| /var/log/stratistor-ai-go/ | Timestamped detailed logs. Nonroot logging uses a temporary per user directory. |
| stratistor-llama-MODEL.service | Independent inference process. |
| stratistor-dist-MODEL.service and stratistor-rpc-MODEL.service | Distributed coordinator inference and CPU RPC worker. |
| stratistor-open-webui-MODEL.service and stratistor-ai-searxng.service | One primary shared WebUI unit by default; additional model-specific units may exist from earlier per-model deployments. SearXNG remains shared. |
| stratistor-ai-gateway-MODEL.service and stratistor-ai-ragsync-MODEL.service or .timer | Optional gateway and scheduled document ingestion. |
MINIMUM SITE DEPLOYMENT RECORD
Retain host role and addresses; storage, installation and shared-owner paths; complete deploy and serve commands; checkpoint revision and verification mode; model ID and API port; primary WebUI model and URL; Docker/Podman selection and installation policy; context/threads, measured performance, credentials and recovery contacts; and RAG/memory/identity test evidence.