STEELDOME TECHNOLOGIES

AI DEPLOYMENT GUIDE

General user guide for deploying and using the AI Deployment Tool

STRATISYSTEM AI DEPLOYMENT TOOL USER GUIDE

SteelDome Technologies · StratiSYSTEM OS

Deploy and operate self-hosted AI inference using deployai. This guide covers planning, installation, local inference, GPU acceleration, distributed CPU execution, the shared Open WebUI, knowledge ingestion, application integration, backup, recovery, and ongoing operations.

Audience: IT operators, SteelDome partners, and solution architects. Edition: 12 November 2026.

OPERATOR CONVENTIONS

Example convention Commands run in a root Bash shell. The executable is deployai. /ai-models is the general storage example; /stratistor/clustermounts/machines is the three-node CephFS example. Substitute your hostnames, private RPC IPs, certificates, and Knowledge Base IDs.

Keep a deployment record Retain the complete successful serve command. A later serve call reconciles the options you supply and their defaults; update is the workflow that reloads the saved deployment configuration.

AI CONCEPTS

WHAT THIS TOOL PROVIDES

deployai assembles local AI inference services from several established components. It stages trained GGUF models, builds the inference backend, configures systemd services, and connects multiple local model endpoints to one Open WebUI by default. Optional document retrieval, memory, web search, and an authenticated application API extend the service.

INFERENCE USES A MODEL THAT ALREADY EXISTS

A language model has learned patterns during training. Inference applies those learned weights to a prompt and generates an answer one token at a time. deployai prepares inference; it does not train or fine tune a model.

A QUESTION BECOMES A RESPONSE

CHOOSE THE INFORMATION SOURCE FOR THE TASK

NeedApproachWhat it changes
General writing or explanationModel chatUses the trained model and current conversation.
Answers from internal documentsRAG with a selected KBAdds retrieved source text to the prompt.
External web informationWeb search through SearXNGAdds external search results and fetched content where supported.
User preferences across chatsPersistent memoryMakes saved user information available to later conversations.

Quality expectation A fluent answer can still be wrong. Retrieval improves access to relevant evidence, but model choice, document quality, retrieval settings, and the question all affect the result.

AI TERMS IN PRACTICAL LANGUAGE

TermMeaning for the operator
LLM and parametersA large language model generates text. Its parameters are learned numeric weights. Parameter count is a model characteristic, not a user limit.
Token and promptA token is a unit of text, often part of a word. A prompt contains the instructions and information presented to the model.
Context window and KV cacheContext is the token budget available for input and generation. The KV cache stores intermediate attention data; larger contexts consume additional memory.
Checkpoint and GGUFA checkpoint contains model weights. GGUF is the model file format used by the main llama.cpp workflow.
Quantization and Q4_K_MQuantization reduces weight precision to reduce storage and memory use. Q4_K_M is the quantization label used by the bundled GGUF profiles; quality and speed still require evaluation.
Dense model and MoEA dense model uses its full parameter network for each token. A mixture of experts model activates selected experts. Fewer active parameters do not mean only those weights need storage.
CPU GPU and VRAMThe CPU uses system RAM. A GPU uses its device memory, or VRAM. Offload moves supported inference layers to a GPU; context and workspace also need memory.
CUDA HIP and ROCmCUDA is the NVIDIA compute backend. HIP is the AMD backend built against the ROCm software environment.
Embedding and vector indexAn embedding is a numeric representation of content. A vector index finds related passages for retrieval.
RAG and Knowledge BaseRetrieval augmented generation combines retrieved text with the prompt. A Knowledge Base is an indexed document collection in Open WebUI.
Tokens per secondA throughput measure. Prompt processing speed and output generation speed are different measurements.
API TLS OIDC and LDAPAn API lets applications call a service. TLS encrypts the connection. OIDC and LDAP integrate browser sign in with an external identity service.
RPC and coordinatorRPC sends inference work to a process on another host. The coordinator runs the front end inference server and assembles the response.

ARCHITECTURE

HOW REQUESTS MOVE THROUGH THE STACK

Browser workflow

Browser user
HTTPS chat
→Open WebUI
Accounts, Knowledge Bases, memory and tools
→Local data
Chats, uploaded documents and memory

Private model inference

Open WebUI
Connects to the selected model
→llama.cpp
CPU, CUDA or HIP inference
→GGUF checkpoint
Model weights on disk

Application workflow

Application
HTTPS request with API key
→Optional Go gateway
TLS, authorization and rate limiting
→llama.cpp
Direct model inference

External search when enabled

Open WebUI
Requests web search
→SearXNG
Queries external search engines
→Search results
Returned to the WebUI workflow

Figure 1 The browser and application paths remain distinct. This single-model view shows one selection; in shared mode Open WebUI can route chats to several independently served llama.cpp model endpoints. The gateway forwards directly to its selected inference service.

BROWSER AND APPLICATION BEHAVIOR DIFFER

Open WebUI supplies the user experience, accounts, selected-model routing, document retrieval, and memory features. Shared mode registers multiple local OpenAI-compatible inference APIs into one model selector. The optional Go gateway adds TLS, a bearer API key, and rate limiting in front of selected llama.cpp endpoints; it does not automatically run WebUI retrieval, memory, or search tools.

Web traffic SearXNG queries external search services when web search is used. Model inference remains local, but a web search workflow can send query terms outside the host. Turn web search off where that behavior is unsuitable.

WHAT RUNS ON THE HOST

Administration and service management

Administrator
CLI and health checks
→deployai
Deploy, serve and maintain
→systemd
Starts, supervises and schedules services

Managed workloads and operational data

Managed workloadRun locationAssociated data or action
llama.cpp and optional Go gatewayHost processesGGUF checkpoints in model storage
Open WebUI and SearXNGSelected Docker or Podman runtimeWebUI database, uploads and application state
RAG synchronizationsystemd one-shot service and timerOperator-scheduled knowledge ingestion
Backup operationExplicit operator commandBackup files; not automatically scheduled

Figure 2 deployai configures host processes, Docker or Podman containers, and systemd jobs. In default shared mode one Open WebUI container and database serve multiple active local model processes. Backups are an explicit operator action.

ComponentRuns asPrimary responsibility
llama.cppHost systemd serviceLoads the selected GGUF and serves inference.
Open WebUI and SearXNGContainers supervised by systemdUser access and web search.
Gateway and scheduled syncGo executable and systemd unitsOptional application access and knowledge ingestion.

Topology boundary The tool does not install Kubernetes, a workload scheduler, a load balancer, or an HA browser application. Independent hosts and distributed CPU RPC solve different needs; neither adds automatic failover.

PLANNING

PREPARE THE HOST AND NETWORK

Choose the host role and storage before downloading a model. For a predictable first run, use one Linux x86 64 host, one model, a verified storage path, and an explicit CPU or GPU selection.

AreaRequired preparation
AdministrationUse a root shell for installation and service work. The host needs systemd, working DNS, and the package tools used by the selected Enterprise Linux workflow.
Storage rootCreate an existing real absolute directory. A symlink is rejected. Single host storage may be local; multi host execution requires a genuinely shared filesystem at the same path.
Memory and CPUAllow for weights, context cache, inference workspace, the OS, Open WebUI, indexing, and other host workloads. Checkpoint size is not the RAM or VRAM requirement.
DownloadsInitial installation may reach GitHub, Hugging Face, Python packages, the chosen OCI registries, and selected GPU or OS repositories. Embedding models may download on first use.
ContainersDocker and Podman can coexist. Choose the runtime that will run the AI containers. Installation never deliberately removes either runtime or Ceph-managed containers. --container-install selected installs the chosen runtime if needed; off uses an existing installation; both requests both families and may fail on package conflicts.
GPU ownershipConfirm the GPU belongs to the inference environment. VFIO passthrough, vGPU host managers, and active AMD SR IOV virtual functions can block host driver changes.
Installation rootGPU preparation resolves /stratistor, /stratiserv, or /baseos. Set --install-root when several roots exist or the correct root is ambiguous.

DEFAULT LISTENERS FOR THE WORKED QWEN PROFILE

ServiceDefault addressWho should reach it
Open WebUIHTTPS 0.0.0.0:18103Approved browser users on a restricted network.
llama.cpp and metricsHTTP 127.0.0.1:8103Local components and protected monitoring access.
SearXNGHTTP 127.0.0.1:18888Local search integration.
Optional gatewayHTTPS 127.0.0.1:18104Local applications until explicitly published.

Firewall Plan and apply network access separately. The tool’s bind settings do not constitute a firewall policy. Keep the private inference API and distributed RPC traffic restricted to their intended peers.

CHOOSE A MODEL AND SIZE RESOURCES

Use list-models for the packaged profiles. Start with a smaller model to establish deployment and response quality, then compare larger candidates with representative questions and an identical benchmark procedure.

Model IDApproximate fileAPI portUI port
qwen3-8b5.0 GB810118101
qwen3-14b9.0 GB810218102
qwen3-30b-a3b18.6 GB810318103
qwen3-32b19.8 GB810418104
llama3.1-8b4.9 GB810518105
gemma3-12b7.3 GB810618106
deepseek-r1-14b9.0 GB810718107
custom-ggufDepends on file819918199

Sizes are approximate profile labels. In default shared mode, an additional model keeps its own inference port but reuses the existing primary WebUI port and database; the listed UI ports are only per-model defaults or the first-model shared owner. Kimi K3 remains a separate approximately 1.56 TB CPU engine without this HTTP/WebUI workflow.

deployai list-models
deployai gpu-status
deployai plan --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --resource-profile auto

WHAT AUTOMATIC SIZING DOES

For deploy, serve, and preflight, auto sizing normally chooses physical core threads; shared hosts receive roughly two thirds of physical cores. It chooses context 2048 below 24 GiB RAM, 4096 from 24 GiB, and 8192 from 64 GiB. Explicit --threads and --ctx-size take precedence. These are heuristics rather than a model fit guarantee.

Shared host protection Auto isolation detects active Ceph and virtualization services and applies systemd controls plus container limits. Review the actual limits and measure storage or VM latency under load. A dedicated profile does not reserve the host for AI.

Port planning The optional gateway default port 18104 can collide with qwen3-32b per-model WebUI port. Each active model needs a distinct inference API port. Only the selected primary WebUI port is exposed in shared mode. A custom-gguf service has one fixed profile identity and port, so use one custom GGUF at a time unless extended.

QUICK START

INSTALL AND STAGE THE FIRST MODEL

This walkthrough uses CPU inference with Qwen3 30B A3B and Docker. Specify model and node count because raw defaults are kimi-k3 and three hosts. Use a smaller model if the host has less capacity. The first served model owns the shared WebUI by default; later models reuse it.

1 BUILD OR INSTALL THE EXECUTABLE

The attached source package contains go.mod and the Go source. Build it with Go 1.26.8 or a compatible newer toolchain. Build on a suitable build host if the serving host does not have Go. The source archive has no Makefile or prebuilt bin directory.

# From the extracted source directory
go build -trimpath -o ./deployai .
install -m 0755 ./deployai /usr/local/bin/deployai
deployai --help

2 CREATE AND INSPECT THE MODEL STORAGE PATH

install -d -m 0750 /ai-models
df -h /ai-models
findmnt -T /ai-models

Use an existing mounted data filesystem where appropriate. Creating a directory does not mount or provision storage. The checkpoint downloader checks free capacity against the resolved file size plus an 8 GiB margin; backups and future revisions need additional space.

3 CHECK READINESS AND DEPLOY

deployai preflight --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu none --gpu-install off

deployai deploy --nodes 1 --model qwen3-30b-a3b
--storage-root /ai-models --gpu none --gpu-install off

Expected result deploy prepares llama.cpp binaries, downloads a GGUF from a pinned Hugging Face revision, and validates its integrity by default. With --model-verify off it checks the pinned revision and expected size but skips the full checksum. The CLI smoke test runs unless disabled. deploy stages files; serve starts the API and browser services.

START SERVICES AND SIGN IN

4 SERVE THE MODEL AND BROWSER INTERFACE

deployai serve --nodes 1 --model qwen3-30b-a3b \
  --mode independent --storage-root /ai-models \
  --gpu none --gpu-install off \
  --container-runtime docker --container-install selected \
  --listen-address 127.0.0.1 --webui-bind 0.0.0.0 \
  --webui-tls on --web-ui on --web-search on \
  --rag on --memory on --metrics on

Use --container-runtime podman to select Podman; on a system where the selected runtime is already installed, use --container-install off. Docker and Podman may coexist safely. serve requires the checkpoint and matching backend binaries from deploy and then reconciles inference, search, the shared Open WebUI, and requested optional services.

5 VERIFY SERVICE HEALTH

deployai status --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models
deployai doctor --full
curl -fsS http://127.0.0.1:8103/health
curl --cacert \
  /etc/stratistor-ai/open-webui/qwen3-30b-a3b/tls/server.crt \
  https://127.0.0.1:18103/health

With one recorded deployment, doctor loads its saved configuration. status should be given an explicit model and node count. An inactive distributed or RPC unit is normal for an independent deployment; interpret status entries against the selected topology.

6 OPEN THE BROWSER INTERFACE

Browse to https://SERVER-IP:18103 for a first Qwen3 30B deployment, or to the pinned primary WebUI port for a multi-model installation. Trust the initial self-signed certificate through your approved procedure. Read the root-protected primary instance credential file; subsequent models reuse that account and do not create a second login.

cat /etc/stratistor-ai/open-webui/qwen3-30b-a3b/admin-credentials

Initial account The generated administrator email is admin@stratistor.local with a random password. Keep these details private, change the initial password through the application, and create named accounts for routine access. Public signup is disabled; new account permissions require administrator attention.

MODEL INTEGRITY

CHOOSE A CHECKPOINT VERIFICATION POLICY

GGUF verification confirms that a downloaded checkpoint matches the expected Hugging Face content. Full checksum verification is the safer default; a size-only mode is available for controlled labs where fast staging is more important than cryptographic integrity.

OptionResult and practical use
--model-verify onDefault: use Hugging Face metadata and SHA-256 (when available), otherwise hf cache verify against the pinned revision.
--model-verify offObtain the pinned revision and expected size; check local size; skip full-file SHA-256 and hf cache verify.
--reverify-modelForce checking a prior checkpoint rather than accepting an existing marker; retain on for cryptographic verification.
--verify-jobs NKimi compatibility verification concurrency; not a switch to disable native GGUF hashing.

PRACTICAL OPERATOR COMMANDS

# Fast size-only GGUF staging (not cryptographic verification)
./deployai deploy --nodes 1 --model qwen3-8b \
  --storage-root /ai-models --gpu none --gpu-install off \
  --model-verify off

Return to full integrity checking for a subsequent run

./deployai deploy --nodes 1 --model qwen3-8b
--storage-root /ai-models --gpu none --gpu-install off
--model-verify on --reverify-model

Receipt behavior Full checking records .gguf-verified; size-only checking records .gguf-size-validated. A matching prior full-verification receipt can be reused. Off does not mean offline: metadata is still requested from Hugging Face and missing/incorrect-sized files must be downloaded.

Performance and risk Full SHA-256 needs to read the entire GGUF, which can be costly on CephFS. Size-only avoids that read, but file length alone cannot detect silent corruption or prove authenticity. Keep on for first production validation and after unexpected storage or model issues.

EVERYDAY USE

USE AND VALIDATE THE BROWSER FEATURES

A healthy service is the starting point. Test the actual user workflow before handing it to a team. UI labels and permission options can vary with the Open WebUI image you deploy.

CHAT AND CONVERSATION CONTEXT

Select the served model and ask a simple question. Follow with a related question in the same chat, then start a new chat and compare behavior. Conversation history consumes context; long chats may leave less room for retrieved documents and generated output.

DOCUMENT QUESTIONS

Create or open a Knowledge Base in Workspace or the equivalent Knowledge area. Upload a short readable document, wait for processing, and attach or select the KB in a chat. Ask a question whose answer appears only in that document. Inspect the cited passage and verify that an unauthorized account cannot access a restricted collection.

WEB RESEARCH

Enable or select the web search tool for the conversation where available. Ask for a source backed response and inspect the resulting links. An enabled server setting allows the feature; it does not prove that every model or every prompt will choose a web tool. Use --web-search off for workflows without external search.

PERSISTENT MEMORY

With memory enabled and the user permitted to use it, save a harmless preference such as a preferred response format. Review the saved memory in the application, start a new chat, and check recall. Delete the fixture afterward. Background memory review runs according to the configured turn cadence; enabled settings alone do not prove useful recall.

Acceptance testPass condition
Basic inferenceA real request produces a coherent response without service errors.
Knowledge retrievalA unique document fact is returned with the correct source.
MemoryA saved preference is available in a second authorized conversation.
PermissionsNamed users see only the collections and features they should.
Web searchA requested search returns usable external results and sources.

COMMON USE CASES

CPU AND SHARED HOST RECIPES

A SMALL SINGLE HOST PILOT

Use a smaller profile to verify installation and usability before committing a large host. Both commands must use the same model and backend.

deployai deploy --nodes 1 --model qwen3-8b \
  --storage-root /ai-models --gpu none --gpu-install off
deployai serve --nodes 1 --model qwen3-8b \
  --storage-root /ai-models --gpu none --gpu-install off \
  --container-runtime docker --container-install selected \
  --web-ui on --webui-tls on --web-search off

Verify For a fresh first deployment, open https://SERVER-IP:18101. If a shared WebUI already exists, open its retained primary URL instead and select qwen3-8b from the model list. A second model adds an inference service, not another browser database.

A PODMAN HOST SHARED WITH STORAGE OR VIRTUALIZATION

Keep existing Docker or Podman installations intact and choose the one intended for AI containers. Use --container-install off when your selected runtime is already present. Force the shared resource profile if service detection does not capture the host role; the following serve recipe assumes the checkpoint and binaries are staged.

deployai serve --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu none --gpu-install off \
  --container-runtime podman --container-install off \
  --resource-profile shared --resource-isolation on \
  --threads 24 --ctx-size 8192 \
  --listen-address 127.0.0.1 --webui-bind 0.0.0.0 \
  --webui-tls on --web-ui on --web-search on \
  --rag on --memory on

Verify Confirm the proposed context fits the host, inspect service resource properties, and compare storage or VM latency before and during inference. The 24 thread and 8192 context values are examples, not universal sizing targets.

A SERVICE FOR LOCAL APPLICATIONS ONLY

For a fresh inference service without Open WebUI, use --web-ui off --web-search off --rag off --memory off --api-gateway off on serve. This leaves the local llama.cpp API. For the tool’s authenticated TLS gateway, keep the TLS enabled WebUI configuration required by the implementation.

NVIDIA AND AMD ACCELERATION RECIPES

GPU acceleration applies to independent GGUF deployments. It requires compatible hardware and a working compute stack. The same --gpu value must be used when building, staging, and serving the model.

NVIDIA WITH A PREPARED CUDA ENVIRONMENT

deployai preflight --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu nvidia --gpu-install off
deployai deploy --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu nvidia --gpu-install off
deployai serve --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu nvidia --gpu-install off \
  --container-runtime podman --container-install off \
  --gpu-layers -1 --webui-tls on

AMD WITH A PREPARED ROCM AND HIP ENVIRONMENT

deployai preflight --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu amd --gpu-install off
deployai deploy --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu amd --gpu-install off
deployai serve --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu amd --gpu-install off \
  --container-runtime podman --container-install off \
  --gpu-layers -1 --webui-tls on

Use your installed runtime; these examples assume Podman is already present. Auto offload requests the backend’s maximum layer offload, represented as 999. It does not calculate a guaranteed VRAM fit. Reduce --gpu-layers or context when required and inspect the inference journal to confirm actual offload.

When prerequisites are missing deploy with --gpu-install on may install permitted prerequisites. A newly installed kernel driver stops the run and requires a reboot and verification before rerun. On a custom kernel, missing AMDGPU is refused unless the supported kernel path is satisfied. Resolve passthrough, vGPU host, MOK, and signing conflicts through the platform owner; do not bypass them.

STAGE A CUSTOM GGUF MODEL

Choose custom-gguf when a required model is not in the packaged list. You need a Hugging Face repository and one compatible GGUF file at its root. The CLI validates a single basename, so nested paths and a set of split GGUF shards are outside this workflow.

CHECK THE SOURCE BEFORE DEPLOYING

DEPLOY AND SERVE THE SELECTED FILE

Replace ORG/REPO and Model-Q4_K_M.gguf below with real values. Include the same custom source options on both commands and on explicit operational calls that require model lookup.

deployai deploy --nodes 1 --model custom-gguf \
  --hf-repo ORG/REPO --hf-file Model-Q4_K_M.gguf \
  --storage-root /ai-models --gpu none --gpu-install off
deployai serve --nodes 1 --model custom-gguf \
  --hf-repo ORG/REPO --hf-file Model-Q4_K_M.gguf \
  --storage-root /ai-models --gpu none --gpu-install off \
  --container-runtime podman --container-install off \
  --webui-tls on --web-search off

Expected result The custom profile uses private API port 8199. On a fresh system its per-model default browser port is 18199; in shared mode it registers with the existing primary WebUI instead. Its checkpoint directory includes a repository/file hash. Verify the served model with a real inference request.

RUN SEVERAL BUILT IN PROFILES DELIBERATELY

Built-in profiles have distinct inference ports and service identities. In shared mode, active locally served profiles appear in one primary WebUI model selector and share its chats, credentials, and RAG state. Each model still consumes its own CPU, RAM or VRAM. The custom-gguf profile shares one service name and port 8199 across sources; different hashed checkpoint directories do not create simultaneous custom endpoints.

CONTAINER OPERATIONS

RUN DOCKER AND PODMAN SAFELY TOGETHER

Docker Engine and Podman can be installed side by side on a StratiSYSTEM host. Ceph may depend on Podman, so deployai never intentionally removes Podman, Docker, runc, or containerd to make the selected runtime install succeed. Containers already managed by Ceph are outside the AI deployment scope.

Flag or conditionExpected behavior
--container-runtime dockerUse Docker for managed Open WebUI and SearXNG containers.
--container-runtime podmanUse Podman for managed AI containers.
--container-install offDo not install a runtime; require the selected one to be present and functional.
--container-install selectedAttempt to install only the chosen runtime, preserving existing runtime packages.
--container-install bothAttempt installation of both families, without removing either one.
DNF conflict / podman-docker aliasFail safely with a useful error rather than using forced package erasure; resolve real package conflicts separately.
# Docker already installed: select it, skip package changes
--container-runtime docker --container-install off

Docker not installed: request Docker installation only

--container-runtime docker --container-install selected

Confirm both engines and protect Ceph services

docker info >/dev/null && podman info >/dev/null
systemctl status ceph.target --no-pager

Caution A DNF dependency conflict is a legitimate failure, not permission to remove Podman or Ceph components. Do not use --allowerasing as a shortcut. If podman-docker supplies the docker CLI alias, installation may require explicit administrator remediation. AI runtime selection does not change Ceph deployment.

UNIFIED WEBUI

ONE INTERFACE WITH SELECTABLE MODELS

Shared mode is now the default. Each GGUF model runs in its own llama-server systemd service with a distinct private API port. A single Open WebUI container connects to the active local inference APIs and presents those models in one dropdown. Chats, login accounts, RAG Knowledge Bases, memory, TLS, and settings belong to the primary WebUI.

One browser interface

Browser users
One HTTPS sign-in
→Primary Open WebUI
One model selector, chats, RAG and memory

Active local inference connections

Selectable modelLocal endpointOperation
Llama 3.1 8BHTTP API port 8105Separate llama.cpp service
Qwen3 0.6B (custom GGUF)HTTP API port 8199Separate llama.cpp service
Other supported GGUF modelIts own available local API portSeparate model service identity

All models share the primary WebUI database and accounts; their inference services and memory consumption remain separate.

Figure 5 One Open WebUI selects among separate local inference services. Model weights may reside on shared CephFS; the browser database remains on the coordinator and is not automatically clustered.

ConfigurationMeaning
--webui-mode sharedDefault: reuse or create one primary WebUI and manage its local OpenAI-compatible connections.
--webui-mode per-modelLegacy behavior: create a separate WebUI for each model.
--webui-primary-model llama3.1-8bOn first shared-mode serve, pin an already deployed Llama WebUI as primary.
shared-owner marker/etc/stratistor-ai/open-webui/shared-owner records the selected primary identity.

Migration An existing active WebUI is preferred. Primary credentials, TLS material, and database are retained; secondary legacy WebUI databases are not merged, deleted, or automatically stopped. After the primary owner is pinned, switching it is refused rather than risking a silent data replacement.

ADD A SMALL SECOND MODEL TO YOUR EXISTING UI

Example environment: Llama 3.1 8B is already served across stratistor1-1 through stratistor1-3, and its shared WebUI is https://stratistor1-1:18105. For a lightweight test, stage one custom Qwen3 0.6B Q4_K_M checkpoint on the coordinator only. Model selection is independent of whether Llama uses distributed RPC.

1 STAGE THE SECOND MODEL (DO NOT REDEPLOY LLAMA)

./deployai deploy \
  --nodes 1 --model custom-gguf \
  --hf-repo unsloth/Qwen3-0.6B-GGUF \
  --hf-file Qwen3-0.6B-Q4_K_M.gguf \
  --mode independent \
  --storage-root /stratistor/clustermounts/machines \
  --install-root /stratistor --gpu none --gpu-install off \
  --threads 4 --ctx-size 4096 --build-jobs 4 \
  --resource-profile shared --resource-isolation auto

2 SERVE IT AND REGISTER WITH THE PINNED WEBUI

./deployai serve \
  --nodes 1 --model custom-gguf \
  --hf-repo unsloth/Qwen3-0.6B-GGUF \
  --hf-file Qwen3-0.6B-Q4_K_M.gguf \
  --mode independent \
  --storage-root /stratistor/clustermounts/machines \
  --install-root /stratistor --gpu none --gpu-install off \
  --threads 4 --ctx-size 4096 \
  --resource-profile shared --resource-isolation auto \
  --container-runtime docker --container-install off \
  --web-ui on --webui-mode shared \
  --webui-primary-model llama3.1-8b \
  --web-search on --rag on --memory on --webui-tls on

3 Open https://stratistor1-1:18105 and select the second model. The existing Llama inference API remains on 8105; the custom model serves on 8199. Model labels may be prefixed by their deployai service identity. The primary WebUI may restart briefly when its connection list changes.

VERIFY AND OPERATE SHARED-MODEL SELECTION

CHECK THE OWNER, SERVICES, AND REGISTERED ENDPOINTS

cat /etc/stratistor-ai/open-webui/shared-owner
systemctl is-active stratistor-open-webui-llama3.1-8b.service
systemctl is-active stratistor-llama-custom-gguf.service
# For distributed Llama, the coordinator service is stratistor-dist-llama3.1-8b
ss -lntp | grep -E ":(8105|8199|18105)\b"
grep ^OPENAI_API_BASE_URLS= \
 /etc/stratistor-ai/open-webui/llama3.1-8b/environment

The expected connection list contains the active local model APIs (for example http://127.0.0.1:8105/v1 and http://127.0.0.1:8199/v1). Only active local systemd inference services are discovered; it does not automatically combine independent models running exclusively on other hosts.

SAFE LIFECYCLE PRACTICES

OperationWhat to expect
Serve a new modelRegisters the new active local API; primary WebUI settings are reconciled and the UI may restart.
Stop one secondary modelStops that model; shared WebUI remains available for other models.
Backup/restoreRun under the primary model identity to capture shared UI state consistently.
RAG or admin credentialsUse the primary WebUI endpoint and existing root-protected credentials.
Legacy per-model UIOld secondary containers and data remain intact until an intentional migration/retirement.

Security The managed connection list is written into the primary environment file (including OPENAI_API_BASE_URLS). Back up any hand-added OpenAI connections before enabling shared mode, because deployai manages those values. API endpoints should remain loopback-only or reachable only through appropriate private controls.

SPECIALIZED DEPLOYMENT

USE THE SPECIALIZED KIMI ENGINE

The kimi-k3 profile follows a separate CPU C engine compatibility workflow with an approximately 1.56 TB checkpoint. It is intended for a carefully planned high capacity environment. It does not supply the normal llama.cpp HTTP server, Open WebUI, RAG, or gateway workflow.

PLAN STORAGE AND MEMORY BEFORE STAGING

Review the compatibility engine requirements and the memory preset that fits the host. The preset changes the engine’s working memory behavior; it does not make the complete checkpoint download disappear. Leave capacity for verification, staging, and the other host workloads. Use the standard GGUF workflow when a browser chat service is the goal.

deployai preflight --nodes 1 --model kimi-k3 \
  --storage-root /ai-models --gpu none
deployai deploy --nodes 1 --model kimi-k3 \
  --storage-root /ai-models --gpu none --preset auto
deployai test --nodes 1 --model kimi-k3 \
  --storage-root /ai-models --gpu none --preset auto
deployai status --nodes 1 --model kimi-k3 \
  --storage-root /ai-models --gpu none

The compatibility worker preserves the specialized download and integrity pipeline. The Go preflight checks host and storage before handing the action to that worker; it is not a substitute for checking the full engine capacity requirements.

Setting or commandMeaning for Kimi
--presetauto, server, max, workstation, desktop, laptop, or ultra. The compatibility engine evaluates the selected memory policy.
--verify-jobs and --download-workersPassed to the compatibility worker. These controls do not tune the native single GGUF download loop.
serve and stopRejected for this profile by the Go parser. Use the engine specific operational workflow.
Distributed mode or GPU flagsDistributed mode and explicit NVIDIA or AMD inference are rejected. Kimi uses independent CPU execution.
Generic update and backupsDo not treat the GGUF updater or operator state archive as a full Kimi checkpoint lifecycle. Retain its model material and recovery procedure separately.

MULTIPLE HOSTS

CHOOSE THE RIGHT MULTI HOST TOPOLOGY

TopologyBehaviorChoose it when
Independent single hostOne model server uses one host. UI and optional integrations are local.You need the simplest pilot or a dedicated team endpoint.
Independent two or three hostsEach host runs a separate model service. The tool fans out operations over SSH and verifies shared model storage.You need separate endpoints on several hosts and can manage their access individually.
Distributed CPU RPCOne coordinator splits inference work among local and remote CPU RPC workers.You are evaluating a model across combined host resources and can accept the added network and operational requirements.

PREPARE MULTI HOST ACCESS

INDEPENDENT HOST EXAMPLE

deployai deploy --nodes ai01,ai02 --mode independent \
  --model qwen3-8b --storage-root /shared/models \
  --gpu none --gpu-install off
deployai serve --nodes ai01,ai02 --mode independent \
  --model qwen3-8b --storage-root /shared/models \
  --gpu none --gpu-install off \
  --container-runtime podman --container-install off

Advanced settings For custom TLS, OIDC, or LDAP on independent remote nodes, configure and validate each host locally. The multi host SSH path does not forward every optional security setting. Independent endpoints have separate local UI state; they do not become an HA application.

DEPLOY DISTRIBUTED CPU INFERENCE

Distributed request path

Browser or API client
Connects to coordinator
→Coordinator
Open WebUI and llama-server
→CPU RPC workers
Collaborative inference across nodes

CPU RPC and model storage placement

ComponentLocationNetwork or storage requirement
Local CPU RPCCoordinator hostLoopback address 127.0.0.1
CPU RPC worker 2Second hostRestricted private IPv4 connection
CPU RPC worker 3 (optional)Third hostRestricted private IPv4 connection
Shared GGUF storageMounted on every participating hostSame absolute mounted storage path

Figure 3 A distributed deployment has one coordinator and two or three CPU RPC processes in total. The shared storage path holds the model; it does not turn the WebUI into a clustered database.

Run from the first listed host, ai01. Here ai02 uses 10.10.25.12 and ai03 uses 10.10.25.13. The default RPC port for qwen3-30b-a3b is 48103, calculated as 40000 plus the API port.

deployai deploy --nodes ai01,ai02,ai03 --mode distributed \
  --model qwen3-30b-a3b --storage-root /shared/models \
  --gpu none --rpc-address-2 10.10.25.12 \
  --rpc-address-3 10.10.25.13
deployai serve --nodes ai01,ai02,ai03 --mode distributed \
  --model qwen3-30b-a3b --storage-root /shared/models \
  --gpu none --rpc-address-2 10.10.25.12 \
  --rpc-address-3 10.10.25.13 \
  --container-runtime podman --container-install off

Operate Validate a real coordinator API request. The standalone distributed test command is not implemented; generic update keeps the distributed backend binaries. Stop and inspect services locally on each host. Multi host distributed status and stop are not complete cluster operations in this release. Benchmark before expecting a speed benefit.

RAG WORKFLOWS

UNDERSTAND KNOWLEDGE INGESTION AND RETRIEVAL

Document ingestion — incremental adds/updates; mirror may delete source-absent documents

Source folder
Local regular files
→RAG sync client
Authenticated Open WebUI API
→Knowledge Base
Extraction, indexing and embeddings

Question answering — retrieval does not train the model

User question
Selects a Knowledge Base
→KB retrieval
Finds relevant passages
→llama.cpp
Receives question and retrieved context
→Response
User verifies citations and content

Figure 4 Sync prepares the Knowledge Base. Retrieval selects relevant passages at question time. The model checkpoint is unchanged by both operations.

PREPARE A USEFUL SOURCE COLLECTION

Use documents with readable text, clear titles, current content, and a deliberate access policy. Begin with a small collection of Markdown or plain text files. Supported document extraction depends on the deployed Open WebUI image; a file uploaded successfully still needs successful indexing and a retrieval test.

Scope --rag on configures document retrieval in the UI. It does not select a source folder, create a KB, or schedule ingestion. The rag-sync command performs one synchronization; --rag-sync on on serve creates a timer for repeated runs.

CREATE A KNOWLEDGE BASE AND RUN SYNC

1 CREATE THE COLLECTION AND CREDENTIAL

In the primary shared Open WebUI, create a Knowledge Base and copy its ID from the collection URL or details. Give the syncing account permission to the collection. If account API keys are unavailable, rerun the complete working primary serve configuration with --webui-api-keys on; then check administrator permissions before generating the account key. An inference gateway key cannot authenticate this operation.

install -d -m 0700 /etc/stratistor-ai
install -d -m 0750 /ai-models/company-docs
umask 077
read -r -s -p 'Open WebUI account API key: ' rag_api_key
printf '\n'
printf '%s\n' "$rag_api_key" > /etc/stratistor-ai/rag-sync-key
unset rag_api_key
chmod 0600 /etc/stratistor-ai/rag-sync-key

Place the approved source documents in /ai-models/company-docs. Replace YOUR-KB-ID below with the real collection ID. Keep this nonsecret value available for the scheduled workflow.

2 PREVIEW THE INCREMENTAL DIFFERENCE

deployai rag-sync --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models \
  --rag-source /ai-models/company-docs \
  --rag-knowledge-id YOUR-KB-ID \
  --rag-token-file /etc/stratistor-ai/rag-sync-key \
  --rag-sync-mode incremental --webui-tls on --dry-run

3 APPLY AND VERIFY

Repeat the command without --dry-run. Incremental sync reports added, updated, unchanged, and source absent counts. It uploads changed files and waits for indexing before cleaning the superseded revisions of those changed files. Attach the KB to a chat and verify a unique fact with its source.

Authentication behavior A real one shot sync can enable Open WebUI API key support and restart Open WebUI if required. A dry run does not perform that settings change; a reported authentication failure means the preview did not establish a remote diff. The key must belong to an authorized account and be stored in a protected regular file.

CHOOSE INCREMENTAL OR MIRROR SYNC

Source or collection eventIncremental defaultMirror opt in
New source fileUploads and indexes.Uploads and indexes.
Changed source fileIndexes the replacement, then cleans its superseded revision.Reconciles the changed source content.
Unchanged fileSkips unchanged content.Reconciles according to the sync client.
File absent from sourceKeeps the indexed KB document.Can delete the corresponding KB document.
Separately managed KB contentPreserves source absent content.Can reconcile it away if absent from the source.
ImplementationNative Go checksum diff and upload.Isolated oikb Python client.

INCREMENTAL INPUT RULES

WHEN MIRROR IS APPROPRIATE

Use a dedicated KB whose entire document lifecycle is owned by the source folder. Take an operator backup before the first destructive reconciliation and test additions, updates, and removals in a disposable collection. Select --rag-sync-mode mirror explicitly only after confirming the source mount and contents.

Legacy flags --rag-sync-prune on selects mirror unless --rag-sync-mode is explicitly supplied. Existing saved records that lack a mode can retain the older mirror behavior. Always state the intended mode when changing a scheduled configuration.

Preview limit Incremental dry run attempts a real remote diff and can fall back to local inventory with a warning. In this source version, mirror dry run logs the intended oikb command without executing a full remote preview. Treat it as a command preview, not proof of which documents would be deleted.

SCHEDULE AND OPERATE KNOWLEDGE SYNC

After a successful one shot sync, add scheduling to the complete working serve configuration. This example keeps Podman, CPU inference, and the standard UI features. Preserve any gateway, identity, TLS, and sizing options from your deployment record.

deployai serve --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu none --gpu-install off \
  --container-runtime podman --container-install off \
  --listen-address 127.0.0.1 --webui-bind 0.0.0.0 \
  --webui-tls on --web-ui on --web-search on \
  --rag on --memory on --rag-sync on \
  --rag-source /ai-models/company-docs \
  --rag-knowledge-id YOUR-KB-ID \
  --rag-token-file /etc/stratistor-ai/rag-sync-key \
  --rag-sync-mode incremental --rag-sync-minutes 30

CONFIRM THE JOB AND FIRST UNATTENDED RUN

systemctl list-timers --all 'stratistor-ai-ragsync-*'
systemctl start stratistor-ai-ragsync-qwen3-30b-a3b.service
journalctl -u stratistor-ai-ragsync-qwen3-30b-a3b.service \
  -n 60 --no-pager
systemctl cat stratistor-ai-ragsync-qwen3-30b-a3b.timer

The timer uses OnBootSec=5min and OnUnitInactiveSec=30min for this recipe. The interval is relative to completion of the previous run, not a wall clock every half hour. The job is a oneshot service; inactive after successful completion is normal. Check its last result and journal, not only whether it is currently active.

PAUSE AND CHANGE SCHEDULING

# Pause now; this does not change the persisted deployment options
systemctl stop stratistor-ai-ragsync-qwen3-30b-a3b.timer

For a durable configuration change, rerun the full serve command with --rag-sync off or with the revised interval or mode. Confirm the timer’s enabled state and next activation afterward. Stop an already running sync separately or wait for it to finish before maintenance; stopping a timer alone does not cancel its active service.

Avoid collisions Do not intentionally overlap manual and scheduled sync runs. Keep the source filesystem mounted and readable, monitor credential rotation, and verify retrieval after document changes. See the release behavior page for the incremental doctor client check.

APPLICATION ACCESS

EXPOSE AN AUTHENTICATED APPLICATION API

Enable the gateway when software needs a controlled inference endpoint. The default bind is loopback. For remote clients, choose an approved host address, a certificate covering the name clients use, and a separate network access policy.

deployai serve --nodes 1 --model qwen3-30b-a3b \
  --storage-root /ai-models --gpu none --gpu-install off \
  --container-runtime podman --container-install off \
  --web-ui on --webui-tls on \
  --api-gateway on --gateway-bind 127.0.0.1 \
  --gateway-port 18104 --gateway-rpm 60

The tool creates a persistent random key unless --gateway-key-file points to an existing protected key. It reuses the managed WebUI certificate and private key. A supplied gateway key must be at least 32 characters; protect it with mode 0600.

EndpointMethodBehavior
/healthGETGateway liveness without a key. Does not prove inference health.
/v1/modelsGETAuthenticated model discovery.
/v1/chat/completionsPOSTAuthenticated chat completion proxy.
/v1/completionsPOSTAuthenticated completion proxy.
/v1/embeddingsPOSTProxy only; the selected model and backend must support the requested embedding behavior.

Limits Rate limiting is per source IP in one minute windows. Clients behind one NAT or proxy may share a limit. The handler caps requests at 4 MiB, rejects unlisted routes or methods, and returns 401 for a missing key, 403 for an invalid key, and 429 when rate limited.

RAG applications This gateway reaches llama.cpp directly. An application requiring Open WebUI Knowledge Base retrieval or user memory needs a separately designed authenticated Open WebUI integration. The gateway key and WebUI account API key are not interchangeable.

TEST API ACCESS AND CONFIGURE HTTPS

A LOCAL AUTHENTICATED REQUEST

The following Bash example loads the key into a temporary curl configuration through process substitution, keeping the expanded bearer value out of the curl argument list. Do not enable shell tracing while handling credentials.

api_key=$(cat /etc/stratistor-ai/api-gateway/qwen3-30b-a3b/api-key)
curl --cacert \
  /etc/stratistor-ai/open-webui/qwen3-30b-a3b/tls/server.crt \
  --config <(printf 'header = "Authorization: Bearer %s"\n' "$api_key") \
  https://127.0.0.1:18104/v1/models
unset api_key

Use the returned model ID for a completion request. First prove authenticated model discovery, then submit a short /v1/chat/completions request from the actual client environment. A successful /health response alone only confirms the gateway listener.

SELF SIGNED AND ORGANIZATION ISSUED CERTIFICATES

HTTPS is provided directly by Open WebUI’s Uvicorn server; no extra reverse proxy is installed. Default generated certificates contain localhost, hostnames, local addresses, and requested SANs. Trust the certificate on the client and use a matching DNS name or address.

# Add to the complete working serve command for a self signed identity
--webui-tls on --webui-tls-san ai.example.com,192.168.59.101

Or supply a real matching PEM pair

--webui-tls on --webui-tls-cert /etc/pki/ai/fullchain.pem
--webui-tls-key /etc/pki/ai/private-key.pem

The supplied files must be regular files and form a valid current certificate and key pair. They are copied into the private managed TLS directory. If local RAG sync uses HTTPS, the certificate also needs to support the loopback name or address used by that client, normally 127.0.0.1. Confirm both browser and sync trust after replacement.

Renewal Generated certificates are checked during reconciliation and renewed when near expiry or their identity set changes. There is no separate renewal daemon. Keep running your planned maintenance process; renew supplied certificates through your organization’s certificate process and reconcile the full deployment settings.

Network rule Leave --listen-address 127.0.0.1 in place. Managed health checks and the gateway upstream expect the local inference address. Publish only the approved HTTPS listener and restrict source networks.

IDENTITY

INTEGRATE COMPANY IDENTITY

Local accounts are adequate for a pilot. Use OIDC or LDAP when the team needs centralized sign in. Prepare the external service, trust chain, account approval process, and a tested local administrator recovery login before changing an active deployment.

OIDC SIGN IN

Register a client with your identity provider. For the example public URL below, the documented callback is https://ai.example.com:18103/oauth/oidc/callback. Confirm the callback against the exact Open WebUI image and provider configuration you deploy. Store the client secret in a root protected file.

# Append these options to the full working serve command
--identity oidc --identity-url https://ai.example.com:18103 \
  --oidc-client-id YOUR-CLIENT-ID \
  --oidc-discovery-url https://id.example.com/.well-known/openid-configuration \
  --oidc-secret-file /etc/stratistor-ai/oidc-client-secret \
  --oidc-provider-name 'Company SSO' \
  --webui-tls on --webui-tls-san ai.example.com

LDAP OR ACTIVE DIRECTORY SIGN IN

Use an LDAPS endpoint whose server certificate is trusted by the application environment. The tool enables certificate validation. Create a directory bind identity with the necessary read permissions and keep its password in a protected file.

# Alternative identity options for the full serve command
--identity ldap --ldap-host dc.example.com --ldap-port 636 \
  --ldap-search-base 'dc=example,dc=com' \
  --ldap-bind-dn 'cn=ai-bind,ou=service,dc=example,dc=com' \
  --ldap-password-file /etc/stratistor-ai/ldap-bind-secret \
  --ldap-tls on --webui-tls on

VALIDATE AS REAL USERS

Scope The tool configures Open WebUI integration; it does not deploy an identity provider or promise automatic enterprise group mapping. New user approval and permission behavior require validation in the deployed application.

ROUTINE OPERATIONS

OBSERVE HEALTH AND THROUGHPUT

Use a simple baseline: service state, a real inference request, browser sign in, a known document question, and resource measurements. Compare later changes against that baseline.

RUN DIAGNOSTICS

# With one recorded deployment
deployai doctor --full
deployai benchmark --benchmark-runs 5 --benchmark-tokens 256
deployai metrics

Service and log checks for the worked model

systemctl status stratistor-llama-qwen3-30b-a3b --no-pager
journalctl -u stratistor-llama-qwen3-30b-a3b -n 80 --no-pager

doctor checks service state, checkpoint presence, inference health, search responses, browser health, optional gateway behavior, and configured features. --full adds a real 32 token generation. Configuration checks for RAG and memory do not prove retrieval or recall. The gateway check rejects anonymous access but does not validate your real client key.

BENCHMARK WITH A REPEATABLE METHOD

benchmark performs a warm up followed by the requested number of identical completion requests. It reports prompt processing and generation speed, elapsed time, and median, mean, minimum, and maximum output throughput. Record model revision, backend, context, threads, layer offload, resource limits, and concurrent workloads. The server is configured with one parallel slot; this test is not a multiuser capacity or latency SLA test.

COLLECT TELEMETRY

When the llama.cpp binary supports it, --metrics on enables /metrics on the same inference listener. No Prometheus or Grafana service is installed. Use an existing local collector or a protected connection; the metrics listener inherits --listen-address and is not independently secured.

curl -fsS http://127.0.0.1:8103/metrics
systemctl show stratistor-llama-qwen3-30b-a3b \
  -p CPUQuotaPerSecUSec -p MemoryHigh -p MemoryMax

MAINTAIN SERVICES WITHOUT LOSING STATE

RECORD CONFIGURATION BEFORE CHANGING IT

Save the successful serve command and the matching deployment registry entry. serve reconciles supplied values and parser defaults; it is not a general partial settings merge. Omitting --api-gateway on or --rag-sync on can stop those optional services, and omitting identity or sizing options can replace that configuration.

STOP THE SELECTED INFERENCE AND UI SERVICES

# Pause optional scheduled ingestion and gateway if enabled
systemctl stop stratistor-ai-ragsync-qwen3-30b-a3b.timer
systemctl stop stratistor-ai-gateway-qwen3-30b-a3b.service

deployai stop --nodes 1 --model qwen3-30b-a3b
--storage-root /ai-models

Only run optional unit commands when those units exist. deployai stop disables the selected model inference and RPC units, but retains the shared WebUI when --webui-mode shared is in effect. A per-model legacy UI follows the legacy stop behavior. Shared SearXNG is not automatically stopped; verify dependent services before maintenance.

RESTART A CONFIGURED INDEPENDENT DEPLOYMENT

systemctl enable --now stratistor-llama-qwen3-30b-a3b.service
systemctl enable --now stratistor-open-webui-qwen3-30b-a3b.service
deployai doctor --full

Restore optional gateway and timer services when appropriate. For an inference setting change, rerun the complete recorded serve command. A secondary model must not stop or replace the primary shared WebUI; do not restart Ceph-managed Podman or storage services as part of AI maintenance.

MULTIPLE INSTALLED MODELS

With several installed models, identify the intended inference deployment explicitly on status, stop, and other model-scoped calls. In shared mode, model endpoints are discovered from active local systemd units. Run WebUI operator-state backup/restore under the pinned primary model; a secondary model backup is refused to protect consistent UI state. Store the shared owner identity with the site record.

Keep the executable available Scheduled sync and the gateway use a stable installed helper under /usr/local/libexec/stratistor-ai. Replacing the CLI in another directory does not automatically update every managed helper; reconcile the full serve configuration when rolling out a new deployai executable.

LIVE CONFIGURATION

INCREASE CONTEXT WITHOUT REDEPLOYING WEIGHTS

The message “request (8860 tokens) exceeds the available context size (4096 tokens)” means the assembled request, including chat history and any RAG or search passages, is too large for the active inference window. For Llama 3.1 8B start with --ctx-size 16384 and monitor memory. A bigger context increases KV-cache RAM requirements.

RERUN SERVE, NOT DEPLOY, FOR THE EXISTING CLUSTER

./deployai serve \
  --nodes stratistor1-1,stratistor1-2,stratistor1-3 \
  --model llama3.1-8b --mode distributed \
  --storage-root /stratistor/clustermounts/machines \
  --install-root /stratistor --gpu none --gpu-install off \
  --threads 8 --ctx-size 16384 \
  --resource-profile shared --resource-isolation auto \
  --rpc-address-2 "$RPC2" --rpc-address-3 "$RPC3" \
  --container-runtime docker --container-install off \
  --web-ui on --webui-mode shared \
  --web-search on --rag on --memory on --webui-tls on

Use the complete previously successful serve command and change only the intended parameter; the example assumes RPC2/RPC3 contain the correct private worker IPv4 addresses. serve reconciles services in place, without rebuilding or re-downloading model weights. An inference-service restart can interrupt active chats.

CONFIRM THE SETTING APPLIED

systemctl cat stratistor-dist-llama3.1-8b.service | grep -E -- "ctx-size| -c "
ps -eo pid,args | grep "[l]lama-server"
curl -fsS http://127.0.0.1:8105/health

If the prompt remains too long Trim chat history and reduce RAG passages or retrieved documents. A 16K window is a starting recommendation, not a resource guarantee. Re-run a real request after changing the setting.

BACKUP AND RECOVERY

BACK UP OPERATOR STATE

A recoverable deployment needs two sets of assets: application state and the software or model material needed to run it. The built in backup captures operator state and intentionally excludes model weights.

Captured when presentRetain separately
Open WebUI database, chats, uploaded documents, vector index, and memory data.GGUF or Kimi checkpoints, revision and verification metadata, and any required Hugging Face access credentials.
Per model UI configuration, managed TLS material, session secrets, and deployment registry.OS, kernel, GPU stack, llama.cpp binaries or a reproducible build record, and container images or pinned image references.
Selected AI units, managed gateway configuration and key, SearXNG settings, and RAG timer definitions.External identity provider configuration, original document source folders, and externally managed secrets outside the captured paths.
Default RAG key and configured RAG key files inside /etc/stratistor-ai.Embedding model download caches, which may need to be downloaded again. Keep an offline cache copy if recovery has no internet access.

CREATE AND VALIDATE AN ARCHIVE

deployai backup
ls -lht /var/backups/stratistor-ai/

Set the exact archive path reported by backup

BACKUP=/var/backups/stratistor-ai/EXACT-operator-state-ARCHIVE.tar.gz
tar -tzf "$BACKUP" >/dev/null
sha256sum "$BACKUP"

A running RAG sync blocks backup. In shared mode perform a WebUI state backup using the pinned primary model. On an approved backup, the tool pauses the timer, quiesces WebUI for a consistent database snapshot, archives state, restarts the UI, and checks readiness. Model weights remain excluded. Use an explicit primary model when multiple deployments exist.

Retention Archives contain credentials and personal or company content. Copy them to an approved off host destination, preserve restrictive permissions, and apply the organization’s encryption and retention controls. The tool has no automatic backup schedule or retention pruning command.

VALIDATE AND EXECUTE A RESTORE

Restore writes the saved state back to managed paths and interrupts affected services. Test on an isolated target with the same model identity and expected paths before relying on it for disaster recovery. Preserve the current state and the exact archive you plan to restore.

1 VALIDATE THE ARCHIVE WITHOUT APPLYING IT

deployai restore --model qwen3-30b-a3b --nodes 1 \
  --storage-root /ai-models --restore-file "$BACKUP" --dry-run

Dry run checks archive structure, allowed paths, embedded manifest, model identity, and safe symlinks. It does not prove the database is compatible with a different image or that every external dependency is available. Match the original WebUI data root and custom RAG key path when applicable.

2 PERFORM THE PLANNED RESTORE

deployai restore --model qwen3-30b-a3b --nodes 1 \
  --storage-root /ai-models --restore-file "$BACKUP"

The tool takes a safeguard backup, records active service state, stops affected services, restores validated data, and resumes previously active dependencies. It stages the previous UI trees while extracting. A failed extraction or restart still requires operator review; the full environment is not an atomic transaction.

3 VERIFY APPLICATION AND SERVICE BEHAVIOR

New host recovery Prepare the OS, runtime, matching model weights, binaries, and storage mounts before the restore. Retain matching container image versions; application schema migrations may prevent recovery by image downgrade alone. Restore does not reinstall a complete server or reconstruct remote RPC workers.

MAINTENANCE

UPDATE THE DEPLOYED SOFTWARE

update discovers saved GGUF deployments and preserves their recorded runtime, paths, ports, and service options. Without --model it processes all discovered deployments. Plan a maintenance window for changes that restart inference or the UI.

deployai update --dry-run
deployai backup
deployai update --model qwen3-30b-a3b
deployai doctor --full
ChoiceEffect
updateChecks the configured UI and search images, updates compatible independent llama.cpp software, reconciles services, and updates an enabled mirror sync client.
--update-llama offSkips the llama.cpp build update while continuing UI and search update handling.
--update-models onExplicitly permits checkpoint revision replacement. New content is staged and verified; allow additional storage and download time.
--model IDScopes the discovered deployment selection. It does not isolate shared backend binaries or the shared search service.
--dry-runLists the deployments and intended checks. It does not fetch an exhaustive version diff or prove an upstream update will be compatible.

WHAT THE UPDATER PROTECTS AND WHAT TO VERIFY

Independent backend updates build in staging and validate capabilities before promotion; earlier binaries are retained for rollback attempts. Checkpoint updates respect --model-verify on|off. Shared WebUI state and its selected primary owner are preserved; changing registered API connections may restart the primary WebUI container. Validate inference and chat access afterward.

Open WebUI defaults to a moving main tag and SearXNG to latest. Select a tested Open WebUI image reference through --webui-image on serve if controlled versioning is required. update pulls the configured UI image without needing --update-webui; that Boolean flag controls image refresh during ordinary setup.

Separate maintenance The generic updater does not update the deployai executable, OS, custom kernel, GPU drivers, signed DKMS, DOCA, or Ceph. It keeps distributed RPC backend binaries pending a coordinated rollout. Back up each model’s state when several deployments are in scope.

TROUBLESHOOTING

TROUBLESHOOT BY THE FAILING LAYER

SymptomFirst checks and corrective action
Missing llama-server or backendRun deploy with the same model, GPU choice, and mode. Check /opt/llama.cpp backend paths and the deployment log.
GPU preflight stopsRead the exact ownership or prerequisite error. Verify device binding, vGPU host manager, /dev/kfd, CUDA or HIP, kernel build tree, MOK, and Secure Boot as relevant.
Browser cannot connectCheck the model UI port, HTTPS scheme, bind address, systemd unit, network policy, and certificate trust. Inspect the WebUI journal before restarting.
UI works but inference failsCheck the private inference /health, model file, RAM or VRAM, context, and inference journal. Keep the managed loopback address available.
Request tokens exceed contextRerun the complete existing serve command with a larger explicit --ctx-size (e.g. 16384); verify active llama-server process args, monitor KV-cache memory, and trim RAG/history.
Secondary model missing in WebUIVerify shared-owner, secondary inference systemd service, private API /health, and OPENAI_API_BASE_URLS in the primary environment. Run serve for that model to refresh local connections.
Docker install fails while Podman existsPreserve Ceph Podman/runc; resolve a genuine DNF package conflict or podman-docker alias manually. Never use forced package removals as an AI installer workaround.
RAG sync returns 401 or 403Use the real WebUI account key, check API key support and KB permission, and confirm the token file. A gateway key is the wrong credential.
RAG upload succeeds but answer misses itWait for indexing, attach the correct KB to the chat, check extraction and permissions, then use a unique document fact as the test.
Knowledge sync rejects sourceCheck visible symlinks or special files, the 50 MiB incremental limit, empty eligible inventory, source access, and the actual mount.
Gateway 401 403 429 or 502Check missing key, invalid key, shared source IP rate limit, or unavailable inference respectively. Test /v1/models with the correct key.
Runtime shared library error after updateInspect the updater log and backend library bundle. The updater validates and attempts runtime recovery; preserve the previous binary directory while investigating.
Slow inference or out of memoryReview weights, context, offload, CPU quotas, memory ceilings, indexing load, and concurrent host workloads. Benchmark one change at a time.
Distributed worker unavailableCheck private IPv4, default RPC port, firewall, exact host order, shared filesystem proof, compatible RPC builds, and local worker journals.
journalctl -u stratistor-open-webui-qwen3-30b-a3b -n 80 --no-pager
ss -lntp
ls -lt /var/log/stratistor-ai-go/

VERSION SPECIFIC BEHAVIOR TO RECOGNIZE

Use these behaviors when interpreting deployai 1.3.4 with the 9 November 2026 patches. A successful size check is not a cryptographic hash check. WebUI ownership and selected inference service are independent, and the shared interface is not a high-availability cluster.

READINESS AND DRY RUN SCOPE

preflight avoids driver or compute package installation, but the AMD path can still reconcile the stable ROCm repository file even with --gpu-install off. Use plan and gpu-status for initial inspection when repository changes must be avoided. Diagnostic and benchmark requests can contact services even when --dry-run is present; it is not a global network or inference suppression switch.

SCHEDULED INCREMENTAL SYNC AND DOCTOR

The doctor implementation checks for the oikb binary whenever scheduled RAG is enabled, including native incremental mode that does not require it. A missing client diagnostic can therefore be unrelated to an incremental job’s ability to run. Verify the sync job journal, the KB contents, and retrieval before treating this isolated check as ingestion failure.

EXTERNAL SEARCH AND HEALTH CHECKS

When search is enabled, doctor checks the local SearXNG systemd unit as well as a live query. If --searxng-url selects an external service, an inactive local unit can still appear as a failure. Interpret it against the external endpoint test. Health checks generally use loopback; custom bind designs must preserve the required local connectivity.

SINGLE MODEL DISCOVERY

doctor, benchmark, metrics, backup, and restore can load a single saved deployment when --model is omitted. update has its own all deployment discovery. status and stop require an explicit intended model for reliable selection. Always provide full target details for one shot rag-sync; its required sync options are validated before operational discovery.

SHARED SERVICES AND DISTRIBUTED CONTROL

SearXNG and backend build paths are shared. Open WebUI is also shared by default on each coordinator: the primary database and TLS/credentials are reused, while each active model has its own API. A model-scoped stop does not stop the shared UI. Multi-host distributed status, stop, and standalone test remain limited; use local systemd checks when necessary.

Operational acceptance Service health, source reported test results, and a plan are separate forms of evidence. Complete the real account, retrieval, memory, GPU, and recovery tests needed for your installation before production acceptance.

PRODUCTION READINESS

ACCEPT AND HAND OVER A DEPLOYMENT

Use this checklist to agree on the intended service and demonstrate that it works. Record the evidence, owner, and date in the site’s deployment record.

AreaAcceptance evidence
Model and capacityChosen model and revision; representative answer quality; acceptable prompt and generation throughput; tested context and concurrent use.
Host coexistenceObserved CPU, RAM, GPU, and container limits; acceptable storage and VM latency while inference and ingestion are active.
Network and TLSPrivate inference and RPC scope; approved browser and gateway clients; valid certificate identity, trust, and renewal ownership.
Users and identityNamed administrators; tested local recovery login; expected approval and collection access; validated OIDC or LDAP if used.
KnowledgeApproved source path and owner; known KB ID; declared incremental or mirror policy; real indexing and retrieval acceptance test.
CredentialsProtected WebUI, RAG, gateway, and identity secrets; rotation and recovery procedures; approved custody of backup archives.
MaintenanceComplete serve command and registry record; pinned or recorded image choices; update procedure; service dependencies and rollback assets.
RecoveryOff host operator backup; separately retained weights, binaries or build record, secrets, and offline caches where needed; successful isolated restore.
MonitoringReal inference and HTTPS checks; timer job results; local metrics collection; disk free space and backup completion review.
OwnershipNamed operational owner; support route; maintenance window; agreed recovery time and recovery point objectives.

A PRACTICAL HANDOVER TEST

Ask a new authorized user to sign in, select the model, retrieve a fact from a permitted KB, and report the source. Have the operator show a successful backup and a recent sync result, then explain how to pause the service and restore it. This connects infrastructure checks with the service the team actually needs.

Keep current Store changes to the serve command, secret paths, model revision, and image choice with the site record. Rerun the relevant acceptance tests after changes to the model, identity, document extraction, network path, or recovery environment.

COMMAND REFERENCE

SELECT THE RIGHT COMMAND

Syntax is deployai COMMAND [OPTIONS]. Place options after the command. With arguments but no explicit command, the parser defaults to deploy; use a named command in runbooks. With no arguments, the program shows help.

CommandPurpose and operational effect
list-modelsLists built in GGUF profiles, the Kimi compatibility profile, and the custom option.
gpu-statusReads PCI GPU inventory, loaded modules, and available NVIDIA or AMD runtime signals.
planCalculates CPU and context recommendations and displays hardware information. Does not establish model fit or apply quotas.
preflightValidates host, storage, and selected GPU readiness without driver or toolkit installation. See the AMD repository side effect note.
deployBuilds the chosen backend, downloads a pinned checkpoint, checks full integrity by default (or size only with --model-verify off), and normally runs a CLI smoke test.
serveStarts/reconciles the selected model inference service and default shared Open WebUI (or per-model UI if requested), search, gateway, and sync services.
testRuns a standalone model CLI smoke test. Uses a fixed 2048 context in the GGUF test path; does not validate the online UI.
statusReports selected model units and model path. Inactive unused topology units can be normal. Explicitly select the model.
stopStops the selected model inference/RPC services; preserves the pinned shared WebUI when --webui-mode shared. Model data is retained.
doctorChecks configured services and endpoints. --full issues real inference. User retrieval and memory still need separate tests.
benchmarkRuns one warm up and repeated fixed inference generations against the local serving API.
metricsPrints the local llama.cpp Prometheus response to standard output, with normal tool logging also present.

MAINTENANCE COMMANDS AND HELP

CommandPurpose and operational effect
backupCaptures operator state, briefly quiescing WebUI and a scheduled sync timer. Excludes model weights.
restoreValidates and applies a selected operator archive. --restore-file is required; --dry-run validates without applying.
updateDiscovers deployed GGUF configurations, checks software updates, and reconciles service configuration.
rag-syncSynchronizes one source directory to an existing Knowledge Base using the specified WebUI account key.
rpc-workerAdvanced managed worker setup action used by distributed deployment. Not the normal operator entry point.
gateway-runtimeInternal systemd runtime. It consumes --gateway-config PATH, not the ordinary deployment option set.
mcp-stdioInternal native SearXNG MCP transport. It accepts --searxng-url for its runtime; standard deployment logging is suppressed to preserve the protocol.

HELP FORMS

deployai --help
deployai help all
deployai --help-all
deployai help gpu
deployai help rag
deployai help api
deployai help identity
deployai help update
deployai help distributed

The patched deployment parser registers 88 documented flags, including --model-verify, --webui-mode, and --webui-primary-model. -h and --help are top-level help shortcuts; --gateway-config is an internal runtime argument. Use deployai help all to confirm support in the executable you installed.

BOOLEAN FLAGS AND ON OR OFF SETTINGS

A Boolean flag is enabled by writing the flag alone, for example --verbose or --dry-run. Use --verbose=false to explicitly clear one. String toggles require a value, for example --web-search off. A bare --web-search is invalid. Quoted paths are accepted; bind addresses must be literal IP addresses rather than DNS names.

Preview with care Use plan for sizing, update --dry-run for deployment discovery, restore --dry-run for archive validation, and incremental rag-sync --dry-run for a remote diff attempt. Review command specific behavior rather than assuming one dry run contract for every command.

PARAMETER REFERENCE

DEPLOYMENT AND MODEL SELECTION

All 88 registered deployment flags are organized across the eight parameter-reference pages. Defaults are raw source-code defaults before environment values, auto sizing, or saved deployment discovery. Unset means no parser value.

OptionDefaultUse and constraints
--modelkimi-k3Profile ID. Select explicitly for a new GGUF deployment; list-models shows choices.
--modeindependentindependent or distributed. Distributed requires two or three hosts and explicit CPU selection.
--nodes31 for the local host; 2 or 3 expands to the built in stratistor1 hostnames. Otherwise use one to three unique comma separated hostnames.
--storage-rootUnsetExisting real absolute directory for checkpoints. Required for deploy, serve, preflight, and multi host work.
--shared-rootUnsetLegacy alias for --storage-root. Prefer the current name and do not supply conflicting aliases.
--install-rootautoauto, /stratistor, /stratiserv, or /baseos. Resolves platform and MOK material for GPU preparation.
--hf-repoUnsetORG/REPO Hugging Face source for custom-gguf only. Required together with --hf-file.
--hf-fileUnsetSingle basename ending .gguf for custom-gguf only; no directory path. The custom service identity and port are shared.

INFERENCE SIZING AND STAGING

OptionDefaultUse and constraints
--threads64 rawCPU inference threads, 1 to 1024. Auto sizing can replace the raw default. An explicit CLI value wins.
--ctx-size4096 rawContext tokens, 512 to 131072. Auto sizing can replace the raw default; actual model and memory limits still apply.
--sizingautoauto or off. Auto applies thread and context heuristics during deploy, serve, preflight, and plan.
--resource-profileautoauto, shared, or dedicated. Selects sizing and isolation policy; auto detects active Ceph or virtualization services.
--resource-isolationautoauto, on, or off. Auto adds systemd and OCI resource ceilings on detected shared hosts.
--build-jobs16Parallel compiler jobs, 1 to 256. Controls building, not inference request concurrency.
--tokens2CLI smoke test output tokens, 1 to 128. Does not set the browser chat output limit.
--verify-jobs8Compatibility checksum workers for Kimi; native GGUF hashing is not controlled by this flag. For GGUF integrity use --model-verify.
--model-verifyonon performs the standard pinned-Hub SHA-256 or hf cache verification. off checks pinned revision and file size only; no cryptographic validation. GGUF workflow; default on.
--download-workers16Compatibility download worker count passed to Kimi. Native single GGUF download does not use this flag for its transfer concurrency.
--presetautoKimi only: auto, server, max, workstation, desktop, laptop, or ultra. No effect on normal GGUF serving.

Override rule To force a thread or context value, supply the CLI flag explicitly. Values inherited from environment variables do not set the parser’s explicit override markers and can be replaced by automatic sizing.

GPU REASONING AND RPC

OptionDefaultUse and constraints
--gpuautoauto, none, nvidia, or amd. Auto selects a detected vendor, CPU if none, and errors if both vendors are present.
--gpu-installonon or off. Allows permitted missing GPU prerequisite installation. Off does not turn all deployment package installation off.
--gpu-layers-1-1 for maximum requested offload, or 0 to 999 layers. Does not guarantee VRAM fit. Positive layers are invalid with CPU selection.
--nvidia-driver-flavoropenopen or proprietary. Chooses the NVIDIA driver path if installation is needed; it is not permission to replace a conflicting host stack.
--qwen-thinkingoffon or off. Off asks compatible Qwen3 server builds to disable reasoning output through supported template or reasoning flags.
--rpc-port00 derives 40000 plus the model API port. Use the same valid TCP port on all RPC peers; distributed default is recommended.
--rpc-address-2UnsetPrivate IPv4 address for the second listed distributed worker host. Required for distributed multi host actions.
--rpc-address-3UnsetPrivate IPv4 address for the third listed worker. Required when selecting three distributed hosts.

Distributed ports Use the derived RPC port in this release unless you have validated all peer startup commands. Remote worker setup does not propagate a nondefault --rpc-port consistently.

RUNTIME BROWSER AND FEATURES

OptionDefaultUse and constraints
--container-runtimedockerdocker or podman. Both may coexist; selects the runtime used by managed AI containers. Does not request uninstall of the other runtime.
--container-installbothboth, selected, or off. selected attempts install only the chosen runtime; off uses what exists. Existing Podman/Docker/Ceph packages are protected; real RPM conflicts fail without forced replacement.
--web-uionon or off. In shared mode (default), this controls one primary Open WebUI for all active local model inference services.
--webui-modesharedshared (default): one primary UI across active local models. per-model: legacy one UI per model.
--webui-primary-modelUnsetOptional existing primary model ID during first shared-mode serve; owner is pinned and cannot be changed implicitly afterward.
--web-searchonon or off. Configures SearXNG integration. A UI disabled fresh deployment should also set this off unless an MCP adapter is intentionally configured.
--ragonon or off. Configures Open WebUI document retrieval; does not automatically select, create, or sync a KB.
--memoryonon or off. Configures user memory capabilities and review settings. Recall requires real application testing.
--memory-review-turns10Memory background review cadence, 1 to 1000 conversation turns.
--listen-address127.0.0.1Literal inference bind IP. Metrics inherit this address. Keep loopback for the managed health and gateway workflows.
--webui-bind0.0.0.0Literal UI bind IP. All IPv4 interfaces by default; constrain network access separately.
--webui-port00 derives API port plus 10000. For an override, choose a free valid TCP port and reuse it in diagnostics and sync.
--webui-imagemain imageDefault ghcr.io/open-webui/open-webui:main. Accepts an OCI image reference; use a tested reference for controlled updates.

DATA SEARCH TLS AND API KEYS

OptionDefaultUse and constraints
--webui-data-root/var/lib/stratistor-ai/open-webuiPersistent state root. Default shared mode preserves the chosen primary model data directory; secondary models do not create replacement databases.
--webui-tlsonon or off. Native HTTPS in the UI container. The gateway and external identity require on.
--webui-tls-sanUnsetComma separated extra DNS names or IPs for generated certificates. Include the name clients actually use.
--webui-tls-certUnsetAbsolute existing regular PEM certificate file, supplied together with --webui-tls-key. The pair must match and be current.
--webui-tls-keyUnsetAbsolute matching regular PEM private key. Material is copied to the managed TLS folder; source paths are not mounted directly.
--webui-tls-renew-days30Generated certificate renewal threshold, 1 to 365 days. Evaluated on reconciliation; no standalone renewal daemon.
--searxng-port18888Local SearXNG loopback port, 1024 to 65535. Search is shared, so coordinate changes across models.
--searxng-urlUnsetExternal SearXNG HTTP or HTTPS base URL, replacing local setup for that configuration.
--webui-api-keysautoauto, on, or off. Auto preserves the existing setting; requested real sync enables API key support when needed. Cannot be off with sync.

KNOWLEDGE SYNC AND GATEWAY

OptionDefaultUse and constraints
--rag-syncoffon or off. On in serve creates a scheduled ingestion timer. Requires RAG and WebUI plus source, KB, and token settings.
--rag-sourceUnsetAbsolute local source directory. Incremental mode rejects visible symlinks, special files, empty eligible content, and files above 50 MiB.
--rag-knowledge-idUnsetExisting Open WebUI collection ID. The syncing account needs permission; the tool does not create the collection.
--rag-token-fileUnsetAbsolute protected regular file containing the real Open WebUI account API key. Use mode 0600.
--rag-sync-modeincrementalincremental adds and updates while preserving absent documents; mirror also reconciles deletions. Explicit mode overrides the legacy alias.
--rag-sync-pruneoffLegacy on or off alias. on selects mirror unless an explicit --rag-sync-mode wins.
--rag-sync-minutes30Minimum 5 minutes. Interval is measured from the previous oneshot job becoming inactive; boot delay is five minutes.
--api-gatewayoffon or off. Optional authenticated HTTPS inference proxy; requires TLS enabled WebUI.
--gateway-bind127.0.0.1Literal gateway listener IP. Change only with an approved remote client and network design.
--gateway-port18104Gateway TCP port, 1024 to 65535. Avoid collision with another model UI port.
--gateway-rpm60Per source IP request limit per minute, 1 to 100000. NAT or proxy clients can share the limit.
--gateway-key-fileUnsetOptional absolute protected existing key file. If omitted, creates a managed key. Runtime requires at least 32 characters.

IDENTITY INTEGRATION

OptionDefaultUse and constraints
--identitylocallocal, oidc, or ldap. Configures browser authentication; document roles and authorization still require validation.
--identity-urlUnsetExternal HTTPS UI URL. Required for OIDC redirect configuration, including a nonstandard port when used.
--oidc-client-idUnsetOIDC client registration identifier. Required for oidc.
--oidc-discovery-urlUnsetHTTPS OpenID configuration discovery URL. Required for oidc.
--oidc-secret-fileUnsetAbsolute protected file holding the OIDC client secret. Required for oidc; keep mode 0600.
--oidc-provider-nameStratiSYSTEM SSOLabel shown for the external login provider.
--ldap-hostUnsetDirectory server hostname. Required for ldap.
--ldap-port636Directory server TCP port, 1 to 65535; normally LDAPS 636.
--ldap-search-baseUnsetDirectory base DN to search. Required for ldap.
--ldap-bind-dnUnsetDirectory bind principal DN. Required for ldap.
--ldap-password-fileUnsetAbsolute protected file containing the directory bind password. Required for ldap.
--ldap-tlsonon or off. Enables TLS for LDAP and certificate validation. Use a trusted directory certificate.

DIAGNOSTICS UPDATES AND CONTROL FLAGS

OptionDefaultUse and constraints
--fullfalseBoolean. doctor performs real inference in addition to endpoint and configuration checks.
--metricsonon or off. Requests the llama.cpp metrics endpoint when supported; shares the inference listener.
--benchmark-runs3Measured generation runs, 1 to 30, following a warm up.
--benchmark-tokens128Requested output tokens per measured run, 16 to 4096.
--restore-fileUnsetAbsolute operator archive path for restore. Use --dry-run first for structural validation.
--update-llamaonon or off for update. Distributed backend binary updates are deliberately skipped pending coordinated rollout.
--update-modelsoffon or off for update. Explicit on permits new checkpoint revisions and can consume significant extra space.
--update-sourcefalseBoolean. During deploy, fetches and fast forwards llama.cpp source before building; also passed to the Kimi compatibility worker.
--update-webuifalseBoolean. During normal setup, pulls the selected UI image. update checks that image independently.
--force-buildfalseBoolean. Ignores a matching build receipt and rebuilds during deploy. Does not bypass GPU safety checks.
--force-testfalseBoolean. Repeats the deploy smoke test instead of reusing its prior matching receipt.
--reverify-modelfalseBoolean. Rechecks checkpoint content even when a prior verification marker is reusable.
--no-testfalseBoolean. Skips the automatic post deploy CLI model test. Validate real inference separately.
--verbosefalseBoolean. Shows detailed external command and build output; detailed logs are also written to the log directory.
--dry-runfalseBoolean. Command specific preview; restore validates, update lists, incremental sync attempts a diff. Does not universally suppress all service requests.

ENVIRONMENT REFERENCE

UNDERSTAND DEFAULTS AND PRECEDENCE

Environment variables supply initial values for selected flags. Explicit CLI values then replace them. Automatic sizing follows parsing and can replace thread or context values unless those particular flags appeared on the command line. Operational discovery can instead load an entire saved deployment.

SituationEffective behavior
New deploy or serveUses parsed CLI and environment settings, then applicable automatic sizing. The successful serve configuration is saved.
Existing serve rerunDoes not merge every omitted option from the registry. Reuse the complete recorded command.
One saved deployment and no explicit modeldoctor, benchmark, metrics, backup, and restore can use the saved configuration. Some command specific controls are then applied.
Explicit operational modelDo not assume every saved port, TLS setting, or disabled feature was merged. Supply relevant matching options.
updateDiscovers recorded deployments and applies update controls. Without an explicit model it processes all discovered records.
No dedicated environment mappingUse the documented CLI option or recorded configuration. Do not invent an environment name from a flag.

CORE ENVIRONMENT MAPPINGS

Environment variableEquivalent option
KIMI_MODEL_ID--model
STRATISTOR_AI_MODE--mode
STRATISTOR_AI_NODES--nodes
STRATISTOR_AI_STORAGE_ROOT--storage-root
KIMI_SHARED_ROOTFallback storage root when the current variable is unset
GGUF_CUSTOM_REPO and GGUF_CUSTOM_FILE--hf-repo and --hf-file
STRATISYSTEM_INSTALL_ROOT--install-root
KIMI_THREADS and KIMI_BUILD_JOBS--threads and --build-jobs
STRATISTOR_AI_CONTEXT_SIZE--ctx-size
KIMI_TOKENS and KIMI_PRESET--tokens and --preset
STRATISTOR_VERIFY_JOBS and STRATISTOR_DOWNLOAD_WORKERS--verify-jobs and --download-workers
STRATISTOR_AI_MODEL_VERIFY--model-verify (on|off, default on)

ENVIRONMENT VARIABLE MAPPING

Environment variableEquivalent option
STRATISTOR_AI_GPU--gpu
STRATISTOR_AI_GPU_INSTALL--gpu-install
STRATISTOR_NVIDIA_DRIVER_FLAVOR--nvidia-driver-flavor
STRATISTOR_AI_GPU_LAYERS--gpu-layers
STRATISTOR_AI_CONTAINER_RUNTIME--container-runtime
STRATISTOR_AI_CONTAINER_INSTALL--container-install
STRATISTOR_WEBUI_MODE and STRATISTOR_WEBUI_PRIMARY_MODEL--webui-mode and --webui-primary-model
STRATISTOR_WEB_UI and STRATISTOR_WEB_SEARCH--web-ui and --web-search
STRATISTOR_AI_RAG and STRATISTOR_AI_MEMORY--rag and --memory
GGUF_LISTEN_ADDRESS--listen-address
STRATISTOR_WEBUI_BIND and STRATISTOR_WEBUI_PORT--webui-bind and --webui-port
STRATISTOR_WEBUI_IMAGE--webui-image
STRATISTOR_WEBUI_DATA_ROOT--webui-data-root
STRATISTOR_WEBUI_TLS and STRATISTOR_WEBUI_TLS_SAN--webui-tls and --webui-tls-san
STRATISTOR_WEBUI_TLS_CERT and STRATISTOR_WEBUI_TLS_KEY--webui-tls-cert and --webui-tls-key
STRATISTOR_WEBUI_TLS_RENEW_DAYS--webui-tls-renew-days
STRATISTOR_SEARXNG_PORT and STRATISTOR_SEARXNG_URL--searxng-port and --searxng-url
STRATISTOR_MEMORY_REVIEW_TURNS--memory-review-turns
STRATISTOR_QWEN_THINKING--qwen-thinking
STRATISTOR_RPC_PORT--rpc-port
STRATISTOR_RPC_IP_2 and STRATISTOR_RPC_IP_3--rpc-address-2 and --rpc-address-3
STRATISTOR_AI_UPDATE_LLAMA and STRATISTOR_AI_UPDATE_MODELS--update-llama and --update-models

Secrets Use the protected file options for identity, RAG, and gateway credentials. The general parser does not expose direct environment mappings for every newer operational feature. Avoid recording secret values in shell history or deployment command text.

OPERATOR REFERENCE

PATHS SERVICES AND SITE RECORD

Path or unit patternContents or responsibility
/ai-models/ai/models/MODEL/Checkpoint and verification metadata for the example storage root. custom-gguf adds a repository and file hash subdirectory.
.gguf-verified / .gguf-size-validated (within model directory)Verification receipt after full Hugging Face checksum validation / size-only receipt when checksums are intentionally disabled.
/opt/llama.cpp/Shared source checkout and backend builds: build, build-cuda, build-hip, and build-rpc.
/var/lib/stratistor-ai/open-webui/MODEL/Persistent WebUI state for the pinned primary model (shared mode); standalone state directories remain preserved for legacy per-model instances.
/etc/stratistor-ai/open-webui/MODEL/Environment, admin-credentials and TLS for the pinned primary shared WebUI; secondary model deployments reuse them.
/etc/stratistor-ai/open-webui/shared-ownerPinned primary model identity for the shared WebUI; retain it with backups and deployment records.
/etc/stratistor-ai/deployments/MODEL.jsonRecorded configuration used for deployment discovery and updates.
/etc/stratistor-ai/api-gateway/MODEL/Gateway configuration and default generated API key.
/var/backups/stratistor-ai/Protected operator archives and pre update UI snapshots.
/var/log/stratistor-ai-go/Timestamped detailed logs. Nonroot logging uses a temporary per user directory.
stratistor-llama-MODEL.serviceIndependent inference process.
stratistor-dist-MODEL.service and stratistor-rpc-MODEL.serviceDistributed coordinator inference and CPU RPC worker.
stratistor-open-webui-MODEL.service and stratistor-ai-searxng.serviceOne primary shared WebUI unit by default; additional model-specific units may exist from earlier per-model deployments. SearXNG remains shared.
stratistor-ai-gateway-MODEL.service and stratistor-ai-ragsync-MODEL.service or .timerOptional gateway and scheduled document ingestion.

MINIMUM SITE DEPLOYMENT RECORD

Retain host role and addresses; storage, installation and shared-owner paths; complete deploy and serve commands; checkpoint revision and verification mode; model ID and API port; primary WebUI model and URL; Docker/Podman selection and installation policy; context/threads, measured performance, credentials and recovery contacts; and RAG/memory/identity test evidence.