chat/docs/OPERATIONS.md

Operations

Environment

Copy .env.example to .env. Full list:

Variable	Default	Purpose
NODE_ENV	development	production under the NixOS module (and CI); gates the DRIVER_DEBUG default
PORT / HOST	3000 / 127.0.0.1	listen address (keep localhost behind nginx)
APP_PASSWORD	(required)	admin access key — generate with openssl rand -base64 24
SESSION_TTL_HOURS	720	Bearer session lifetime (30 days)
DATA_DIR	./data	holds kisassistant.db (SQLite — unused when POSTGRESQL_URL is set)
POSTGRESQL_URL	(empty)	Postgres fallback, e.g. postgres://kisassistant:secret@localhost:5432/kisassistant (typically a docker container). Empty = local SQLite. Boot fails fast when unreachable.
CDN_DIR	./cdn	local file CDN storage (avatars; NixOS: under state dir)
OCR_LANG	eng	tesseract language(s) for the OCR tool
OCR_MAX_CHARS	100000	transcription cap (longer results are truncated + flagged)
TESSERACT_BIN	tesseract	path to the tesseract binary (NixOS module puts it on PATH; missing binary cleanly disables OCR with 501)
DRIVER_DEBUG	true dev / false prod	per-driver debug logging to stdout
WIKI_URL	https://git.kisakay.com/k/chat/wiki	remote docs wiki — GET /wiki redirects (302) there
SMTP_HOST / SMTP_PORT / SMTP_SECURE	(empty) / 587 / false	key-recovery mailer — empty host disables it (false = STARTTLS, true = implicit TLS/465)
SMTP_USER / SMTP_PASS	(empty)	SMTP auth (omit both for open relays / local catchers)
SMTP_FROM	KisAssistant <[email protected]>	sender address
APP_URL	http://localhost:3000	public base URL used in reset links — must be the real origin in prod
RESET_TTL_MIN	60	reset-link lifetime (single use)
RATE_LIMIT_MAX / RATE_LIMIT_WINDOW_MIN	5 / 10	login rate limit per IP
OLLAMA_HOST / OLLAMA_ENABLED	http://10.66.66.4:11434 / true	local-model driver
MISTRAL_*, GLM_*, DEEPSEEK_*, ANTHROPIC_*, OPENAI_*, GEMINI_*	disabled	API drivers: set *_ENABLED=true and *_API_KEY
GLM_BASE_URL	https://api.z.ai/api/paas/v4	Zhipu platform override (mainland China: https://open.bigmodel.cn/api/paas/v4)

Local development

bun install
bun run dev              # backend watch mode (restarts on edit), :3000
bun run dev:frontend     # Vite HMR + /api proxy, :5173 (use this for UI work)
bun run dev:all          # both at once
bun run build:frontend   # production build into public/

Smoke tests: start the backend on a free non-3000 port with a throwaway DATA_DIR (port 3000 is the owner's dev server — do not touch it), then drive the API with curl/python: login → models → conversations → chat → share → unshare. See AGENTS.md.

Postgres fallback (docker)

SQLite is the default. For Postgres, run a container and point the app at it:

docker run -d --name kisassistant-pg --restart unless-stopped \
  -e POSTGRES_USER=kisassistant -e POSTGRES_PASSWORD=secret \
  -e POSTGRES_DB=kisassistant -p 127.0.0.1:5432:5432 \
  -v kisassistant-pgdata:/var/lib/postgresql/data \
  postgres:17-alpine

POSTGRESQL_URL=postgres://kisassistant:[email protected]:5432/kisassistant \
  DATA_DIR=/tmp/ka-pg-throwaway \
  bun run src/index.ts

Notes:

• The app uses Bun's native bun:sql driver — no extra dependency.
• Schema is created idempotently at boot (same tables as SQLite; timestamps are BIGINT, auto ids BIGSERIAL). A fresh database starts empty except the bootstrapped admin row — there is no SQLite→Postgres data migration (re-create accounts or keep SQLite).
• Switching back is just unsetting POSTGRESQL_URL (the SQLite file is untouched while Postgres is active).
• Back up with pg_dump (SQLite: copy kisassistant.db, or .backup while running under WAL).

Managing accounts

1. Log in as admin with the APP_PASSWORD key.
2. Open Accounts → New account → username (+ display name).
3. Copy the one-time ka_… key and hand it to the owner (chat, QR code, …).
4. Users set their display name, avatar (upload a picture in the profile settings — hosted on the built-in CDN), theme, accent color and feature flags in the settings.
5. Rotate a compromised key with the regenerate button (old sessions die); delete removes the account with all its chats and shares.

Admin Center (/admin) & self-registration

The sidebar Admin Center button (admin only) opens the /admin page with seven deep-linkable tabs (/admin = accounts, /admin/models, /admin/stats, /admin/access, /admin/reports, /admin/features, /admin/mail): Accounts (same manager as before), Access (access-request wishlist triage, see below), Reports, Models (node pool, per-model policy, Ollama library + HF catalog), Stats (pool load charts + platform host facts, see below), Features (server-side switches + voice-preview lines), Mail (SMTP credentials viewer + connectivity tester: GET /api/admin/mail, POST /api/admin/mail/verify, POST /api/admin/mail/test, admin Bearer only — plus the password-recovery rate limits below).

• registration_enabled (default off): when on, the login page shows a working Register button — anyone can create an account (key shown once) via POST /api/auth/register. When off, the button is greyed with the tooltip "Registration currently disabled on this platform" and the endpoint answers 403. Flipped live, no restart. Turning it on retires the access-request wishlist automatically.
• registration_request_enabled (default on): shows Request access on the login page (wishlist). Only switchable on while public registration is off — the two are mutually exclusive (API answers 409 otherwise).
• tools_ocr_enabled (default on): admin kill-switch for the OCR tool.
• reports_enabled (default on): admin kill-switch for message reports — the chat flag button turns grey with a "Report disabled" tooltip and POST /api/reports answers 403. Surfaced to clients via GET /api/tools.
• username_change_enabled (default on): lets users rename themselves (PATCH /api/me username); off answers 403 and locks the Settings field. Admins bypass it (PATCH /api/admin/users/:id still works). Surfaced to clients via GET /api/tools.
• Flags live in the settings SQLite table, edited via GET/PATCH /api/admin/settings (admin Bearer only). The same endpoint carries the Ollama pool tuning (ollama_timeout_s 5–3600, default 300; ollama_default_weight 0–1000, default 100) and the recovery envelope (recovery_limit_max 1–100, default 5; recovery_limit_window_min 1–1440, default 60).

Password recovery rate limits

POST /api/auth/recover and the /api/auth/reset/:token endpoints share a dedicated per-IP envelope (separate from login brute-force protection): every hit counts, successes included (recovery answers are always-ok by design, so enumeration is impossible). Over-limit answers 429 with retryAfterSec + Retry-After. The limits are admin-configurable live in Admin Center → Mail → Password recovery limits, no restart.

Model access policy (kill-switch + rate limits)

The Admin Center Models tab lists every model across enabled drivers (grouped, Ollama library moved there too). Per model id ("driver:model"): an Enabled switch and hourly / daily per-user request caps (0 = unlimited, admins always bypass). Policy lives in the settings table (model_policy JSON blob, default entries pruned); usage counters in model_usage (fixed UTC hour/day windows, old buckets pruned on write).

• Disabled models vanish from GET /api/models for users (admins still see everything) and POST /api/chat rejects them (403) — attempts count toward limits even when generation later fails; over-limit answers 429.
• Routes: GET /api/admin/model-policy[?refresh=1], PATCH /api/admin/model-policy ({model, enabled?, hourly?, daily?}). Ollama library: POST /api/admin/ollama/pull ({name, nodeId?}, NDJSON progress — library tags AND hf.co/user/repo:quant references), DELETE /api/admin/ollama/models/:name[?node=], and the connectivity probe GET /api/admin/ollama/status (default node — surfaced in the node pool).

Ollama node pool (multi-host, weighted)

Admin Center → Models → Ollama nodes: register extra Ollama hosts (modal with name, host, weight + a Test probe before saving). The env-configured OLLAMA_HOST stays as the implicit default node (read-only row, env badge). Each generation picks a node by weighted random among enabled nodes with weight > 0 (share shown as % — firewall-like priority, heaviest first) and fails over to the next node on error or timeout, but only before any token was produced (a started reply never restarts elsewhere). Weight 0 or disabled = no traffic. Pool changes invalidate the model list cache; listModels merges tags across nodes.

• ollama_timeout_s (default 300 s, Admin Center → Models): max time for one generation per node attempt (caller aborts propagate instantly and never fail over).
• Node detail view: live probe (version, latency, on-disk models, running models with VRAM via /api/ps) + load charts over 1h / 6h / 24h / 7d / 30d (requests, errors, avg latency, tokens/s from app-side generation telemetry in ollama_stats, 35-day retention — Ollama exposes no CPU/RAM API, so host CPU/RAM meters are intentionally absent).
• HF catalog (same tab): live Hugging Face search (filter=gguf, sorted by downloads), per-repo quantization picker, one-click pull to the selected node. Needs Ollama with hf.co/… support + internet on that host.
• Routes: GET/POST /api/admin/ollama/nodes, PATCH/DELETE /api/admin/ollama/nodes/:id, GET …/nodes/:id/status, GET …/nodes/:id/stats?window=1h|6h|24h|7d|30d, GET /api/admin/ollama/probe?host=….

Telemetry sub-page (/admin/stats) + platform facts

The Stats tab is the pool observatory: node picker, window picker (1h / 6h / 24h / 7d / 30d), chart-type dock (bar / area / line SVG charts with hover tooltips: requests, errors, avg latency, tokens), totals tiles, and a latency underlay. Below, platform host facts for this server — hostname, kernel, arch/CPU, app + system uptime, load average, memory bar, runtime, network interfaces/IPs (GET /api/admin/platform, admin only). Disk meters are intentionally absent (models live on the Ollama hosts).

Voice: dictation + playback

• Composer mic (browser speech recognition — Chrome/Edge): records with a live gradient waveform, timer and live transcript; Stop drops the transcript into the composer, editable before sending; Discard throws it away. Unsupported browsers get a disabled mic with an explanatory tooltip. No audio leaves the browser (no server STT endpoint yet — whisper.cpp via WHISPER_BIN is the documented next step).
• Message playback: every assistant bubble has a Listen button with a Siri-style breathing glow while speaking (click again to stop; playback stops on send / new message / conversation switch). Engine per user (Settings → Voice): auto (neural server voice when available, browser fallback), browser, or server.
• Neural server TTS (POST /api/tools/tts {text, voice?} → audio bytes, GET /api/tools/tts/info → provider + voices): ElevenLabs (any voice_id, most natural) → OpenAI tts-1 (6 stock voices, reuses the LLM key) → local piper (PIPER_BIN + PIPER_MODEL, offline WAV). TTS_PROVIDER forces one; TTS_VOICE / ELEVENLABS_VOICE set defaults. Output is cached on disk (sha of provider|voice|text, 200 files cap), rate-limited to 30 jobs/hour/account (429), kill-switched via tools_tts_enabled (Admin Center → Features). Test overrides: ELEVENLABS_BASE_URL / OPENAI_BASE_URL.
• Settings → Voice (per browser, localStorage): engine, language (auto-detect per message or forced), browser voice/rate/pitch, neural voice id, and one-tap previews of the admin-set lines.
• Admin Center → Features → Voice previews: the default playback lines (voice_previews setting, 1–6 lines {text, lang, rate, pitch, timbre, voice}, each with its OWN voice character — masculine/feminine/any timbre hint, speed, pitch, optional server voice id — so previews sound different from each other; timbre matching is best-effort over the browser's voice names, rate/pitch always apply; empty list resets to the built-ins), also served to users via GET /api/tools. Users can adopt any preview's character as their own playback voice from Settings → Voice.

Personal providers (BYOK)

Users can connect their own OpenAI / Anthropic / DeepSeek / Gemini API keys in Settings → Providers (per account, no admin involved). Connected providers add their models to that user's model menu at the top (no page refresh needed); usage is billed to the user's own provider account, and the admin model policy (kill-switch + rate limits) still applies.

• Keys are stored server-side in user_provider_keys and never returned to any client (GET /api/me/providers exposes presence + last 4 chars only; never logged). A stored key greys out its field (Replace to rotate, Remove to disconnect). Deleting the account wipes its keys.
• Routes: GET /api/me/providers, PUT /api/me/providers/:id ({apiKey}, 8–256 chars), DELETE /api/me/providers/:id (all Bearer, owner-scoped).

Access-request wishlist (registration approval queue)

When public registration is off, visitors can still reserve a username via Request access on the login page: username + email (required) + a motivation message (10+ chars). This creates a ticket (access_requests + access_messages tables) with a personal page at /review/<id> (unguessable UUID, same pattern as share links).

• Requester: follows the decision, argues their case and replies on the /review page. Closed tickets (accepted/refused) are read-only.
• Admin: Access tab in the Admin Center — filter by status, open a ticket, reply (emailed to the requester), and set pending → reviewing → accepted / refused with a reason. Accepting creates the account and returns the access key once (also emailed). Terminal tickets can't be reopened.
• Email (needs SMTP_*): received-confirmation, reviewing/accepted/refused notices, and every admin reply — all HTML, all stating email is notification-only and replies belong on the /review page.
• Routes: POST /api/access/request (public, rate-limited), GET /api/access/ticket/:id + POST .../message (public, ticket bearer), GET /api/admin/access, GET/PATCH /api/admin/access/:id, POST /api/admin/access/:id/message (admin).
• Live updates ride WebSockets (no polling): GET /api/access/ws/:id (ticket bearer — pushes access_message + access_status for that ticket) and GET /api/admin/ws?token=… (admin Bearer in query string, browsers can't set WS headers — pushes access_message + access_status + access_created for every ticket plus report_created + report_status, driving the AdminCenter badges and the triage lists). Every event carries its full payload (access_created bundles the ticket, its first message and the message count), so clients apply it straight to state — HTTP resync only happens on (re)connect to cover the offline gap. Clients ping every 25 s ({"type":"ping"} → {"type":"pong"}) with a pong watchdog that redials half-dead sockets.

Content reports (flagged AI responses)

Every assistant message carries three actions: Copy, Search the web (DuckDuckGo, new tab) and Report. Reporting opens a modal with a reason picker (copyright, gore, falseinfo, bug) plus optional details, and creates a row in the reports table with a content snapshot (survives later edits/deletion of the chat), the triggering user prompt, the model, and the reporter.

• Reporter: capped at 20 open (open + reviewing) reports (429 beyond). Reporting the same message twice returns the existing report (409 already reported).
• Admin: Reports tab in the Admin Center (badge counts open reports, excluding shadow-banned reporters) — filter by status, expand for the full snapshot + reporter details, and set open → reviewing → resolved / dismissed with an internal note. An admin can also shadow-ban a noisy reporter (toggle on the report detail, with confirm): their reports are still accepted but flagged unreliable and hidden from the badge.
• Routes: POST /api/reports (Bearer), GET /api/admin/reports, PATCH /api/admin/reports/:id, POST /api/admin/users/:id/shadowban (admin).

Key recovery via email (optional)

Set the SMTP_* vars (and the real public APP_URL) to enable it; otherwise the login page hides the "Forgot your access key?" link.

1. Admin sets a recovery email at account creation (or the user sets their own in profile settings).
2. User clicks the link on the login page and enters their username. The API always answers {ok:true} (no account enumeration) and, if the account has an email, sends a one-time /reset/<token> link (60 min, single use).
3. Opening the link shows a confirm page; confirming rotates the key immediately — the new ka_… key is displayed once, old sessions die. (admin itself is excluded: its key lives in .env.)

Local file CDN (avatars)

PUT /cdn/<ns>/<key> (Bearer) / GET /cdn/<ns>/<key>.<ext> (public), e.g. /cdn/avatar/<account-id>.png. Namespaces live in CDN_NAMESPACES (src/cdn.ts) — add one to extend (future usage).

• Allowed types only jpg/png/webp, verified by magic bytes, never by extension (client filename is ignored; the stored extension comes from detection). Everything else → 415, unknown namespace/ext → 404.
• avatar rules: 5 MB max (413 above), 5 uploads max per 2 h per account (429 + Retry-After), ownership enforced (own id, or admin).
• Served with fixed Content-Type, nosniff, Cache-Control: public, max-age=3600. No listing, no traversal (strict charsets + SPA fallback).

Platform tools & attachments

Uploads always go through platform tools (GET /api/tools lists them with availability). Today: OCR (POST /api/tools/ocr, tesseract).

• Images (jpg/png/webp, 5 MB max, magic-verified, 20 jobs/hour/account) are transcribed on the backend to TXT — the model never receives image bytes. Text files ride the text CDN namespace (UTF-8 validated, 500 KB max, keys prefixed with the account id).
• The UI (paperclip menu in the composer) explains this and always shows a pre-transcription preview: the user reviews/edits the text, then attaches it. Attachments travel as labeled text blocks inside the message, so they persist in history and work with every model.

NixOS deployment (chat.kisakay.com)

Full step-by-step guide: NIXOS-HOSTING.md.

flake.nix exposes packages.<system>.default (backend + prebuilt frontend, frontend compiled offline from frontend/bun.lock via a fixed-output bun install) and nixosModules.default.

imports = [ kisassistant.nixosModules.default ];
services.kisassistant = {
  enable = true;
  package = kisassistant.packages.${pkgs.system}.default;  # required
  domain = "chat.kisakay.com";   # legacy switch: implies nginx + ACME
  enableNginx = true;            # explicit; requires domain
  passwordFile = "/run/secrets/kisassistant-password";  # agenix/sops-nix
  ollamaHost = "http://10.66.66.4:11434";
  extraEnv = { MISTRAL_ENABLED = "true"; };              # optional
};
security.acme.acceptTerms = true;  # required for the ACME/TLS vhost

The module sets NODE_ENV=production, CDN_DIR=/var/lib/kisassistant/cdn and DRIVER_DEBUG=false for the systemd service. enableNginx is a tri-state: true/false force the bundled reverse proxy on/off, null (the default) keeps the legacy behaviour (nginx iff domain is set).

Secrets (passwordFile, API keys) must come from files (agenix/sops-nix), never baked into the Nix store. The service stores SQLite under /var/lib/kisassistant/data; nginx terminates TLS and proxies to 127.0.0.1:3000.