chat/docs/OPERATIONS.md

# Operations

## Environment

Copy `.env.example` to `.env`. Full list:

| Variable | Default | Purpose |
|---|---|---|
| `NODE_ENV` | `development` | `production` under the NixOS module (and CI); gates the `DRIVER_DEBUG` default |
| `PORT` / `HOST` | `3000` / `127.0.0.1` | listen address (keep localhost behind nginx) |
| `APP_PASSWORD` | *(required)* | **admin** access key — generate with `openssl rand -base64 24` |
| `SESSION_TTL_HOURS` | `720` | Bearer session lifetime (30 days) |
| `DATA_DIR` | `./data` | holds `kisassistant.db` (SQLite — unused when `POSTGRESQL_URL` is set) |
| `POSTGRESQL_URL` | *(empty)* | Postgres fallback, e.g. `postgres://kisassistant:secret@localhost:5432/kisassistant` (typically a docker container). Empty = local SQLite. Boot fails fast when unreachable. |
| `CDN_DIR` | `./cdn` | local file CDN storage (avatars; NixOS: under state dir) |
| `OCR_LANG` | `eng` | tesseract language(s) for the OCR tool |
| `OCR_MAX_CHARS` | `100000` | transcription cap (longer results are truncated + flagged) |
| `TESSERACT_BIN` | `tesseract` | path to the tesseract binary (NixOS module puts it on PATH; missing binary cleanly disables OCR with 501) |
| `DRIVER_DEBUG` | `true` dev / `false` prod | per-driver debug logging to stdout |
| `WIKI_URL` | `https://git.kisakay.com/k/chat/wiki` | remote docs wiki — `GET /wiki` redirects (302) there |
| `SMTP_HOST` / `SMTP_PORT` / `SMTP_SECURE` | *(empty)* / `587` / `false` | key-recovery mailer — empty host disables it (`false` = STARTTLS, `true` = implicit TLS/465) |
| `SMTP_USER` / `SMTP_PASS` | *(empty)* | SMTP auth (omit both for open relays / local catchers) |
| `SMTP_FROM` | `KisAssistant <[email protected]>` | sender address |
| `APP_URL` | `http://localhost:3000` | public base URL used in reset links — **must** be the real origin in prod |
| `RESET_TTL_MIN` | `60` | reset-link lifetime (single use) |
| `RATE_LIMIT_MAX` / `RATE_LIMIT_WINDOW_MIN` | `5` / `10` | login rate limit per IP |
| `OLLAMA_HOST` / `OLLAMA_ENABLED` | `http://10.66.66.4:11434` / `true` | local-model driver |
| `MISTRAL_*`, `GLM_*`, `DEEPSEEK_*`, `ANTHROPIC_*`, `OPENAI_*`, `GEMINI_*` | disabled | API drivers: set `*_ENABLED=true` **and** `*_API_KEY` |
| `GLM_BASE_URL` | `https://api.z.ai/api/paas/v4` | Zhipu platform override (mainland China: `https://open.bigmodel.cn/api/paas/v4`) |

## Local development

```bash
bun install
bun run dev              # backend watch mode (restarts on edit), :3000
bun run dev:frontend     # Vite HMR + /api proxy, :5173 (use this for UI work)
bun run dev:all          # both at once
bun run build:frontend   # production build into public/
```

Smoke tests: start the backend on a **free non-3000 port** with a throwaway
`DATA_DIR` (port 3000 is the owner's dev server — do not touch it), then drive
the API with curl/python: login → models → conversations → chat → share →
unshare. See `AGENTS.md`.

## Postgres fallback (docker)

SQLite is the default. For Postgres, run a container and point the app at it:

```bash
docker run -d --name kisassistant-pg --restart unless-stopped \
  -e POSTGRES_USER=kisassistant -e POSTGRES_PASSWORD=secret \
  -e POSTGRES_DB=kisassistant -p 127.0.0.1:5432:5432 \
  -v kisassistant-pgdata:/var/lib/postgresql/data \
  postgres:17-alpine

POSTGRESQL_URL=postgres://kisassistant:[email protected]:5432/kisassistant \
  DATA_DIR=/tmp/ka-pg-throwaway \
  bun run src/index.ts
```

Notes:

- The app uses Bun's native `bun:sql` driver — no extra dependency.
- Schema is created idempotently at boot (same tables as SQLite; timestamps
  are `BIGINT`, auto ids `BIGSERIAL`). A fresh database starts empty except
  the bootstrapped `admin` row — **there is no SQLite→Postgres data
  migration** (re-create accounts or keep SQLite).
- Switching back is just unsetting `POSTGRESQL_URL` (the SQLite file is
  untouched while Postgres is active).
- Back up with `pg_dump` (SQLite: copy `kisassistant.db`, or
  `.backup` while running under WAL).

## Managing accounts

1. Log in as `admin` with the `APP_PASSWORD` key.
2. Open **Accounts** → *New account* → username (+ display name).
3. Copy the one-time `ka_…` key and hand it to the owner (chat, QR code, …).
4. Users set their display name, avatar (upload a picture in the profile settings —
   hosted on the built-in CDN), theme, accent color and feature flags in the settings.
5. Rotate a compromised key with the regenerate button (old sessions die);
   delete removes the account with all its chats and shares.

## Admin Center (`/admin`) & self-registration

The sidebar **Admin Center** button (admin only) opens the `/admin` page with
seven deep-linkable tabs (`/admin` = accounts, `/admin/models`,
`/admin/stats`, `/admin/access`, `/admin/reports`, `/admin/features`,
`/admin/mail`): **Accounts** (same manager as before), **Access**
(access-request wishlist triage, see below), **Reports**, **Models** (node
pool, per-model policy, Ollama library + HF catalog), **Stats** (pool load
charts + platform host facts, see below), **Features** (server-side
switches + voice-preview lines), **Mail** (SMTP credentials viewer +
connectivity tester: `GET /api/admin/mail`, `POST /api/admin/mail/verify`,
`POST /api/admin/mail/test`, admin Bearer only — plus the password-recovery
rate limits below).

- `registration_enabled` (default off): when on, the login page shows a
  working **Register** button — anyone can create an account (key shown once)
  via `POST /api/auth/register`. When off, the button is greyed with the
  tooltip *"Registration currently disabled on this platform"* and the
  endpoint answers 403. Flipped live, no restart. Turning it on retires the
  access-request wishlist automatically.
- `registration_request_enabled` (default on): shows **Request access** on the
  login page (wishlist). Only switchable on while public registration is off —
  the two are mutually exclusive (API answers 409 otherwise).
- `tools_ocr_enabled` (default on): admin kill-switch for the OCR tool.
- `reports_enabled` (default on): admin kill-switch for message reports —
  the chat flag button turns grey with a "Report disabled" tooltip and
  `POST /api/reports` answers 403. Surfaced to clients via `GET /api/tools`.
- `username_change_enabled` (default on): lets users rename themselves
  (`PATCH /api/me` `username`); off answers 403 and locks the Settings field.
  Admins bypass it (`PATCH /api/admin/users/:id` still works). Surfaced to
  clients via `GET /api/tools`.
- Flags live in the `settings` SQLite table, edited via
  `GET/PATCH /api/admin/settings` (admin Bearer only). The same endpoint
  carries the Ollama pool tuning (`ollama_timeout_s` 5–3600, default 300;
  `ollama_default_weight` 0–1000, default 100) and the recovery envelope
  (`recovery_limit_max` 1–100, default 5; `recovery_limit_window_min`
  1–1440, default 60).

## Password recovery rate limits

`POST /api/auth/recover` and the `/api/auth/reset/:token` endpoints share a
dedicated per-IP envelope (separate from login brute-force protection):
every hit counts, successes included (recovery answers are always-ok by
design, so enumeration is impossible). Over-limit answers 429 with
`retryAfterSec` + `Retry-After`. The limits are admin-configurable live in
Admin Center → Mail → Password recovery limits, no restart.

## Model access policy (kill-switch + rate limits)

The Admin Center **Models** tab lists every model across enabled drivers
(grouped, Ollama library moved there too). Per model id (`"driver:model"`):
an **Enabled** switch and **hourly / daily** per-user request caps
(0 = unlimited, admins always bypass). Policy lives in the `settings` table
(`model_policy` JSON blob, default entries pruned); usage counters in
`model_usage` (fixed UTC hour/day windows, old buckets pruned on write).

- Disabled models vanish from `GET /api/models` for users (admins still see
  everything) and `POST /api/chat` rejects them (403) — attempts count
  toward limits even when generation later fails; over-limit answers 429.
- Routes: `GET /api/admin/model-policy[?refresh=1]`,
  `PATCH /api/admin/model-policy` (`{model, enabled?, hourly?, daily?}`).
  Ollama library: `POST /api/admin/ollama/pull` (`{name, nodeId?}`, NDJSON
  progress — library tags AND `hf.co/user/repo:quant` references),
  `DELETE /api/admin/ollama/models/:name[?node=]`, and the connectivity probe
  `GET /api/admin/ollama/status` (default node — surfaced in the node pool).

## Ollama node pool (multi-host, weighted)

Admin Center → Models → **Ollama nodes**: register extra Ollama hosts
(modal with name, host, weight + a Test probe before saving). The
env-configured `OLLAMA_HOST` stays as the implicit `default` node (read-only
row, `env` badge). Each generation picks a node by **weighted random**
among enabled nodes with weight > 0 (share shown as % — firewall-like
priority, heaviest first) and fails over to the next node on error or
timeout, but only before any token was produced (a started reply never
restarts elsewhere). Weight 0 or disabled = no traffic. Pool changes
invalidate the model list cache; `listModels` merges tags across nodes.

- `ollama_timeout_s` (default 300 s, Admin Center → Models): max time for
  one generation per node attempt (caller aborts propagate instantly and
  never fail over).
- Node detail view: live probe (version, latency, on-disk models, running
  models with VRAM via `/api/ps`) + load charts over 1h / 6h / 24h / 7d /
  30d (requests, errors, avg latency, tokens/s from app-side generation
  telemetry in `ollama_stats`, 35-day retention — Ollama exposes no CPU/RAM
  API, so host CPU/RAM meters are intentionally absent).
- HF catalog (same tab): live Hugging Face search (`filter=gguf`, sorted by
  downloads), per-repo quantization picker, one-click pull to the selected
  node. Needs Ollama with `hf.co/…` support + internet on that host.
- Routes: `GET/POST /api/admin/ollama/nodes`, `PATCH/DELETE
  /api/admin/ollama/nodes/:id`, `GET …/nodes/:id/status`,
  `GET …/nodes/:id/stats?window=1h|6h|24h|7d|30d`,
  `GET /api/admin/ollama/probe?host=…`.

## Telemetry sub-page (`/admin/stats`) + platform facts

The **Stats** tab is the pool observatory: node picker, window picker
(1h / 6h / 24h / 7d / 30d), chart-type dock (bar / area / line SVG charts
with hover tooltips: requests, errors, avg latency, tokens), totals tiles,
and a latency underlay. Below, **platform host facts** for this server —
hostname, kernel, arch/CPU, app + system uptime, load average, memory bar,
runtime, network interfaces/IPs (`GET /api/admin/platform`, admin only).
Disk meters are intentionally absent (models live on the Ollama hosts).

## Voice: dictation + playback

- **Composer mic** (browser speech recognition — Chrome/Edge): records with
  a live gradient waveform, timer and live transcript; **Stop** drops the
  transcript into the composer, editable before sending; **Discard** throws
  it away. Unsupported browsers get a disabled mic with an explanatory
  tooltip. No audio leaves the browser (no server STT endpoint yet —
  whisper.cpp via `WHISPER_BIN` is the documented next step).
- **Message playback**: every assistant bubble has a Listen button with a
  Siri-style breathing glow while speaking (click again to stop; playback
  stops on send / new message / conversation switch). Engine per user
  (Settings → Voice): **auto** (neural server voice when available, browser
  fallback), **browser**, or **server**.
- **Neural server TTS** (`POST /api/tools/tts` `{text, voice?}` → audio
  bytes, `GET /api/tools/tts/info` → provider + voices): ElevenLabs
  (any voice_id, most natural) → OpenAI `tts-1` (6 stock voices, reuses the
  LLM key) → local piper (`PIPER_BIN` + `PIPER_MODEL`, offline WAV).
  `TTS_PROVIDER` forces one; `TTS_VOICE` / `ELEVENLABS_VOICE` set defaults.
  Output is cached on disk (sha of provider|voice|text, 200 files cap),
  rate-limited to 30 jobs/hour/account (429), kill-switched via
  `tools_tts_enabled` (Admin Center → Features). Preview lines are
  pre-generated once at boot and right after each preview save, so serving
  users always hits the cache (`X-TTS-Cache: HIT/MISS` proves it). Test
  overrides: `ELEVENLABS_BASE_URL` / `OPENAI_BASE_URL`.
- **Settings → Voice** (per browser, localStorage): engine, language
  (auto-detect per message or forced), browser voice/rate/pitch, neural
  voice id, and one-tap previews of the admin-set lines.
- **Admin Center → Features → Voice previews**: the default playback lines
  (`voice_previews` setting, 1–6 lines `{text, lang, rate, pitch, timbre,
  voice}`, each with its OWN voice character — masculine/feminine/any
  timbre hint, speed, pitch, optional server voice id picked from a select
  menu fed by the provider catalog), also served to users via
  `GET /api/tools`. Users can adopt any preview's character as their own
  playback voice from Settings → Voice.

## Personal providers (BYOK)

Users can connect their own `OpenAI` / `Anthropic` / `DeepSeek` / `Gemini`
API keys in **Settings → Providers** (per account, no admin involved).
Connected providers add their models to that user's model menu at the top
(no page refresh needed); usage is billed to the user's own provider
account, and the admin model policy (kill-switch + rate limits) still
applies.

- Keys are stored server-side in `user_provider_keys` and **never returned**
  to any client (`GET /api/me/providers` exposes presence + last 4 chars
  only; never logged). A stored key greys out its field (Replace to rotate,
  Remove to disconnect). Deleting the account wipes its keys.
- Routes: `GET /api/me/providers`, `PUT /api/me/providers/:id`
  (`{apiKey}`, 8–256 chars), `DELETE /api/me/providers/:id` (all Bearer,
  owner-scoped).

## Access-request wishlist (registration approval queue)

When public registration is off, visitors can still reserve a username via
**Request access** on the login page: username + email (required) + a
motivation message (10+ chars). This creates a ticket (`access_requests` +
`access_messages` tables) with a personal page at `/review/<id>` (unguessable
UUID, same pattern as share links).

- Requester: follows the decision, argues their case and replies on the
  `/review` page. Closed tickets (accepted/refused) are read-only.
- Admin: **Access** tab in the Admin Center — filter by status, open a ticket,
  reply (emailed to the requester), and set `pending → reviewing → accepted /
  refused` with a reason. **Accepting creates the account** and returns the
  access key once (also emailed). Terminal tickets can't be reopened.
- Email (needs `SMTP_*`): received-confirmation, reviewing/accepted/refused
  notices, and every admin reply — all HTML, all stating email is
  notification-only and replies belong on the `/review` page.
- Routes: `POST /api/access/request` (public, rate-limited),
  `GET /api/access/ticket/:id` + `POST .../message` (public, ticket bearer),
  `GET /api/admin/access`, `GET/PATCH /api/admin/access/:id`,
  `POST /api/admin/access/:id/message` (admin).
- Live updates ride WebSockets (no polling): `GET /api/access/ws/:id`
  (ticket bearer — pushes `access_message` + `access_status` for that ticket)
  and `GET /api/admin/ws?token=…` (admin Bearer in query string, browsers
  can't set WS headers — pushes `access_message` + `access_status` +
  `access_created` for every ticket plus `report_created` + `report_status`,
  driving the AdminCenter badges and the triage lists). Every event carries
  its full payload (`access_created` bundles the ticket, its first message
  and the message count), so clients apply it straight to state — HTTP
  resync only happens on (re)connect to cover the offline gap. Clients ping
  every 25 s (`{"type":"ping"}` → `{"type":"pong"}`) with a pong watchdog
  that redials half-dead sockets.

## Content reports (flagged AI responses)

Every assistant message carries three actions: **Copy**, **Search the web**
(DuckDuckGo, new tab) and **Report**. Reporting opens a modal with a reason
picker (`copyright`, `gore`, `falseinfo`, `bug`) plus optional details, and
creates a row in the `reports` table with a content snapshot (survives later
edits/deletion of the chat), the triggering user prompt, the model, and the
reporter.

- Reporter: capped at 20 open (`open` + `reviewing`) reports (429 beyond).
  Reporting the same message twice returns the existing report (409
  `already reported`).
- Admin: **Reports** tab in the Admin Center (badge counts open reports,
  excluding shadow-banned reporters) — filter by status, expand for the full
  snapshot + reporter details, and set `open → reviewing → resolved /
  dismissed` with an internal note. An admin can also **shadow-ban** a
  noisy reporter (toggle on the report detail, with confirm): their reports
  are still accepted but flagged unreliable and hidden from the badge.
- Routes: `POST /api/reports` (Bearer), `GET /api/admin/reports`,
  `PATCH /api/admin/reports/:id`,
  `POST /api/admin/users/:id/shadowban` (admin).

## Key recovery via email (optional)

Set the `SMTP_*` vars (and the real public `APP_URL`) to enable it; otherwise
the login page hides the "Forgot your access key?" link.

1. Admin sets a recovery email at account creation (or the user sets their own
   in profile settings).
2. User clicks the link on the login page and enters their username. The API
   always answers `{ok:true}` (no account enumeration) and, if the account has
   an email, sends a one-time `/reset/<token>` link (60 min, single use).
3. Opening the link shows a confirm page; confirming **rotates the key
   immediately** — the new `ka_…` key is displayed once, old sessions die.
   (`admin` itself is excluded: its key lives in `.env`.)

## Local file CDN (avatars)
`PUT /cdn/<ns>/<key>` (Bearer) / `GET /cdn/<ns>/<key>.<ext>` (public), e.g.
`/cdn/avatar/<account-id>.png`. Namespaces live in `CDN_NAMESPACES`
(`src/cdn.ts`) — add one to extend (future usage).

- Allowed types **only** `jpg`/`png`/`webp`, verified by **magic bytes**, never
  by extension (client filename is ignored; the stored extension comes from
  detection). Everything else → `415`, unknown namespace/ext → `404`.
- `avatar` rules: 5 MB max (`413` above), **5 uploads max per 2 h per account**
  (`429` + `Retry-After`), ownership enforced (own id, or admin).
- Served with fixed `Content-Type`, `nosniff`, `Cache-Control: public,
  max-age=3600`. No listing, no traversal (strict charsets + SPA fallback).

## Platform tools & attachments

Uploads always go through platform tools (`GET /api/tools` lists them with
availability). Today: **OCR** (`POST /api/tools/ocr`, tesseract).

- Images (jpg/png/webp, 5 MB max, magic-verified, 20 jobs/hour/account) are
  transcribed **on the backend to TXT** — the model never receives image
  bytes. Text files ride the `text` CDN namespace (UTF-8 validated, 500 KB
  max, keys prefixed with the account id).
- The UI (paperclip menu in the composer) explains this and always shows a
  **pre-transcription preview**: the user reviews/edits the text, then
  attaches it. Attachments travel as labeled text blocks inside the message,
  so they persist in history and work with every model.

## NixOS deployment (`chat.kisakay.com`)

Full step-by-step guide: **[NIXOS-HOSTING.md](./NIXOS-HOSTING.md)**.

`flake.nix` exposes `packages.<system>.default` (backend + prebuilt frontend,
frontend compiled offline from `frontend/bun.lock` via a fixed-output
`bun install`) and `nixosModules.default`.

```nix
imports = [ kisassistant.nixosModules.default ];
services.kisassistant = {
  enable = true;
  package = kisassistant.packages.${pkgs.system}.default;  # required
  domain = "chat.kisakay.com";   # legacy switch: implies nginx + ACME
  enableNginx = true;            # explicit; requires domain
  passwordFile = "/run/secrets/kisassistant-password";  # agenix/sops-nix
  ollamaHost = "http://10.66.66.4:11434";
  extraEnv = { MISTRAL_ENABLED = "true"; };              # optional
};
security.acme.acceptTerms = true;  # required for the ACME/TLS vhost
```

The module sets `NODE_ENV=production`, `CDN_DIR=/var/lib/kisassistant/cdn`
and `DRIVER_DEBUG=false` for the systemd service. `enableNginx` is a
tri-state: `true`/`false` force the bundled reverse proxy on/off, `null`
(the default) keeps the legacy behaviour (nginx iff `domain` is set).

Secrets (`passwordFile`, API keys) must come from files (agenix/sops-nix), never
baked into the Nix store. The service stores SQLite under
`/var/lib/kisassistant/data`; nginx terminates TLS and proxies to
`127.0.0.1:3000`.