# Knowledge indexing

How documents become searchable for `sok_dokumenter`, and how PDFs on the customer's website are
indexed automatically.

## Contextual retrieval

Documents are split into ~1200-character chunks (`chunkText`). Each chunk is embedded together
with its document's title and page (`embeddingInput` in `agent/lib/chunking.ts`):

```
Dokument: Juki DDL-8000C produktark, side 3

<chunk text>
```

The stored chunk text is unchanged, so answers quote it as before. `documents.context_version`
records how a document was embedded; documents from before (version 0) are re-embedded by the
knowledge cron a few at a time from their stored chunks, without reading the file again. Until
then, old and new embeddings live side by side and search works for both.

## `sok_dokumenter` only when there is something to find

`agent/tools/sok_dokumenter.ts` is a dynamic tool resolved on every turn: tenants without a ready
document do not get it at all, so the model never spends a call on an empty knowledge base. The
check is cached for a minute per instance.

## PDFs from the website

Setting: **Kunnskap → Dokumenter → «Indekser PDF-er fra nettsiden»** (`pdfIndexing`). «Indekser
nå» starts a run; with the setting on, a run starts automatically when the last one is a week old.

A run (`knowledge_jobs`, work list in `knowledge_job_items`):

1. **Discovering.** Reads the sitemaps of the site's first host (from `robots.txt`, else
   `/sitemap.xml`, sitemap indexes followed). PDFs listed in the sitemaps are taken as they are.
   Then every page is read with `readPage`, which returns the PDFs the page links to. Pages are
   read section by section (one sitemap = one section), the first pages of every section first;
   a section whose first 30 pages link no PDF is skipped (thousands of thread colours share one
   template). Without a sitemap, links are followed from the front page (up to 6000 pages).
   Pages are restricted to the tenant's site hosts; the PDFs they link to may live elsewhere
   (amatec.no keeps them on S3), but only on public https hosts. PDFs are deduped by normalised
   URL.
2. **Indexing.** Each PDF is downloaded (max 25 MB, redirects to public hosts only), compared with
   what was indexed before (conditional GET with `ETag`/`Last-Modified`, then a SHA-256 of the
   bytes), and only new or changed ones are extracted and embedded. A PDF becomes a document with
   `source_type: "pdf"`, `source_url` = the PDF and `origin_url` = the page that linked it. A PDF
   that fails keeps its error on the document (shown in the list) and is retried next run.
3. **Done.** PDFs the run did not find any more are removed, unless the run found none, or fewer
   than 30 % of what is indexed (a broken sitemap must not wipe the knowledge base).

The admin polls the run every 2 s while it is active (progress bar: pages read while discovering,
PDFs handled while indexing) and can cancel it.

### Running on serverless

Work happens in chunks of ~150 s. The runner holds the job's claim (`claimed_at`,
`snakkbar_claim_knowledge_job`), releases it at the end of its chunk and nudges
`POST /api/v1/internal/knowledge-jobs` (bearer `SNAKKBAR_INTERNAL_SECRET`), which continues in a
fresh invocation. `GET /api/v1/internal/knowledge` (Vercel Cron, every 10 minutes, `CRON_SECRET`)
resumes jobs that went quiet for 3 minutes, starts weekly runs and re-embeds old documents. One
active job per tenant (unique index).

API (operator token): `GET /operator/knowledge/jobs`, `GET /operator/knowledge/jobs/:id`,
`POST /operator/knowledge/jobs` (admin), `POST /operator/knowledge/jobs/:id/cancel` (admin).
SDK: `knowledge.jobs()`, `job(id)`, `startJob()`, `cancelJob(id)`.

### amatec.no

A dry run (October 2026): 56 sub-sitemaps, 4 874 pages and 80 PDFs listed in the sitemaps; 1 142
pages read (17 sections skipped), 81 unique PDFs, 62 of them with a linking product page. All
PDFs are on `amatec-images.s3.eu-north-1.amazonaws.com`.

Tests: `npm run test:lib` (discovery, dedupe, titles, embedding input).