Knowledge indexing

How documents become searchable for sok_dokumenter, and how PDFs on the customer's website are indexed automatically.

#Contextual retrieval

Documents are split into ~1200-character chunks (chunkText). Each chunk is embedded together with its document's title and page (embeddingInput in agent/lib/chunking.ts):

Dokument: Juki DDL-8000C produktark, side 3

<chunk text>

The stored chunk text is unchanged, so answers quote it as before. documents.context_version records how a document was embedded; documents from before (version 0) are re-embedded by the knowledge cron a few at a time from their stored chunks, without reading the file again. Until then, old and new embeddings live side by side and search works for both.

#sok_dokumenter only when there is something to find

agent/tools/sok_dokumenter.ts is a dynamic tool resolved on every turn: tenants without a ready document do not get it at all, so the model never spends a call on an empty knowledge base. The check is cached for a minute per instance.

#PDFs from the website

Setting: Kunnskap → Dokumenter → «Indekser PDF-er fra nettsiden» (pdfIndexing). «Indekser nå» starts a run; with the setting on, a run starts automatically when the last one is a week old.

A run (knowledge_jobs, work list in knowledge_job_items):

1. Discovering. Reads the sitemaps of the site's first host (from robots.txt, else /sitemap.xml, sitemap indexes followed). PDFs listed in the sitemaps are taken as they are. Then every page is read with readPage, which returns the PDFs the page links to. Pages are read section by section (one sitemap = one section), the first pages of every section first; a section whose first 30 pages link no PDF is skipped (thousands of thread colours share one template). Without a sitemap, links are followed from the front page (up to 6000 pages). Pages are restricted to the tenant's site hosts; the PDFs they link to may live elsewhere (amatec.no keeps them on S3), but only on public https hosts. PDFs are deduped by normalised URL.
2. Indexing. Each PDF is downloaded (max 25 MB, redirects to public hosts only), compared with what was indexed before (conditional GET with ETag/Last-Modified, then a SHA-256 of the bytes), and only new or changed ones are extracted and embedded. A PDF becomes a document with source_type: "pdf", source_url = the PDF and origin_url = the page that linked it. A PDF that fails keeps its error on the document (shown in the list) and is retried next run.
3. Done. PDFs the run did not find any more are removed, unless the run found none, or fewer than 30 % of what is indexed (a broken sitemap must not wipe the knowledge base).

The admin polls the run every 2 s while it is active (progress bar: pages read while discovering, PDFs handled while indexing) and can cancel it.

#Running on serverless

Work happens in chunks of ~150 s. The runner holds the job's claim (claimed_at, snakkbar_claim_knowledge_job), releases it at the end of its chunk and nudges POST /api/v1/internal/knowledge-jobs (bearer SNAKKBAR_INTERNAL_SECRET), which continues in a fresh invocation. GET /api/v1/internal/knowledge (Vercel Cron, every 10 minutes, CRON_SECRET) resumes jobs that went quiet for 3 minutes, starts weekly runs and re-embeds old documents. One active job per tenant (unique index).

API (operator token): GET /operator/knowledge/jobs, GET /operator/knowledge/jobs/:id, POST /operator/knowledge/jobs (admin), POST /operator/knowledge/jobs/:id/cancel (admin). SDK: knowledge.jobs(), job(id), startJob(), cancelJob(id).

#amatec.no

A dry run (October 2026): 56 sub-sitemaps, 4 874 pages and 80 PDFs listed in the sitemaps; 1 142 pages read (17 sections skipped), 81 unique PDFs, 62 of them with a linking product page. All PDFs are on amatec-images.s3.eu-north-1.amazonaws.com.

Tests: npm run test:lib (discovery, dedupe, titles, embedding input).