How data ingestion works

Kapa is a retrieval-augmented generation (RAG) system. When someone asks a question, Kapa searches an index of your content for the most relevant passages and uses them to generate an answer grounded in your own knowledge sources, with citations. If you build on Kapa, your agent or application can also call this retrieval directly and work with the passages themselves. Either way, the quality of every answer starts with the quality of that index: it must contain everything relevant, and it must be up to date.

Building and maintaining that index is the job of the ingestion layer. It assembles a unified knowledge base from 20+ types of sources: documentation sites, ticketing systems, community threads, PDFs, API specifications, and more, and keeps it constantly up to date as that content changes, for most sources within minutes of the change.

The design of everything below follows from a single chain of reasoning: indexing content is expensive, so re-processing everything on every refresh is unaffordable, so Kapa must sync only what changed. And syncing only changes is itself a hard problem, because the systems your content lives in are rarely able to tell you what changed. This page walks through that chain.

Re-indexing everything is too expensive

The simplest way to keep a source current is to replace it wholesale: every time you want to refresh, pull all of the source's content, process it, and swap the result into the index. This is how most retrieval pipelines start, because it is beautifully stateless: nothing to track, nothing to reconcile, whatever exists upstream is what ends up indexed.

The problem is the cost of that processing step:

• Text needs to be embedded. Every single piece of text is run through an embedding model to become semantically searchable, a per-item cost that scales with how much content you process.
• PDFs need to be converted. A PDF must be converted into clean, structured text before it can be indexed at all, an expensive per-document operation, and knowledge bases routinely contain thousands of them, sitting in storage buckets or hosted across websites.
• Images need to be annotated. Technical documentation is full of system diagrams, schematics, and architecture charts, and a single project can contain tens of thousands. Kapa runs them through a cascade of increasingly expensive steps: cheap heuristics discard decorative images first, a lightweight model then judges which of the remaining images carry useful information, and only those are annotated with vision models so their content can be retrieved. Read more in How we index images for RAG.

With full replacement, you pay all of these costs for the entire source on every refresh, even though the overwhelming majority of the content did not change since the last one. Keeping a knowledge base fresh this way means re-embedding, re-converting, and re-annotating everything, over and over, to capture a handful of edits. At any real scale the cost is prohibitive: with this approach, even a weekly refresh cadence is often too expensive, let alone updates within minutes. And cost aside, it is slow: fully re-indexing a large source takes hours, so a ten-minute refresh cycle is not just unaffordable, it is impossible. The approach sets a hard ceiling on freshness, cost, and scale all at once.

The only viable design is a pipeline that identifies exactly what changed and re-processes only that.

That decision has two consequences. First, the pipeline must be stateful. A system that replaces everything can be ignorant of history. A system that syncs changes must remember what it has fetched, what has changed since, and what is currently indexed, for every item in every source, and it must keep that bookkeeping correct even when things fail mid-update.

Second, syncing only changes in content requires being able to detect changes, and the systems where the knowledge is stored typically make that very difficult. There is no generic answer: each source type needs its own change detection strategy, built around what its API can and cannot report, and web crawling, where there is no API at all, gets a change detection pipeline of its own.

Syncing only what changed

For every source, Kapa continuously monitors the external system for changes, fetching them in small batches and always remembering where it left off. This is where the unevenness of upstream systems is absorbed. Each connector is built around what its API can actually answer: some can report every change since a timestamp, many cannot report deletions at all, and some can barely support synchronization. Slack, for example, offers no way to ask what changed and enforces tight rate limits, so Kapa re-scans the last seven days of threads on a rolling basis instead (Refreshes describes the tradeoff this makes).

Each source also runs on several schedules. New and updated content can usually be queried cheaply, so it is picked up every few minutes. Deletions are different: most APIs cannot be asked what was removed, so detecting them means comparing what exists upstream against what is in the knowledge base, a far more expensive check that runs as its own, less frequent pass. The effort is worth it, because an assistant that keeps answering from deleted content is worse than one that is briefly behind. When a source is first connected, the same machinery fetches the full history batch by batch, then switches to tracking changes.

Everything downstream runs only on what fetching identified as changed. An edited documentation page is re-converted and re-embedded while the thousands of pages around it stay untouched; a new PDF is converted once and an unchanged one never again; an image is re-annotated only when it changes. The result then replaces the previous version of that content in the search index.

Web crawling: synchronization without an API

Websites are the extreme case: there is no API, so there are no diffs to fetch. Kapa has to produce the diff itself, and it has to do so safely.

Web crawling therefore runs as a change detection pipeline. Sites are re-crawled daily. Each crawled page's current content is compared against what is already indexed, ignoring purely cosmetic changes in the HTML. Every page is classified as new, modified, unchanged, or deleted, with deletions detected by comparing the crawl against the known set of pages.

Then comes the part that makes constant crawling safe to leave unsupervised. Changes fetched from an API can be applied automatically with confidence: the API is a stable contract, and what it returns is what the system of record contains. A website carries no such guarantee. It belongs to you, and you change it: sites get redesigned, documentation trees get moved, build tooling gets replaced, and any of these can silently break a configured crawl. A simple crawler that blindly syncs whatever it finds would push that breakage straight into your production assistant. So Kapa judges every crawl before applying it:

• If a large share of a site's pages changed or disappeared in a single crawl, nothing is applied and the crawl is routed to human review.
• A large spike in new pages relative to existing ones trips the same gate, since it usually indicates a restructure or a crawl configuration problem rather than new content.

Day-to-day edits touch a handful of pages and flow through automatically. The threshold is deliberately conservative: a false alarm costs a moment of review, while a corrupted knowledge base costs your users. Approved changes are then processed and indexed like changes from any other source.

What you see as a result

In the platform, all of this surfaces as a few quiet facts: the Sources view shows when each source last checked the upstream system for changes, crawls that need attention wait for your approval, and answers cite content that is minutes rather than weeks old.