Skip to main content

Choose a retrieval mode

This guide is for anyone who has connected Kapa's retrieval to an agent and now has to decide which retrieval mode it should run, and then how much to let it return. The agent may be one you built into your product, a support workflow, a coding assistant reading your documentation, or a pipeline that needs grounded context for each step. The choice is the same whether you connect through the Hosted MCP server or the HTTP API: both accept the same mode parameter and the same caps.

An in-product agent is the typical case: a large share of the questions it receives cannot be answered by its native tools, and retrieval from your knowledge sources closes that gap. Read more in our research on knowledge base search in agents.

The comparison between the modes in this guide is based on runs of the Company Knowledge Bench, our internal benchmark of 1,000 queries taken from production, each annotated with the chunks of the knowledge base that answer it.

Decision factors​

Three factors separate the modes. The right choice depends on which one matters most to your agent.

Recall. How much of what the answer needs retrieval returns. The benchmark measures it per query as all or nothing: a query counts only if the returned chunks fully answer it, and where several sources could answer, only if the chunks come from the authoritative, current one. Recall is the share of queries that pass. Recall matters because your agent cannot see a chunk that was not retrieved. A missed chunk leads to an incomplete or wrong answer, and nothing the agent does afterward can fix it.

Precision. How much of what retrieval returns the answer needs. The benchmark measures it per query as the share of the returned chunks that are needed to answer it. If retrieval returns ten chunks and three of them are needed, precision is 30%. This is a strict measure. It does not mean the other seven are unrelated to the query, only that they are not needed: they may contribute background, repeat what a needed chunk already says, or come from a less authoritative source. Precision matters because everything that is retrieved ends up in your agent's context. Every additional chunk costs tokens on every call to your model, so lower precision means higher inference cost on your side. And as the context fills with material that is irrelevant to the query, the model reasons worse, a problem known as context rot. This matters most for complex agents with many tools, where retrieval results compete with everything else in the context.

Latency. How long a call takes. Some agents care about the typical call, because a user is waiting for the answer. Others care about the slowest calls, because a step that runs too long hits a timeout or holds up everything queued behind it.

The modes in numbers​

The table below shows the Company Knowledge Bench results for both modes. The blog post describes the method in full. All runs use the default caps. Your project may differ, depending on your sources and the queries your agent sends, but the shape of the comparison holds.

defaultdeep
Recall0.610.65
Precision12%26%
Tokens returned, average9,9005,100
Time per query, average3.3 seconds5.1 seconds

The deep mode differs from the default mode on each measure for these reasons:

  • Recall is higher because the deep mode keeps gathering until it is confident it has enough.
  • Precision is higher because the deep mode reads everything it gathered and drops what it judges irrelevant.
  • Fewer tokens are returned for the same reason. The default mode always returns as many chunks as its caps allow.
  • Time per query is higher because the deep mode reasons more and has more tools to call. On hard queries it is allowed to search more and for longer.

How retrieval works explains both modes in full.

When to use the deep mode​

We recommend the deep mode for every agent that can tolerate its latency. It scores highest on recall and precision, and because it returns fewer chunks, your agent's context stays lean and each call to your model costs less. A smaller, more relevant context is also an easier task to reason over, which in many scenarios lets you run a simpler model for your agent than you otherwise could.

The price is higher latency. If your agent can absorb that, the deep mode gives you the best results.

Leave top_k and max_chars at their defaults. The deep mode decides how much to return by relevance, and the caps are a ceiling it rarely reaches.

When to use the default mode​

The default mode is a latency tradeoff. It gives you strong results, faster. Use it when your agent is latency constrained and cannot absorb the deep mode's extra seconds on a typical query.

Use it as well when you need less variance in latency. The default mode's latency varies less from query to query, so if what you care about is the slowest calls rather than the typical one, this is the mode to use.

Managing context size​

In the default mode, the two caps are the only lever you have over context size. Both apply, and whichever is more restrictive takes effect:

  • top_k: the maximum number of chunks to return, 15 by default.
  • max_chars: the maximum number of characters across all returned chunks, 35,000 by default. Chunks are included in order until the next one would exceed the limit; they are never truncated.

Leave them at their defaults for the most recall; raising them gains almost nothing. To cut context, lower max_chars, with top_k as an additional upper bound. max_chars is the more effective cap because it adapts to varying chunk lengths: short chunks leave room for more results, while long chunks naturally reduce the count. Lowering it adds no latency, but it cuts by size rather than relevance, so every cut costs recall. At the default cap and two tighter settings:

max_charsChunks returned, averageRecall
35,000 (default)13.70.61
30,00012.60.59
25,00011.10.56

Below that, recall drops steeply. If you are looking for a meaningful reduction in context, use the deep mode if you can.

Pruning​

use_pruning still exists in the default mode, but it is deprecated and no longer recommended. It adds a step after retrieval in which a small LLM judges every retrieved chunk against the query and drops those it deems irrelevant, always keeping the 2 most relevant chunks. On average it drops about two-thirds of the retrieved context while preserving about 96% of recall, at roughly 0.7 seconds of added latency. Requests that combine it with the deep mode are rejected, because the deep mode does this as part of retrieval. Read the research behind pruning in How we prune RAG context.

If you use pruning today, or are considering it, switch to the deep mode instead.