Tune retrieval size

This guide is for teams that have an in-product agent with its own tools and have added Kapa's knowledge base retrieval as a tool call, either via a Hosted MCP server or the HTTP API.

Teams typically do this because agents exposed to customers receive a significant share of questions that native tools alone cannot answer. Documentation retrieval addresses these gaps and makes the agent's existing tools more effective by giving it context about your product. Read more about why this matters in our research on knowledge base search in agents.

This guide covers the first practical decision you face once retrieval is connected: how many results to retrieve, balancing recall against context size. For instructing your agent to use the results well, see Prompt your agent for grounded answers.

Choosing how many results to retrieve

When your agent calls Kapa's retrieval, it receives a ranked list of chunks: short, self-contained snippets of text from documents in your knowledge base. There are three parameters that control how much content is returned:

• use_pruning: Optionally prune low relevance chunks after retrieval, at the cost of added latency. The number of returned chunks becomes variable and may be significantly lower than top_k; pruning always keeps the 2 most relevant chunks where top_k and max_chars permit.
• top_k: The maximum number of chunks to return. Fewer may be returned if max_chars or use_pruning reduce the result set.
• max_chars: Maximum number of characters across all returned chunks. Chunks are included in order of relevance, but only up to the point where the total character count stays within this limit. Chunks are never truncated. This is an upper bound, not a target: especially with use_pruning enabled, the returned total may be well below this limit.

Both max_chars and top_k are enforced independently, so whichever is more restrictive takes effect. Pruning runs before them, attempting to remove the chunks that do not contribute to answering the query; the caps then apply to whatever remains. These are the main knobs you have for balancing retrieval quality against context size.

What is recall?

Recall is a common metric in machine learning that measures the fraction of relevant items that were retrieved. Here it measures how completely your retrieval results cover the information needed to answer a question. There are two useful ways to think about it:

• Chunk recall: Of all the relevant chunks in your knowledge base for a given question, what fraction ends up in the retrieval results? If a question has 5 relevant chunks and the retrieval returns 4 of them, chunk recall is 80%.
• Question recall: For what percentage of questions does the retrieval return every relevant chunk? This is a stricter measure. It tells you how often you get complete coverage.

High recall means the agent has the information it needs to give a complete, accurate answer. Low recall means the agent is working with partial information, which can lead to incomplete or wrong answers.

The tradeoff: recall versus context size

Retrieving more content improves recall, but there are real costs to returning too much. Each additional chunk increases the token count in your agent's context window, which means higher LLM costs and slightly higher latency. As the context grows larger, LLM reasoning quality degrades, a problem known as context rot, which can lead to worse answers even when the relevant information is present.

The key question is finding the right balance between retrieving enough context and keeping the overall context lean. To help you pick, we built a test set of thousands of real questions from across Kapa deployments. Each question is annotated, meaning for each question we know which chunks in the knowledge base are relevant to answering it. The curves below come from running this set at different settings. Three configurations cover most cases; pick by what you are optimizing for.

Maximum recall

Keep the defaults: max_chars at 35,000 and top_k at 15, without pruning. This is the configuration Kapa's own default chat mode runs on, the one you consume through the Website Widget, the Slack Bot, or the Internal Technical Assistant. Pick it when your agent is not overly complicated, so context rot is not much of a problem, and you are not particularly cost-sensitive: it achieves the best answer quality possible. The curves below show why there is little to gain by going higher. The left panel shows chunk recall and question recall at different character limits; the right panel shows how many chunks are typically returned.

At a 35,000-character cap, recall is nearly as high as with no cap at all, while returning an average of 13.7 chunks per query.

The middle ground: pruning

Pick this when your agent is getting complicated and the amount of context that retrieval returns is becoming a concern, but you are not particularly latency-sensitive. Enable pruning and keep the caps at their defaults: on average it drops about two-thirds of the retrieved context while preserving about 96% of recall, at the cost of roughly 0.7 seconds of added latency. Read the research behind it in How we prune RAG context.

Unlike the caps, which cut by size and position, pruning cuts by relevance: a small LLM judges every retrieved chunk against the query and drops those it deems irrelevant, which preserves far more recall for the same reduction.

Reduced context under tight latency

If you need a leaner context but cannot afford the pruner's added latency, lower max_chars instead, with top_k as an additional upper bound. max_chars is the more effective cap because it adapts to varying chunk lengths: short chunks leave room for more results, while long chunks naturally reduce the count. Lowering it adds no latency, but it cuts by size rather than relevance, so the same reduction costs more recall. If you cut two-thirds of the context this way, recall drops to about 86%, where pruning would have kept it at about 96%. Use the curves under Maximum recall to pick your point; below 15,000 characters, recall drops steeply.

These numbers are averaged across many Kapa deployments. Your specific project may look slightly different, but they are a good proxy for the general shape of the tradeoffs.