Tuning knowledge base search for your agent

This guide is for teams that have an in-product agent with its own tools and have added Kapa's knowledge base search as a tool call, either via a Hosted MCP server or the HTTP API.

Teams typically do this because agents exposed to customers receive a significant share of questions that native tools alone cannot answer. Documentation retrieval addresses these gaps and makes the agent's existing tools more effective by giving it context about your product. Read more about why this matters in our research on knowledge base search in agents.

This guide covers the practical decisions you face once retrieval is connected: how many results to retrieve, how to keep context manageable, and how to prompt your agent for trustworthy answers.

Choosing how many results to retrieve

When your agent calls Kapa's retrieval, it receives a ranked list of text chunks, sections of documents from your knowledge base. There are three parameters that control how much content is returned:

• use_pruning: Optionally prune low relevance chunks after retrieval, at the cost of added latency. The number of returned chunks becomes variable and may be significantly lower than top_k; pruning always keeps the 2 most relevant chunks where top_k and max_chars permit.
• top_k: The maximum number of chunks to return. Fewer may be returned if max_chars or use_pruning reduce the result set.
• max_chars: Maximum number of characters across all returned chunks. Chunks are included in order of relevance, but only up to the point where the total character count stays within this limit. Chunks are never truncated. This is an upper bound, not a target: especially with use_pruning enabled, the returned total may be well below this limit.

Both max_chars and top_k are enforced independently, so whichever is more restrictive takes effect. Pruning runs before them, attempting to remove the chunks that do not contribute to answering the query; the caps then apply to whatever remains. These are the main knobs you have for balancing retrieval quality against context size.

What is recall?

Recall is a common metric in machine learning that measures the fraction of relevant items that were retrieved. Here it measures how completely your retrieval results cover the information needed to answer a question. There are two useful ways to think about it:

• Chunk recall: Of all the relevant chunks in your knowledge base for a given question, what fraction ends up in the retrieval results? If a question has 5 relevant chunks and the retrieval returns 4 of them, chunk recall is 80%.
• Question recall: For what percentage of questions does the retrieval return every relevant chunk? This is a stricter measure. It tells you how often you get complete coverage.

High recall means the agent has the information it needs to give a complete, accurate answer. Low recall means the agent is working with partial information, which can lead to incomplete or wrong answers.

The tradeoff: recall versus context size

Retrieving more content improves recall, but there are real costs to returning too much. Each additional chunk increases the token count in your agent's context window, which means higher LLM costs and slightly higher latency. As the context grows larger, LLM reasoning quality degrades, a problem known as context rot, which can lead to worse answers even when the relevant information is present.

The key question is finding the right balance between retrieving enough context and keeping the overall context lean. To help you pick, we built a test set of thousands of real questions from across Kapa deployments. Each question is annotated, meaning for each question we know which chunks in the knowledge base are relevant to answering it. The curves below come from running this set at different settings. Three configurations cover most cases; pick by what you are optimizing for.

Maximum recall

Keep the defaults: max_chars at 35,000 and top_k at 15, without pruning. This is the configuration Kapa's own default chat mode runs on, the one you consume through the Website Widget, the Slack Bot, or the Internal Technical Assistant. Pick it when your agent is not overly complicated, so context rot is not much of a problem, and you are not particularly cost-sensitive: it achieves the best answer quality possible. The curves below show why there is little to gain by going higher. The left panel shows chunk recall and question recall at different character limits; the right panel shows how many chunks are typically returned.

At a 35,000-character cap, recall is nearly as high as with no cap at all, while returning an average of 13.7 chunks per query.

The middle ground: pruning

Pick this when your agent is getting complicated and the amount of context that knowledge base search returns is becoming a concern, but you are not particularly latency-sensitive. Enable pruning and keep the caps at their defaults: on average it drops about two-thirds of the retrieved context while preserving about 96% of recall, at the cost of roughly 0.7 seconds of added latency. Read the research behind it in How we prune RAG context.

Unlike the caps, which cut by size and position, pruning cuts by relevance: a small LLM judges every retrieved chunk against the query and drops those it deems irrelevant, which preserves far more recall for the same reduction.

Reduced context under tight latency

If you need a leaner context but cannot afford the pruner's added latency, lower max_chars instead, with top_k as an additional upper bound. max_chars is the more effective cap because it adapts to varying chunk lengths: short chunks leave room for more results, while long chunks naturally reduce the count. Lowering it adds no latency, but it cuts by size rather than relevance, so the same reduction costs more recall. If you cut two-thirds of the context this way, recall drops to about 86%, where pruning would have kept it at about 96%. Use the curves under Maximum recall to pick your point; below 15,000 characters, recall drops steeply.

These numbers are averaged across many Kapa deployments. Your specific project may look slightly different, but they are a good proxy for the general shape of the tradeoffs.

Prompting for uncertainty and source citations

Two of the most important qualities of a knowledge-base-powered answer are expressing uncertainty when information is missing and citing sources so users can verify claims. These are behaviors that users value highly in out-of-the-box Kapa experiences like the widget, but they are not automatic when you build your own agent. You need to explicitly instruct your agent to produce them.

If you are using the Kapa Agent SDK, these behaviors are already built in. If you are building your own agent with the API or MCP, you will need to add this guidance yourself, in both the tool description and the system prompt.

We think this split applies to any tool: the tool description should tell the model what the tool does and what to expect back from it, while the system prompt should guide how to use the tool and how to handle its results. Here we show what this looks like for knowledge base search specifically.

Tool description

The tool name and description are where you set baseline expectations about what the tool does and what it returns. The name matters because it is the first signal the model uses when deciding which tool to call. If you are using a Kapa Hosted MCP server, both are already configured for you, and you can customize them; see Customizing the tool name and description for when that helps. The default tool name is search_<PRODUCT_NAME>_knowledge_sources and the default description is:

Perform semantic retrieval over the documentation and other knowledge
sources of [Product Name] and return the most relevant chunks for a
given query. A "chunk" is a short, self-contained snippet of text taken
from a single page or item within these sources (for example, part of a
documentation page) and includes its source URL and markdown content.
Chunks are returned in descending order of relevance to the query. If
the knowledge sources do not contain information relevant to the query,
the returned chunks may be only weakly related or entirely unrelated.
Use this tool anytime you need information about [Product Name].

Notice that the description already enforces some of the behavior we want. It tells the model that results include source URLs and content, and critically, that chunks may be weakly related or entirely unrelated to the query. This primes the model to treat results with appropriate skepticism rather than assuming everything returned is relevant.

If you are building your own tool with the HTTP API, use a similar description. The important elements are: explain what the tool returns (chunks with source URLs and content), note that results may not always be relevant, and tell the model when to use it.

System prompt

The system prompt is where you get precise about how the agent should use the tool and handle its results. Below is what we think works well for handling uncertainty and citations. You can use it as a starting point or do your own thing entirely. Keep in mind that different LLMs will behave differently given the same set of instructions:

You are an AI assistant for [Product Name], [product description].

You have different tools to interact with the platform on behalf of the
user. You also have access to a knowledge base search tool to answer the
user's questions and fill in your own understanding of the product.

...your other instructions...

## Knowledge Base Search

Use the search_knowledge_base tool whenever you need information about
[Product Name], whether to answer a user's question directly, to
understand parts of the product in connection with the user's task, or
to interpret results and inputs from other tools. Never rely on your
inherent knowledge or any other tool for product-specific information.

When using the results, follow these rules:

1. Review the content of each knowledge source document carefully and
   assess its relevance to the user query before using it in your answer.
   Some documents may appear relevant at first but are not. If you do not
   find enough information in the knowledge sources to answer the query,
   clearly state that the knowledge sources do not contain enough
   information. Never try to make up an answer.
2. Answer the user query solely based on the relevant knowledge sources.
   Explicitly state your uncertainty with phrases like "I'm sorry, but
   there's no information about", "The knowledge sources do not explicitly
   mention" or similar, if you cannot find enough information in the
   knowledge sources to provide a confident answer. State your uncertainty
   at the beginning of your answer — the user must be made aware of any
   limitations upfront.
3. Cite documents that include a source URL. Format each citation as
   [[short title](URL)] using a 1-4 word title. Place the citation
   immediately after the sentence it supports. If two or more documents
   support the same sentence, include at most two citations inside the
   same brackets separated by a semicolon.