Skip to main content

How to Debug a Kapa Answer

Kapa is designed to be the most accurate system for answering technical questions based on your content. Once created, your Kapa instance will only discuss your product. This is achieved through a retrieval augmented generation (RAG) system optimized for technical questions.

Kapa's Performance Depends on Your Content

While Kapa can provide impressive answers, it's crucial to set realistic expectations:

  • Kapa is intentionally limited to answering based only on your content, minimizing inaccuracies.
  • The quality of your documentation directly impacts Kapa's performance.
  • Some limitations exist with retrieval-based systems, such as:
    • Difficulty with time-sensitive questions (e.g., "What are the latest updates?")
    • Challenges with analytics-based queries (e.g., "What are the most common errors?")

This guide will help you diagnose and resolve issues with Kapa's answers. It covers common problems, how to identify the root cause, and steps to improve Kapa's performance.

Step 1: Identify the Root Cause​

When Kapa provides an incorrect or incomplete answer, it's important to understand why this happened. Given kapa is tied to your content you should start by looking at the content that kapa used to generate the answer.

  1. Locate the specific conversation:

    • Go to app.kapa.ai
    • Find the conversation in the /Conversations tab

  2. Analyze sources used in the reply:

    • Click "View Sources" to see the top 10 sources Kapa found
    • Look at the ones marked "Referenced" that Kapa used to draft the answer
View Sources

Step 2: Steps to Improve Kapa​

Once you've identified the root cause, you can take steps to improve Kapa's performance. Given kapa is tied to your content, the best way to improve Kapa in ~90% of cases is to improve your content. Here are some common strategies:

1. Add more content to help Kapa provide better answers:

  • Add missing source: Double check the source you would expect kapa to use exists in the /Sources tab on the kapa plaform. If it is missing, add it.
  • Draft new source: If you would expect kapa to be able to answer a question but it can't, consider drafting a new source that contains the information needed to answer the question.

2. Remove content that can confuse Kapa:

  • Marketing content or blogs: These often contain non-technical language that can interfere with Kapa's ability to provide precise answers.
  • Outdated information: Old documentation, support tickets or GitHub issues can lead Kapa to provide incorrect or obsolete answers.
  • Change logs or release notes: These can overload Kapa with specific details that may not be relevant to most user queries.

3. Update your content to help Kapa learn:

  • Improve your documentation: Based on the sources Kapa used and potential limitation (e.g., lack of clarity), update your documentation to provide more accurate and detailed information.
Tip: Use Kapa To Improve Your Docs

One unexpected benefit of Kapa is that it provides direct user feedback on the quality of your documentation, helping you identify areas for improvement and enhance your overall documentation strategy.

4. Use "Improve This Answer" to steer Kapa's behavior on specific questions

  • Nudge kapa: If Kapa's answer is close but not quite right and you don't want to update the docs, use the "Improve This Answer" feature to provide specific guidance on how Kapa should respond in the future. For a detailed guide on how to use this feature, read the full guide.

Advanced Debugging with Kapa Team (if needed)​

If none of the above steps apply, please reach out to the Kapa team via your shared Slack channel via email.

They can advise on further optimizations and provide advanced debugging assistance.

How to Effectively Report Issues​

When reporting an issue to the Kapa team, please include the following information:

  1. Share the conversation link:

    • Go to the app.kapa.ai and head to the /Conversations tab
    • Open the relevant conversation
    • Click "Share" in the top right corner to copy the link

  2. Explain the expected answer:

    • Describe what Kapa should have answered
    • Specify which ingested content Kapa should have referred to

The Kapa team has additional advanced tools to help diagnose and fix these issues.

Common Debugging Scenarios​

Here are some common scenarios you may encounter when debugging Kapa's answers.

Issue: Broken links in the documentation can appear in Kapa's answers, as it retrieves and presents information as-is without verifying link validity.

Resolution:

  1. Use "View Sources" to find the broken link
  2. Remove or update the link in your documentation

Issues: "Improve This Answer" Not Working​

Issue: You've improved an answer, but Kapa doesn't reflect the changes in subsequent responses.

Resolution:

  1. Ensure you've clicked the "Ingest" button after making improvements
  2. Review your changes as detailed in the Review Your Changes section of our "How to Track and Improve Kapa" guide

Issue: Handling Multiple Versions​

Issue: Managing multiple versions of your product or documentation in Kapa can lead to duplication and reduced answer quality.

Resolution:

  1. Focus on using only the latest version of your documentation in Kapa
  2. Clearly state the version number in document titles or descriptions
  3. If older versions must be included, consider creating separate Kapa instances for each major version
Tip: Focus on the Latest Version

For optimal performance, prioritize the most recent version in Kapa and clearly indicate version information in your content. Note: This is an active area of research for the Kapa team, and we're working on improving multi-version support.