Troubleshooting and improving Kapa's answers

This guide helps you understand why Kapa might provide incorrect or incomplete answers and walks you through a systematic process to diagnose issues and enhance response quality. You'll learn how to identify the root causes of problematic answers and implement the most effective solutions to fix them.

Understanding how Kapa generates answers

Kapa is designed to be the most accurate system for answering technical questions based on your content. It uses a retrieval augmented generation (RAG) system that:

1. Searches your knowledge base for relevant information
2. Retrieves the most pertinent sources
3. Generates answers based only on those sources

Kapa's performance depends on your content

While Kapa can provide impressive answers, it's important to set realistic expectations:

• Kapa is intentionally limited to answering based only on your content, minimizing inaccuracies
• The quality of your documentation directly impacts Kapa's performance
• Some limitations exist with retrieval-based systems, such as:
  • Difficulty with time-sensitive questions (e.g., "What are the latest updates?")
  • Challenges with analytics-based queries (e.g., "What are the most common errors?")

Step 1: Identify the root cause

When Kapa provides an unsatisfactory response, the first step is to understand why this happened by examining the sources Kapa used.

1. Locate the specific conversation:

   • Go to the Kapa platform
   • Open Conversations and find the specific conversation

2. Analyze sources used in the reply:

   • Click "View Sources" to see the top sources Kapa found

     

   • Look at the ones marked "Referenced" that Kapa used to draft the answer

3. Identify the most likely cause:

   • Missing information: Kapa couldn't find relevant content in your knowledge base
   • Incorrect information: Kapa found and used outdated or inaccurate content
   • Retrieval issues: Kapa found related but not ideal content
   • Interpretation problems: Kapa retrieved good sources but misinterpreted them

Step 2: Improve your knowledge sources

In most cases, the best way to improve Kapa's answers is to enhance your content. This is the most sustainable approach as it addresses the root cause rather than applying a targeted fix for a single question type. Here are effective strategies:

Add missing content

• Check if sources exist: Verify that the documents you'd expect Kapa to use are in the /Sources tab
• Add new sources: Upload documentation that contains the information needed to answer similar questions
• Create targeted content: Draft new documentation specifically addressing gaps identified through user questions

Remove confusing or outdated content

• Marketing content or blogs: These often contain non-technical language that can interfere with Kapa's ability to provide precise answers
• Outdated information: Old documentation, support tickets, or GitHub issues can lead to incorrect answers
• Change logs or release notes: These can overload Kapa with specific details that may not be relevant to most queries

Update existing content

• Improve clarity: Rewrite sections that may be ambiguous or overly complex
• Add missing details: Expand documentation to cover edge cases or common questions
• Restructure for better retrieval: Format content to help Kapa better identify key information

Use Kapa to improve your docs

One benefit of Kapa is that it provides direct user feedback on the quality of your documentation, helping you identify areas for improvement and enhance your overall documentation strategy. Refer to the Analytics section for more details about the insights you can get with Kapa.

Step 3: Use "Improve This Answer" for specific cases

Consider using the "Improve This Answer" feature in these scenarios:

• When you need a quick fix while source updates are in progress
• For edge cases that don't warrant changes to main documentation
• When handling unique questions that rarely occur but need accurate answers
• As a temporary solution while more comprehensive documentation is being developed

Here's how to use this feature:

note

The "Improve This Answer" feature effectively creates a new data source for Kapa. While useful for immediate fixes, remember that this doesn't address underlying issues in your primary knowledge sources.

Creating a correction

1. Open the conversation that needs improvement

2. Click the Improve This Answer button below the response

3. In the improvement modal:

   • Review the original question and answer
   • Replace the answer text with your improved version
   • Check or uncheck "Could be answered with current knowledge" as appropriate

4. Click Submit to save your correction

Managing corrections

1. Navigate to Sources in your Kapa dashboard

2. Find the Corrected Responses source in the list and click Change Setup

   

3. In the list of corrections, find the entry you want to modify and:

   • Click Edit to change the correction
   • Click Delete to remove it

4. Review and approve the changes to deploy them

Common troubleshooting scenarios

Here are specific issues you might encounter and how to address them effectively:

Broken links in answers

Issue: Kapa includes broken links in answers, as it retrieves and presents information as-is without verifying link validity.

Resolution:

1. Use "View Sources" to find the broken link
2. Remove or update the link in your documentation

"Improve This Answer" not working

Issue: You've improved an answer, but Kapa doesn't reflect the changes in subsequent responses.

Resolution:

1. Ensure you've completed the review process after making improvements
2. Verify the question is similar enough to trigger the correction
3. Wait a few minutes for the knowledge index to update

Multiple product versions causing confusion

Issue: Managing multiple versions of your product or documentation in Kapa can lead to duplication and reduced answer quality.

Resolution:

1. Focus on using only the latest version of your documentation in Kapa
2. Clearly state the version number in document titles or descriptions
3. If older versions must be included, consider creating separate Kapa instances for each major version

Focus on the latest version

For optimal performance, prioritize the most recent stable version in Kapa and clearly indicate version information in your content.

Advanced debugging with the Kapa team

If standard troubleshooting doesn't resolve your issues, reach out to the Kapa team via your shared Slack channel or email.

How to effectively report issues

When reporting an issue to the Kapa team, include:

1. Share the conversation link:

   • Go to the Kapa platform and open Conversations
   • Open the relevant conversation
   • Click "Share" in the top right corner to copy the link

2. Explain the expected answer:

   • Describe what Kapa should have answered
   • Specify which ingested content Kapa should have referred to

The Kapa team has additional advanced tools to help diagnose and fix complex issues.

Next steps

After implementing improvements to your content or using the "Improve This Answer" feature:

1. Test similar questions to verify Kapa now provides better answers
2. Review related content that might benefit from similar improvements
3. Implement conversation review best practices to establish a regular process for monitoring and improving Kapa's performance over time

Remember that improving Kapa's answers is an iterative process that leads to continuously better documentation and user experiences.