Index your first source

In this quickstart, you add your first source to a Kapa project and watch it become searchable. No code is involved. It is intended for developers evaluating agentic retrieval and assumes no prior Kapa experience.

By the end, you will have:

• A website crawled, reviewed, and fully indexed in a Kapa project.
• An index ready to search in the next quickstart.

The quickstart uses the Website Crawl source because it needs nothing but a public URL, where most other connectors require an account on an upstream system, which would make this more complicated. It crawls the Kapa documentation so you can follow along exactly as shown, but any site works, including your own. Kapa supports 20+ other source types, including ticketing systems, community threads, and file uploads.

Definitions

• A project is one search space: when you perform retrieval, you pick a project to retrieve over, and you search everything indexed in this project.
• A source belongs to exactly one project, and a project can have many sources. Each source configures, through one of the prebuilt connectors, how to ingest knowledge from one specific place: a Website Crawl source fetches and keeps in sync one website, a GitHub Issues source does the same for the issues of one repository.

Before you start

Before you start, you should have:

• A Kapa account.
• Permission to edit the project's sources. If the Add new source button is greyed out, ask a team admin to grant it.
• A public website to crawl. The examples use the Kapa documentation at https://docs.kapa.ai; you can follow along with it or bring your own site.

Create the source

1. Open app.kapa.ai and select your project.
2. Navigate to Sources (under Configuration in the sidebar) and click Add new source.
3. Give the source a name, pick Website Crawl as the source type, and click Continue.

This opens a two-step setup wizard: first you run a preview crawl to configure which pages of the website to include, then you select which part of each page to actually use, so navigation bars, sidebars, and footers do not clutter the index. The next two sections walk through one step each.

Configure the crawl

1. Enter the Start URL, in this walkthrough https://docs.kapa.ai. The crawler discovers pages by following links and only keeps URLs nested under the start URL, so a start URL like https://example.com/docs/ scopes the crawl to the documentation and keeps the rest of the site out of the index.
2. Click Crawl. This runs a preview without saving anything: the Accepted tab lists the pages that will be indexed once the setup completes.

3. Check the Skipped tab as well: these are pages the crawler also found but will not include, grouped by the reason each was skipped. The groups cover both pages your configuration excludes and pages that encountered errors during the crawl. Skips are usually correct behavior: the expanded Outside included URLs group below holds all the URLs that are linked from the Kapa documentation but are not part of it, which is exactly what the start URL scoping is supposed to keep out.

4. The Website Crawl is a powerful tool with many more configuration options for more complicated websites and setups, tucked away under Advanced configuration and documented on the Website Crawl connector page. For example, if the accepted list is empty or misses pages you expected, your site probably renders its content with JavaScript: enable Render JavaScript and crawl again.
5. Click Save and continue.

Select the content

The second step of the wizard lets you try content selection on the pages you just crawled: you pick a selector, and the preview shows the content it selects from each page as Markdown, the format in which it will be indexed. Your goal is clean Markdown, meaning the actual content is all there and the unnecessary visual elements you do not want in it, such as navigation bars, sidebars, and footers, are gone.

1. Enter a Selector: the CSS or XPath selector of the main content element. Look for one that works across the pages you just crawled, not just on a single page. For the Kapa documentation, a Docusaurus site, it is article .theme-doc-markdown; on most modern documentation sites main or article works. Find the right one by right-clicking the text of a page and inspecting it with your browser's developer tools.
2. Click Convert. Page through the Source Preview and check that the Markdown stays clean across several pages.

3. Click Save. This concludes the preview and starts the real crawl with the configuration you just put together.

Review and ingest

Before your new source is committed to production, it is staged for a final review by you. Whatever enters the index ends up in answers, so this checkpoint exists to make sure you know exactly what goes to production. Here you can see every page you crawled and its staged content, exactly as it will be ingested. It is the same Markdown the preview showed during setup, but for every page in one go: the complete diff of what is about to go to production.

1. In the sources table, the new source shows a Needs review badge with the number of pages waiting once the crawl completes. Click Review.

2. Look over the listed pages.
3. Click Ingest to approve the pages and start indexing.

Wait for ingestion

Back in the sources table, the source row shows the ingestion progress: the count of pages already deployed to your knowledge base ticks up while the rest are being deployed in the background. A typical documentation site of a few hundred pages completes within minutes; large sites take longer.

When the spinner disappears, ingestion is finished: the row shows the final page count, and every one of those pages is now indexed and searchable.

Next steps

Your knowledge base is live. Continue with Connect an AI agent to your knowledge to let an agent search what you just indexed and ground its answers in it.

• HTTP API: call retrieval directly and get the raw chunks as JSON.
• Knowledge sources: the full connector catalog, refreshes, and source management.