Skip to main content

Extract clean content

The Parse step turns each crawled page into markdown, and its configuration is a selection job: you are selecting and removing elements from the page's HTML until Kapa's converter can turn it into clean markdown. The parse preview on the right shows the result for every page, and you keep going until that content renders as clean markdown: all of the text, none of the clutter.

Before you start​

  • You have a Website crawl source with a completed preview crawl, so the Parse step has pages to convert.

Find the main content element​

  1. Open one of your pages in the browser, right-click the main content, and choose Inspect.

A documentation page with the browser context menu open and Inspect highlighted

  1. In the developer tools, walk up the element tree until you find the element that encloses all of the content and as little else as possible; whatever else it drags in, the exclusion options below can remove. Look for a <main> or <article> tag, or a container with a class like content or theme-doc-markdown; on most documentation sites, main or article is the answer.

The developer tools with the article element selected, highlighting the page&#39;s main content

One page is usually enough to inspect: websites are generated by the same build system or theme for every page, so a selector that is right for one page is largely right for all of them.

Test it in the parse preview​

  1. On the Parse step, enter the selector into Content selector. The preview converts as you type.
  2. Page through the previewed pages with Previous page and Next page and confirm the selector holds across page types: an article, an index page, an API page.
  3. If a page comes up empty, the selector did not match there. Try a more general selector.
  4. Use the Raw tab to compare the converted markdown against the unprocessed page when you suspect content is missing.

Remove elements that clash with the markdown​

Even with the right outer content selector, stray elements often survive inside it and clash with the markdown. Some are visible clutter such as navigation, banners, or feedback widgets; others are invisible on the rendered page, such as a hidden table of contents label, and only show up in the converted output. Here, an element renders into the top-level header:

The parse preview with a stray element rendered into the top-level header

Remove them under the Advanced tab:

  1. Add the CSS selectors of unwanted elements to Exclude selectors, for example .breadcrumbs.
  2. Add unwanted CSS classes to Exclude classes, for example sidebar-nav.

Find the offending element's class or selector the same way as the content selector: inspect it in the browser. After the exclusion, the preview re-converts and the clash is gone:

The same page in the parse preview rendering as clean markdown

Verify​

Page through the preview once more: every page shows complete content as clean markdown, with no menus, sidebars, footers, or stray fragments. When it does: on a new source, publish it with Deploy; on an existing source, apply the new configuration with Re-deploy.