Skip to main content

Website Crawl (Firecrawl)

Connect a website as a data source to automatically crawl and ingest page content into your Knowledge. This is useful for bringing external documentation, knowledge bases, or public web content into Revelry Ai for use in chats and artifact generation.

How It Works​

The website crawl data source uses Firecrawl to:

  1. Start from a URL you provide
  2. Crawl linked pages within the same domain
  3. Extract page content (text, headings, structure)
  4. Ingest the content as documents in your Knowledge
  5. Make the content searchable via MCP tools in chat

Setting Up a Website Crawl​

  1. Navigate to Knowledge > Sources > New Data Source
  2. Select Website Crawl as the data source type
  3. Enter the starting URL — the crawl will begin here and follow links
  4. Set the maximum pages to crawl (controls how deep the crawl goes)
  5. Click Connect to start the crawl

Configuration Options​

  • Starting URL — The root page where crawling begins
  • Max Pages — Limits how many pages are crawled. Start with a smaller number to test, then increase as needed

Sync Status​

After creating the data source, you can track the crawl progress:

  • Syncing — The crawl is in progress
  • Synced — All pages have been crawled and ingested
  • Error — Something went wrong (check the URL and try again)

You can re-sync the data source to pick up content changes on the website.

Permissions​

Website crawl data sources respect group permissions like any other data source. When creating the data source, you can scope it to specific projects or make it organization-wide.

Use Cases​

  • Documentation sites — Ingest product docs, API references, or help centers
  • Competitor research — Crawl public competitor pages for analysis
  • Knowledge bases — Bring in wiki or knowledge base content
  • Industry resources — Ingest standards documents or regulatory content

Tips​

  • Start with a small max pages limit to preview what gets crawled before doing a full ingest
  • The crawler follows links within the same domain — it won't crawl external links
  • Re-sync periodically to keep content up to date with the source website

Next Steps​

  • Documents — View crawled content in your Knowledge
  • Collections — Organize crawled documents into collections
  • MCP Tools — Use crawled content in chat conversations