Riven Web Index

The platform's own crawl and index of the live web — the corpus that grounded answers and Discover are built from.

Grounded answers need documents, and documents need to be found, fetched, and kept fresh. Web Index is that pipeline: Riven's own crawl rather than a rented search API, which is what allows freshness and ranking to be tuned for answering rather than for browsing.

The pipeline#

1. Discovery — candidate URLs from feeds, sitemaps, link graph expansion, and query-driven fetches.
2. Fetch — polite crawling that honours robots.txt and rate limits per host.
3. Extraction — main content separated from navigation and advertising, so a retrieved passage is the article rather than the chrome.
4. Embedding — passages vectorized for semantic retrieval alongside a lexical index, because both matter.
5. Freshness — recrawl intervals set per source by observed change rate; news is not treated like reference material.

Who consumes it#

• Riven Grounding — retrieval and citation assembly for answers.
• Riven Chat — cited answers to questions that need current information.
• Riven Discover — candidate stories for the daily feed.
• Riven Computer — source material for research tasks.

Sovereign mode#

Because the index is first-party, a sovereign deployment can carry its own corpus and reach the open web through a controlled egress path, or be restricted to internal sources only. Grounding does not require calling out to a third-party search vendor.