> ## Documentation Index
> Fetch the complete documentation index at: https://docs.onyx.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Web

> Index public or internal web pages

## How it works

The Web Connector scrapes sites based on a base URL.

* It only indexes files from the same domain and containing the same base path.
* It will index pages reachable via hyperlinks from the base URL.
* The text contents are cleaned up via some heuristics and some metadata such as the page Title
  is extracted.

## Setting up

### Authorization

* As long as the page is reachable, no additional authorization is necessary.

### Indexing

<Steps>
  <Step title="Open Web connector">
    Navigate to the Admin Panel and select the **Web** Connector.
  </Step>

  <Step title="Enter base URL and index">
    Input the base URL to index and click on Index.

    <img className="rounded-image" src="https://mintcdn.com/danswer/24Ocig51qMqahMaT/assets/admins/connectors/web/WebConnector.png?fit=max&auto=format&n=24Ocig51qMqahMaT&q=85&s=337179d7fb264ffa59e6376b1ce155ca" alt="Onyx Web connector form to enter base URL for indexing" width="1895" height="1254" data-path="assets/admins/connectors/web/WebConnector.png" />
  </Step>
</Steps>

To see the status of the indexing, visit the Connectors Status page (top left).

<img className="rounded-image" src="https://mintcdn.com/danswer/24Ocig51qMqahMaT/assets/admins/connectors/web/WebConnectorStatus.png?fit=max&auto=format&n=24Ocig51qMqahMaT&q=85&s=21c6f8b78cad6c416717880f8edd3d8a" alt="Onyx Connectors Status page showing web crawl progress" width="2491" height="989" data-path="assets/admins/connectors/web/WebConnectorStatus.png" />