Verify your domains
Algolia’s Crawler is a “robot” that can access, discover and extract data.
By verifying a domain, you allow the Crawler to browse the content of your verified sites. Verifying your domains ensures that your organization owns the crawled data and avoids disputes about the legality of the Crawler’s activity.
Verifying a domain
To verify a domain, you must change your site’s “robots.txt” file.
- Go to the Crawler’s domains page and find your Algolia application ID
- Click the icon to the right of How to verify domain? to copy the verification code to your clipboard. Only people who are team members of your Algolia application with
Configurepermission on the Crawler can verify this code.
- Paste the verification code anywhere in your robots.txt file.
- Click Add new domain and fill in the details (App ID and Domains and subdomains) to add your sites to the Algolia application.
- Click the Verify now button.
The verification code is a unique string for each application. Having access to the code poses no risk, so you shouldn’t worry about exposing it publicly in robots.txt.
If you reset the verification code, all previously verified domains will be invalidated.
You must verify all subdomains
The Crawler doesn’t allow wildcards like *.algolia.com, domains must be an exact match. For example, if you want to crawl algolia.com and all its subdomains, you must individually add the domains: algolia.com, www.algolia.com, and support.algolia.com.
If you wish, contact the Algolia support team and ask them to verify your subdomains for you.
Troubleshooting: Impossible to verify a domain
If you can’t verify your domain, for instance, the platform you are using prevents you from updating robots.txt or your website isn’t at the root level, contact the Algolia support for help.
