Crawler: HostnameAliases

About this parameter

Defines mappings to replace given hostname(s).

Sometimes certain URLs are accessible from multiple domains, for example, due to a dynamic environment change (like staging vs. production) or hosting services like Netlify. You can use this parameter to define key-value pairs to dynamically replace one hostname with another when a site is being crawled. The key is the hostname to replace, and value is the hostname to replace it with.

Keys are exact matches. This parameter doesn’t use micromatch unlike other parameters such as discoveryPatterns.

The hostname is replaced when a URL with a matching hostname is found. URLs can be encountered via links in a sitemap, on a page, via canonical URLs, or redirects. For example, if you create a mapping for { "dev.example.com": "example.com" } and the crawler encounters https://dev.example.com/solutions/voice-search/, the crawler transforms this URL to https://example.com/solutions/voice-search/. If appropriate, records are extracted, and the crawler continues from the transformed URL, not the original version.

The mapping doesn’t apply to URLs listed in the startUrls, siteMaps, pathsToMatch or other configuration variables.

The mapping also doesn’t apply to the content extracted from a page. For example, if you extract text that includes the hostname to be replaced, the original text is extracted without making a replacement.

Examples

1
2
3
4
5
{
  hostnameAliases: {
    'dev.example.com': 'example.com'
  }
}