Tools / Crawler / APIs / Configuration

Crawler: IgnoreRobotsTxtRules

Type: boolean
Default: false
Parameter syntax 
ignoreRobotsTxtRules: true|false
See code examples

About this parameter

When set to true, the crawler will ignore rules set in your robots.txt.

Examples

Copy 
1
2
3

{
  ignoreRobotsTxtRules: true,
}
Previous
sitemaps
Next
ignoreNoIndex
Did you find this page helpful?