Type
RobotsConfig
From mothwing ยท src/config.js:74
What robots.txt turns away.
Properties
blockoptional | string[] | Crawlers turned away from the whole site, by name |
|---|---|---|
disallowoptional | string[] | Paths kept out of every crawler |
Type
RobotsConfigFrom mothwing ยท src/config.js:74
What robots.txt turns away.
blockoptional | string[] | Crawlers turned away from the whole site, by name |
|---|---|---|
disallowoptional | string[] | Paths kept out of every crawler |