Hugo can generated a customized robots.txt in the same way as any other templates.

To enable it, just set enableRobotsTXT option to true in the configuration file. By default, it generates a robots.txt, which allows everything, with the following content:

User-agent: *

Hugo will use the template robots.txt according to the following list in descending precedence:

  • /layouts/robots.txt
  • /themes/THEME/layout/robots.txt

An example of a robots.txt layout is:

User-agent: *

{{range .Data.Pages}}
Disallow: {{.RelPermalink}}{{end}}

This template disallows and all the pages of the site creating one Disallow entry for each one.