Hugo can generated a customized robots.txt in the same way as any other templates.
To enable it, just set enableRobotsTXT option to true in the configuration file. By default, it generates a robots.txt, which allows everything, with the following content:
User-agent: *
Hugo will use the template robots.txt according to the following list in descending precedence:
- /layouts/robots.txt
- /themes/
THEME/layout/robots.txt
An example of a robots.txt layout is:
User-agent: *
{{range .Data.Pages}}
Disallow: {{.RelPermalink}}{{end}}
This template disallows and all the pages of the site creating one Disallow entry for each one.