Do we need a robots.txt file to cater for web crawling?

[njsch]

Not sure if this is needed, but we've never had a custom robots.txt plain text file in the root dir. Do we need this and if so, what would be the best way to generate it?

[Coedice]

I've played with robots.txt before. Since default behaviour for spiders is to crawl, the robots.txt is mainly useful for preventing crawling, but the spiders who we would want to block (such as spiders that are trying to get email addresses to spam people) ignore robots.txt anyway

Is there some other purpose for a robots.txt file you were thinking would be helpful?

I was going to suggest we add the jekyll-sitemap plugin, but I see you have already done that 😀

[njsch]

No, in that case we probably don't need one.