robots.txt generator

Configure crawl directives and download a clean robots.txt file.