Robots.txt Generator

Build a basic robots.txt for crawlers (allow, disallow, sitemap)

User-agent

Allow entire site (Allow: /)

Disallow paths (one per line)

Sitemap URL (optional)

Crawl-delay seconds (optional, non-standard)