Robots.txt Helper

1. Choose a Preset to start from a common pattern, or leave Custom and fill the fields yourself.
2. Set User-agent to the crawler group (use * for all bots that follow this file).
3. Add Disallow and Allow paths, one per line. Paths should look like URLs without the domain (e.g. /admin/); a leading / is added if you omit it.
4. Optionally set Crawl-delay and Sitemap. In presets, replace example.com with your real site.
5. Use Copy or Download to save the text. Upload the file as robots.txt at the root of your site (same host as your pages), e.g. https://yoursite.com/robots.txt.
6. Reset clears the form to defaults. Editing any field after a preset switches you back to Custom.

robots.txt is a polite request to crawlers, not access control. Do not rely on it to hide private URLs.