Generate and validate robots.txt configurations for SEO crawlers.
Search engines like Google and Bing use automated bots (crawlers) to discover and index pages on the web. However, you don't always want every page indexed—for example, your site's admin dashboard, API endpoints, or private user profiles should not appear in Google search results.
The robots.txt file is a simple text file placed at the root of your domain (e.g., https://example.com/robots.txt). It dictates the rules of engagement for these bots using a protocol called the Robots Exclusion Standard.
* means all bots, while Googlebot targets only Google.A very common and dangerous mistake is using robots.txt to "hide" secret URLs.
If you write Disallow: /secret-admin-login-panel, a well-behaved bot like Google will ignore the page. However, robots.txt is a completely public file. Malicious hackers will immediately read your robots.txt file to find exactly where your sensitive endpoints are located. Never use robots.txt for security or access control.
Why should you never put sensitive URLs (like an admin panel path) in your robots.txt file?