Robots.txt Analyzer
Check which crawlers are allowed or blocked by your Lovable app's robots.txt. See rules for Googlebot, GPTBot, ClaudeBot, and 20+ other bots — get a prompt to update your config.
Robots.txt Examples
Copy these templates to control how crawlers access your site.
Allow All Crawlers
Best for most websites wanting maximum visibility
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Block AI Training Only
Allow search engines but block AI training crawlers
User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / Sitemap: https://example.com/sitemap.xml
Allow Search + AI Citations
Be indexed by search engines and cited by AI assistants
User-agent: * Allow: / # Allow AI assistants to cite your content User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://example.com/sitemap.xml
Block Specific Paths
Allow most content but protect admin/private areas
User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /*.json$ Sitemap: https://example.com/sitemap.xml
How to Set Up robots.txt in Your Lovable App
A robots.txt file tells search engines and AI crawlers which parts of your site they can access. Without one, all crawlers are allowed everywhere by default — which is fine for most Lovable apps. But as your app grows, you may want to block crawlers from accessing private pages, API routes, or prevent AI training crawlers from scraping your content.
Check your current robots.txt
Enter your URL above to see if you have a robots.txt file and what rules it contains. If you get a 404, you don't have one yet — that's okay, it means all crawlers are allowed.
Decide what to allow and block
Most Lovable apps should allow all search engine crawlers (Googlebot, Bingbot) for maximum visibility. Consider blocking AI training crawlers (GPTBot, CCBot) if you don't want your content used for AI training. Always block private paths like /auth, /admin, and /api.
Use the templates or copy the prompt
Scroll down to the 'Robots.txt Examples' section for ready-made templates, or click 'Copy Prompt to Update Robots.txt' to get a prompt based on your current configuration.
Add the file to your Lovable app
Paste the prompt into Lovable and it will create or update the robots.txt file in your public directory. The file must be at the root of your domain (e.g., yourapp.com/robots.txt) to work.
Example prompt to paste into Lovable:
Create a robots.txt file for my app with these rules: User-agent: * Allow: / Disallow: /auth Disallow: /api/ Disallow: /private/ # Block AI training crawlers User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / Sitemap: https://myapp.com/sitemap.xml Place this file at the root of the public directory so it's accessible at /robots.txt.
Your robots.txt is one of the first files crawlers check when they visit your site. A well-configured robots.txt protects private content, reduces unnecessary server load from bots, and ensures the right crawlers can access the right pages. Check it whenever you add new sections to your app.
More Free SEO Tools
Continue auditing your site with these companion tools.
Launch Checklist
Score your app across 35 launch-readiness checks
Meta Tag Audit
Audit title, OG, Twitter cards & more
SEO & AI Crawler Audit
Full technical SEO and AI visibility audit
Crawler Simulator
Test how 20+ search and AI bots see your site
OG Image Previewer
Preview social media link cards
Security Headers
Scan for missing security headers with A-F grades
Robots.txt controls crawlers. Bugwarden catches the bugs crawlers can't see.
This tool manages bot access. Bugwarden monitors your Lovable app for real user errors and gives you a prompt to fix each one. Set up in 30 seconds.
Sign up and we'll reach out to help you set up Bugwarden for your Lovable app.