Fetch and parse robots.txt, and check nothing important is blocked.
Free, no signup. We fetch the page once and read what your server sends back.
robots.txt controls which crawlers may request which paths. A single misplaced line can remove an entire site from search, and because nothing visibly breaks, it can sit there for months.
No, and this catches people out. It stops crawling, not indexing. A blocked page with inbound links can still appear in results with no description. Use a meta robots noindex to keep a page out.
Not strictly. Without one everything is crawlable, which is usually what you want. It is still worth having for the sitemap directive.
A genuine decision with a real trade-off. Publishers often do. Businesses wanting to be recommended by AI assistants generally should not. Our AI crawler checker tests what your server actually does, which is often not what robots.txt says.
Usually because robots.txt blocked crawling but the site was linked from somewhere, so it got indexed without being crawled. Password protection is the only reliable fix.
Run the full free report, or have us go through it with you on a call.