Letthebots
Check how your site responds to crawler probes, what content they can read, and what to fix first.
Why I built it
I built it after repeatedly finding robots.txt rules and JS-only rendering blocking AI crawlers on sites I worked on, with no quick way to check. The surprise was how inconsistent crawler behaviour actually is across providers. What blocks one bot doesn't always block another.
What a report shows
The example report uses a fictional article about plant care to show how the tool explains its findings. It shows the page as readable and accessible to the crawler probe, then flags a separate issue: the page has Organization markup, but no Article markup for the visible article. The suggested fix uses the real headline, author and publication date. You can open the report and see why it suggests that change.
A check worth getting right
Some sites return the same HTML and a 200 status for every URL, including /robots.txt. The scanner checks the body before it accepts a response as a crawler policy. A timeout and an HTML fallback also get different findings, so the report tells you which problem to investigate. The bot and browser headers can reveal differences in the responses the scanner receives. The crawl and citation decisions themselves happen elsewhere.