What a 200 response can hide

A site with a catch-all route can show you its homepage when you ask for /robots.txt, complete with a 200 status. I changed Letthebots to inspect the body and flag that as an HTML fallback. That gives you a specific route to fix. A timeout still shows up as an incomplete check. I wrote about both in the Letthebots rebuild.

More on Letthebots