on every WordPress installation there is a default virtual
robots.txt file with the following
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
but on my 2 last project
/robots.txt give me a 404 instead.
This not server related because it work on my local environment with older project
Any idea why ?
Nevermind. It just doesn’t work when wordpress is in a subfolder.
(robots.txt should be on website root so just create your own if wordpress is in a subfolder)