Do you know how to prevent indexing of pages past the home page in WP? I mean I don’t want mysite.com/page/2, mysite.com/page/3 to be indexed. This is because I use home.php for my theme, so that page/2, page/3 are all the same. Please give me a hint or a code snippet please, I don’t want […]
I’ve noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn’t have that line, so why does it appear? Is it maybe a strange behaviour of some plugins?
Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn’t find anything suitable.
The homepage on a site we’ve developed has four pieces of content: the parent page and three further “pages” of text which we bring into the parent page using PHP. Call them Foo, Bar1, Bar2, Bar3 if you like. Maybe this is or isn’t the correct way to solve the problem of a complex page […]
I remove robots.txt file in my server root but google can’t delete that. Webmastertool to create robotx.txt file How can I remove or disable.
By default whenever you disable indexing via Admin Settings [ x ] Discourage search engines from indexing this site It adds a meta tag in the header like so: <meta name=’robots’ content=’noindex,follow’ /> How do I change that to be nofollow instead of follow? I find it odd it enables “follow” and overall want it […]
What is the “best” setup for robots.txt? I’m using the following permalink structure /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ago): User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */comments I want my comments […]