Articles of robots.txt

Preventing Search Engines Indexing Pages 2, 3 and More?

Do you know how to prevent indexing of pages past the home page in WP? I mean I don’t want mysite.com/page/2, mysite.com/page/3 to be indexed. This is because I use home.php for my theme, so that page/2, page/3 are all the same. Please give me a hint or a code snippet please, I don’t want […]

Unwanted “crawl delay: 10” line added to my robots.txt

I’ve noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn’t have that line, so why does it appear? Is it maybe a strange behaviour of some plugins?

Create unique robots.txt for every site on multisite-installation

Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn’t find anything suitable.

Correct way to hide pseudo pages from being shown?

The homepage on a site we’ve developed has four pieces of content: the parent page and three further “pages” of text which we bring into the parent page using PHP. Call them Foo, Bar1, Bar2, Bar3 if you like. Maybe this is or isn’t the correct way to solve the problem of a complex page […]

Stuck on my server root folder, robots.txt file not deleting

I remove robots.txt file in my server root but google can’t delete that. Webmastertool to create robotx.txt file How can I remove or disable.

NoFollow Entire Website

By default whenever you disable indexing via Admin Settings [ x ] Discourage search engines from indexing this site It adds a meta tag in the header like so: <meta name=’robots’ content=’noindex,follow’ /> How do I change that to be nofollow instead of follow? I find it odd it enables “follow” and overall want it […]

What is a good robots.txt?

What is the “best” setup for robots.txt? I’m using the following permalink structure /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ago): User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */comments I want my comments […]