Articles of robots.txt

What is the meaning of User-agent and Disallow in robots.txt file in wordpress

What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.

Disallow: /wp-* in robots.txt?

I have just received an email from Google advising that it can’t accessing certain javascript and css content from my site. I’ve looked at the robots.txt file which contains: User-agent: * Crawl-delay: 5 Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp-* It looks like it is the Disallow: /wp-* that is […]

“No sitemap linked in your robots.txt file”

Relatively new to anything that doesn’t require common sense in WordPress. My previous site was built poorly etc, so cutting a long story short I installed a new theme and started building up pages again etc. I have ran various diagnostics on seo etc while I am working and seem to have problems with indexability […]

Is this robots.txt making Google not to show some of my content? User-agent: * Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp- Sitemap: I just changed to a new server. And my site started to not to appear in the top search results s before. I was wondering if the robots.txt above is blocking some of my WordPress content? […]

How to stop WordPress showing a virtual Robots.txt?

I can see the robots.txt file here : But Google can’t ! I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine and this robots.txt is a virtual one being made by WordPress. Now how to stop wordpress doing that? OR […]

Add robots.txt to root

I have a new WordPress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents. I have checked the “Discourage search engines from indexing this site” box on the Settings > Reading menu, […]

Virtual robots.txt missing

on every WordPress installation there is a default virtual robots.txt file with the following User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php but on my 2 last project /robots.txt give me a 404 instead. I tried to disable all plugin I tried to switch theme with a blank one Tried to look on another older project which […]

Questions about Robots.txt

I need some help with robots.txt. Under Privacy Settings, I selected “I would like my blog to be visible to everyone, including search engines…” As I understand it, WordPress automatically creates a robots.txt file. But, I need to exclude one page from the automatically generated robots.txt file. Let’s say the page is called, “myPage”. How […]

Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?

I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages: “Url blocked by robots.txt.” and “Sitemap contains urls which are blocked by robots.txt.” But I have no robots.txt anywhere. It seems my entire site and every link on it is blocked from being crawled by […]

How to solve Robots.txt problem with WordPress fresh install?

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems. 1- I get this message : “Comments for FirstName LastName‎ A description for this result is not available because of this site’s robots.txt – learn more.” When I google my name. I re-checked setting and I […]