In robots.txt can I write the following relative URL for the sitemap file? sitemap: /sitemap.ashx Or do I have to use the complete (absolute) URL for the sitemap file, like: sitemap: http://subdomain.domain.com/sitemap.ashx Why I wonder: I own a new blog service, www.domain.com, that allow users to blog on accountname.domain.com. I use wildcards, so all subdomains … Read more
Relatively new to anything that doesn’t require common sense in WordPress. My previous site was built poorly etc, so cutting a long story short I installed a new theme and started building up pages again etc. I have ran various diagnostics on seo etc while I am working and seem to have problems with indexability … Read more
I can see the robots.txt file here : http://persian.cc/robots.txt But Google can’t ! I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine and this robots.txt is a virtual one being made by WordPress. Now how to stop wordpress doing that? OR … Read more
I’ve noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn’t have that line, so why does it appear? Is it maybe a strange behaviour of some plugins? Best Answer For those who are using … Read more
I have a subfolder WP installation. It serves multiple domains, which are linked to the same folder. The requested domain is received in wp-config.php (from the $_SERVER variable) and used to define WP_SITEURL, WP_HOME and DOMAIN_CURRENT_SITE. So we can open the same both from domain.com and domain.co.uk. I need to add the link to XML … Read more
Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn’t find anything suitable. 2 Answers 2 Straight from the source, (line 1845 wp-includes/functions.php, 3.3.1): function do_robots() { header( ‘Content-Type: text/plain; charset=utf-8’ ); do_action( ‘do_robotstxt’ ); $output = “User-agent: *\n”; … Read more
The homepage on a site we’ve developed has four pieces of content: the parent page and three further “pages” of text which we bring into the parent page using PHP. Call them Foo, Bar1, Bar2, Bar3 if you like. Maybe this is or isn’t the correct way to solve the problem of a complex page … Read more
I remove robots.txt file in my server root but google can’t delete that. Webmastertool to create robotx.txt file How can I remove or disable. 1 Answer 1 WordPress generate a dynamic robots.txt which does not physically exists. To remove/disable it you have two options: Option 1: Remove do_robots action in your theme functions.php or plugin … Read more
By default whenever you disable indexing via Admin Settings [ x ] Discourage search engines from indexing this site It adds a meta tag in the header like so: <meta name=”robots” content=”noindex,follow” /> How do I change that to be nofollow instead of follow? I find it odd it enables “follow” and overall want it … Read more
When I try to google search my web address, the follow comes up as the meta description “A description for this result is not available because of this site’s robots.txt – learn more.” The website I have tried to edit the robots.txt (downloaded the WP Robot plugin) and although I have changed it to User-agent: … Read more
