Google Webmaster Tools can see it, but I can’t find it. Any idea on how it’s generated or where it is the file structure? I don’t see the file in my root. My website is at http://mikewills.me and the robots.txt file URL is http://mikewills.me/robots.txt.
In the end, Google isn’t indexing my site and I am trying to figure out how to edit the robots.txt so that indexing is allowed. I have changed the privacy option to allow indexing, but that hasn’t updated the robots.txt.
It’s dynamically generated by the function do_robots(), which has both an action (do_robotstxt) and a filter (robots_txt). If you create a robots.txt file in your WordPress root, it will (probably) be served up when /robots.txt is requested, otherwise processing will fall back on WordPress.
Your current file looks like this:
User-agent: *
Disallow:
Sitemap: http://mikewills.me/sitemaps/mikewills-me.xml.gz
The first two settings are appropriate for an indexable blog. So, you may just be waiting for Google at this point. Maybe you need to request reconsideration of your site.