What is the “best” setup for robots.txt
?
I’m using the following permalink structure /%category%/%postname%/
.
My robots.txt
currently looks like this (copied from somewhere a long time ago):
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments
- I want my comments to be indexed. So I can remove this
- Do I want to disallow indexing categories because of my permalink structure?
- An article can have several tags and be in multiple categories. This may cause duplicates in search providers like Google. How should I work around this?
Would you change anything else here?