Using Robots.txt for WordPress Blogs
August 18th, 2009
Robots.txt file is very important for any website and especially for WordPress. Robots.txt helps search engines know what which content, file, directory is to be indexed and which is not. This helps avoid duplicate content on your blog.
Robots.txt for WordPress Self-Hosted blogs. This is just an example, you can customize it according to your blog.
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* # Internet Archiver Wayback Machine User-agent: ia_archiver Disallow: / # digg mirror User-agent: duggmirror Disallow: / Sitemap: http://www.example.com/sitemap.xml
