« DoFollow or NoFollow for Blogs
Putting up Christmas Decorations »

Wordpress duplicate content

This is my first month of Blogging and i have learned quite a lot, today i realised that Wordpress is not very friendly when it comes to avoiding duplicate content on search engines. For anyone who does not understand this term, here is a quick explanation:

Wordpress by default has the main page with your articles, each article can be found on the categories page, the archive page, and it’s own page. The problem with this is search engines will spider (travel from link to link) examining content and eventually listing it in their search engine. Unfortunately in a standard Wordpress installation this will lead to several pages having the same, or very similar content. These days search engines will penalise a site for this, meaning your important page might not be listed, or certain important pages may be lower in the searches than they should be.

After doing quite a bit of research i finally decided on the following solution:

1. I added the following to the header.php of my current theme. This will stop pages that are NOT a single page (a full post), a content page (like about), or the home page from being listed in search engines.

<?php if(is_single() || is_page() || is_home()) { ?>
    <meta name=”robots” content=”all,index,follow” />
<?php } else { ?>
    <meta name=”robots” content=”noindex,follow” />
<?php }?>

2. I added the following to my header.php file of my current theme, this will give unique titles and descriptions depending on what type of page it is. (Make sure you remove any lines relating to <title> or <META NAME=”DESCRIPTION” or <META NAME=”KEYWORDS” otherwise it may not work correctly. NOTE: Please change the text in italics to your titles, descriptions and keywords to suit your site.

<?php
if(is_home() || is_search())   // Is it home page or search page? If so have the following tags.
{
        echo ‘<title>Mad Dozzas Blog - General rants about useful, funny and annoying stuff</title>’;
        echo ‘<META NAME=”DESCRIPTION” CONTENT=”This is Mad Dozzas Blog, read general rants about funny, useful and annoying stuff“>’;
        echo ‘<META NAME=”KEYWORDS” CONTENT=”mad, dozza, blog, blogs, maddozza, general, useful, annoying, funny, stuff, rants“>’;
}
elseif(is_category())  // Is it a catagory page? If so have the following tags.

{
        echo ‘<title>Mad Dozzas Blog - Catagory: ‘.single_cat_title(”", false).’</title>’;
        echo ‘<META NAME=”DESCRIPTION” CONTENT=”‘.single_cat_title(”", false).’ Catagory: this and other general rants about useful, funny and annoying stuff by Mad Dozza“>’;
        echo ‘<META NAME=”KEYWORDS” CONTENT=”mad, dozza, blog, blogs, maddozza, general, useful, annoying, funny, stuff, rants“>’;
}
elseif(is_archive())  // Is it an archive page? If so have the following tags.

{
        echo ‘<title>Mad Dozzas Blog Archives</title>’;
        echo ‘<META NAME=”DESCRIPTION” CONTENT=”This is Mad Dozzas Blog, read general rants about funny, useful and annoying stuff“>’;
        echo ‘<META NAME=”KEYWORDS” CONTENT=”mad, dozza, blog, blogs, maddozza, general, useful, annoying, funny, stuff, rants“>’;
}
else  // Is full post page? If so have the following tags.

{
        echo ‘<title>’.wp_title(”, false).’</title>’;
        echo ‘<META NAME=”DESCRIPTION” CONTENT=”Mad Dozzas Blog - ‘.wp_title(”, false).’ and other general rants about useful, funny and annoying stuff“>’;
        echo ‘<META NAME=”KEYWORDS” CONTENT=”mad, dozza, blog, blogs, maddozza, general, useful, annoying, funny, stuff, rants“>’;
}
?>

3. I use the following robots.txt file which stops search engines viewing some pages. In this case it will allow the Mediapartners-Google bot access (this is used for determining what adverts AdSense displays) which is necessary for relevant ads. The final line is a link to the sitemap file i use.

 # robots.txt for http://blog.maddozza.com/

User-agent: *
Disallow: /cgi-bin
Disallow: /awstats
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /category
Disallow: /tag
Disallow: /author
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /*?*
Disallow: /*.html/$
Disallow: /*feed*
Disallow: /wp-register.php
Disallow: /wp-login.php

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /

Sitemap: http://blog.maddozza.com/sitemapspal.xml

Hopefully this will stop any duplicate content getting to the search engines and should help my ranking. If i find out any more i will update the post.

Bookmark this post:
  • Digg
  • del.icio.us
  • StumbleUpon
  • Technorati
  • Netvouz
  • NewsVine
  • MisterWong
  • Furl
  • Reddit
  • Smarking
  • Netscape
  • Spurl
  • YahooMyWeb

This entry was posted on Tuesday, November 27th, 2007 at 4:29 pm and is filed under Internet. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

59 Responses to “Wordpress duplicate content”

  1. This might be a bit off topic but the nice thing about buying new electronics is they have a warranty. You are buying something that has moving parts or electricity going to it so it has the potential of breaking, there is nothing you can do but replace it, that does not mean the company that made the product sucks, it means the individual item sucked!

  2. I had same problem, my solution was simple - I’ve changed my permalink structure to /%year%/%monthnum%/%day%/%postname%.html
    Note the trailing “.html”.
    The rest is done via .htaccess by redirecting “www.” traffic to www-less domain name.

  3. Nice work around there. I can use portions of this for my own situation..
    Thanks

  4. thanks you

  5. Does anyone know if you use a site map, if the links are still followed?

  6. Yes the links are still followed

  7. hai.. thanks for sharing the meta part :) i’ve never thought of it.. I used to use plugin for that.. I will consider to use it in my blog.

  8. Great page. Thanks a lot.

  9. Thanx for ur tips, i’m using it at my site.

  10. Thanks for that. Some good ideas there, especially changing your meta data. Its basic but heaps of people forget that.

  11. thanks you

  12. Thanks for the great insight. However, I think there is something also known as category killer plugin which automatically kills the duplicate contents, which you only need to install rather than working with the code.

  13. Thanks for the great tip! I’m implementing those tips right away in my site!

  14. thanks for the tips..i will also test it on my WP blog as well

  15. I recommend that anyone with a wordpress blog get the All in One SEO Plugin, which creates descriptions and keywords for you, and it helps with duplicate content because it ads nofollow to your categories and archives automatically, which you can turn on and off in the settings.

  16. Great tip. I will certainly use this on my bog. I appreciate your time. Thanks so much.

  17. I did the same thing with andr. I have not noticed how wp can produce (almost) the same contents. but with better permalink stucture you can avoid to have unique links from all of your posts.

  18. Superior to a plugin and of course much faster. This method affords much better control. However, let me point you to an excellent hack to provide page specific meta tags using custom fields. sorry ’bout the long URL http://www.blogopreneur.com/2006/09/11/a-different-meta-description-and-keywords-for-each-post-for-wordpress/ Used with your duplicate content stopper here, gives a great combination with minimum DB queries. That’s a very important factor!

  19. I have always wondered if the search engines would consider this duplicate content. I wonder if there are Search engine friendly way to have both pages be indexed.

  20. Top regards to this blog owner, great stuff!

  21. Well thanks mate . I would rather have this done when i go for a new blog on wordpress

  22. I can see what you mean, i never thought about it like that, i will look into it further myself, i thoght that it would only be duplicate content if you had 2 of the same pages with different permalinks, if you only have one permalink to the article in question, no matter which categories it is in then surely it is only listed once in the site?

  23. Nice Site. I am new here but will be visiting more often as you have done a good job.

    Thanks

  24. Thanks for this tutorial.. it does help.. and the comments as well..

  25. Great tutorial on avoiding duplicate content slap from Google. I have a blog and i will be implementing this from now on. Thanks.

  26. This is great… Many thanks for the tips. I just hate those irresponsible people that copy and make spam out of our articles.

  27. Thanks for the great tip! I’m implementing those tips right away in my site!

  28. thanks for the article. it’s very good ;)

  29. great tips, I’ll consider to follow your tips. thanks. great job…

  30. Thanks for this tutorial.

  31. Top regards to this blog ownerrr :)

  32. great post, thanks really usefull

  33. Hi
    Nice pieces of code to use.
    However,what I understand is that googlebot is instructed to deal with blogs’ duplicate content and to ignore it.I do not know how accurate the algorithm is.

  34. thank^s admin , very good post , i like to your web site..!!

  35. I liked robots.txt info i’ll use that

  36. Nice! I had to add a bunch of random content on my pages to make every page on my site randomly different to counter this duplicate problem. Your fix is much better. Thanks!

  37. Wordpress is difficult to seo opitimize. I can relate to your struggle. The all in one seo plug in is good to try too.

  38. This code has been very helpful. Thanks very much! :D

  39. nice idea. will try it on my site.

    regards
    thinkjayant

  40. don’t know what you think but i love joomla though

    it’s a really good cms - something similar to wordpress

  41. First and foremost, let me extend my gratitude for accepting my message. This is by far the best ultimate resource for the subject matter. It is a surprise to see dedicated people who really does their homework in providing impeccable information. Im hoping to read more of your valuable advice and knowledge. Thank you and more power to you my friend!

  42. Thanks for that. Some good ideas there, especially changing your meta data. Its basic but heaps of people forget that.

  43. I would have to agree with the comment above, that the all in onve seo plugin is very good. Trying to please the search engines is becoming more and more difficult.

  44. What about tag pages? I’m finding that alot of my traffic is coming from tag pages. I didn’t see any settings for true/false with regard to tag pages. Thx.

  45. This Blog reminds me the reason I like bloging so much, the interaction is very important with readers and you guys have it right. Looks great too, will be back for more posts, David the mover. : - )

  46. Thanks, I came across a similar code in here, but implementing yours has reduced the duplicate content issue on my blog.The robots.txt is spot on.Increase the bots speed around your site.

  47. I am learning how to best use the all in one seo plugin and this comment thread has proved useful in that endeavour.

  48. You get more problems if you use some of the plugins that use the htaccess file to manipulate the URLs.

  49. Having duplicate contents on my blog was really a nuisance to me… i didn’t know how to manipulate the plugin. thanks a lot for sharing this information, it was really useful to me.

  50. Not a bad idea you have there. Have you ever thought about turning this patch into a plugin that would make the changes automatically? Many new users would find that helpful I believe.

  51. I would have to agree with the comment above, that the all in onve seo plugin is very good. Trying to please the search engines is becoming more and more difficult. thanks

  52. Holy crap! This is absolutely ESSENTIAL information for anyone that uses Wordpress as their CMS system. The dupe content penalty sucks, and this is the most straight forward and efficient method to combat that! I’d always been hardcoding changes to my templates (and not efficiently) with HTML rather then PHP. Fantastic write up and thanks for this…

  53. Great blog - just wanted to give credit where credit is due!

  54. it worked great for me and solved a problem that seo people have to grapple with. thanks

    SEO Leeds

  55. Great minds think alike. if you click the link above this bit, you’ll see I’ve done something similar, using things like excerpts or custom fields, on posts, tag pages, categories etc.

  56. hi,

    thanks for stunning insights form blog author. i learned some technical issues related wordpress software. Great job done, will hope for same type of article in future.

  57. I use similar code for meta descriptions (click link) but use a custom field for the actual post. Using the excerpt for meta description might be a half way house, and better than the title + standard text?

  58. Interesting idea…. gotta try this out

  59. Brilliant idea to eliminate duplication, thanks for the codes as well.. it really worked out well for me. Cheers

Leave a Reply