Wordpress duplicate content
This is my first month of Blogging and i have learned quite a lot, today i realised that Wordpress is not very friendly when it comes to avoiding duplicate content on search engines. For anyone who does not understand this term, here is a quick explanation:
Wordpress by default has the main page with your articles, each article can be found on the categories page, the archive page, and it’s own page. The problem with this is search engines will spider (travel from link to link) examining content and eventually listing it in their search engine. Unfortunately in a standard Wordpress installation this will lead to several pages having the same, or very similar content. These days search engines will penalise a site for this, meaning your important page might not be listed, or certain important pages may be lower in the searches than they should be.
After doing quite a bit of research i finally decided on the following solution:
1. I added the following to the header.php of my current theme. This will stop pages that are NOT a single page (a full post), a content page (like about), or the home page from being listed in search engines.
<?php if(is_single() || is_page() || is_home()) { ?>
<meta name=”robots” content=”all,index,follow” />
<?php } else { ?>
<meta name=”robots” content=”noindex,follow” />
<?php }?>
2. I added the following to my header.php file of my current theme, this will give unique titles and descriptions depending on what type of page it is. (Make sure you remove any lines relating to <title> or <META NAME=”DESCRIPTION” or <META NAME=”KEYWORDS” otherwise it may not work correctly. NOTE: Please change the text in italics to your titles, descriptions and keywords to suit your site.
<?php
if(is_home() || is_search()) // Is it home page or search page? If so have the following tags.
{
echo ‘<title>Mad Dozzas Blog – General rants about useful, funny and annoying stuff</title>’;
echo ‘<META NAME=”DESCRIPTION” CONTENT=”This is Mad Dozzas Blog, read general rants about funny, useful and annoying stuff“>’;
echo ‘<META NAME=”KEYWORDS” CONTENT=”mad, dozza, blog, blogs, maddozza, general, useful, annoying, funny, stuff, rants“>’;
}
elseif(is_category()) // Is it a catagory page? If so have the following tags.{
echo ‘<title>Mad Dozzas Blog – Catagory: ‘.single_cat_title(“”, false).’</title>’;
echo ‘<META NAME=”DESCRIPTION” CONTENT=”‘.single_cat_title(“”, false).’ Catagory: this and other general rants about useful, funny and annoying stuff by Mad Dozza“>’;
echo ‘<META NAME=”KEYWORDS” CONTENT=”mad, dozza, blog, blogs, maddozza, general, useful, annoying, funny, stuff, rants“>’;
}
elseif(is_archive()) // Is it an archive page? If so have the following tags.{
echo ‘<title>Mad Dozzas Blog Archives</title>’;
echo ‘<META NAME=”DESCRIPTION” CONTENT=”This is Mad Dozzas Blog, read general rants about funny, useful and annoying stuff“>’;
echo ‘<META NAME=”KEYWORDS” CONTENT=”mad, dozza, blog, blogs, maddozza, general, useful, annoying, funny, stuff, rants“>’;
}
else // Is full post page? If so have the following tags.{
echo ‘<title>’.wp_title(”, false).’</title>’;
echo ‘<META NAME=”DESCRIPTION” CONTENT=”Mad Dozzas Blog - ‘.wp_title(”, false).’ and other general rants about useful, funny and annoying stuff“>’;
echo ‘<META NAME=”KEYWORDS” CONTENT=”mad, dozza, blog, blogs, maddozza, general, useful, annoying, funny, stuff, rants“>’;
}
?>
3. I use the following robots.txt file which stops search engines viewing some pages. In this case it will allow the Mediapartners-Google bot access (this is used for determining what adverts AdSense displays) which is necessary for relevant ads. The final line is a link to the sitemap file i use.
# robots.txt for http://blog.maddozza.com/
User-agent: *
Disallow: /cgi-bin
Disallow: /awstats
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /category
Disallow: /tag
Disallow: /author
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /*?*
Disallow: /*.html/$
Disallow: /*feed*
Disallow: /wp-register.php
Disallow: /wp-login.php# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /Sitemap: http://blog.maddozza.com/sitemapspal.xml
Hopefully this will stop any duplicate content getting to the search engines and should help my ranking. If i find out any more i will update the post.
This might be a bit off topic but the nice thing about buying new electronics is they have a warranty. You are buying something that has moving parts or electricity going to it so it has the potential of breaking, there is nothing you can do but replace it, that does not mean the company that made the product sucks, it means the individual item sucked!
I had same problem, my solution was simple – I’ve changed my permalink structure to /%year%/%monthnum%/%day%/%postname%.html
Note the trailing “.html”.
The rest is done via .htaccess by redirecting “www.” traffic to www-less domain name.
Nice work around there. I can use portions of this for my own situation..
Thanks
thanks you
Does anyone know if you use a site map, if the links are still followed?
Yes the links are still followed
hai.. thanks for sharing the meta part
i’ve never thought of it.. I used to use plugin for that.. I will consider to use it in my blog.
Great page. Thanks a lot.
Thanx for ur tips, i’m using it at my site.
Thanks for that. Some good ideas there, especially changing your meta data. Its basic but heaps of people forget that.
thanks you
Thanks for the great insight. However, I think there is something also known as category killer plugin which automatically kills the duplicate contents, which you only need to install rather than working with the code.
Thanks for the great tip! I’m implementing those tips right away in my site!
thanks for the tips..i will also test it on my WP blog as well
I recommend that anyone with a wordpress blog get the All in One SEO Plugin, which creates descriptions and keywords for you, and it helps with duplicate content because it ads nofollow to your categories and archives automatically, which you can turn on and off in the settings.
Great tip. I will certainly use this on my bog. I appreciate your time. Thanks so much.
I did the same thing with andr. I have not noticed how wp can produce (almost) the same contents. but with better permalink stucture you can avoid to have unique links from all of your posts.
Superior to a plugin and of course much faster. This method affords much better control. However, let me point you to an excellent hack to provide page specific meta tags using custom fields. sorry ’bout the long URL http://www.blogopreneur.com/2006/09/11/a-different-meta-description-and-keywords-for-each-post-for-wordpress/ Used with your duplicate content stopper here, gives a great combination with minimum DB queries. That’s a very important factor!
I have always wondered if the search engines would consider this duplicate content. I wonder if there are Search engine friendly way to have both pages be indexed.
Top regards to this blog owner, great stuff!
Well thanks mate . I would rather have this done when i go for a new blog on wordpress
I can see what you mean, i never thought about it like that, i will look into it further myself, i thoght that it would only be duplicate content if you had 2 of the same pages with different permalinks, if you only have one permalink to the article in question, no matter which categories it is in then surely it is only listed once in the site?
Nice Site. I am new here but will be visiting more often as you have done a good job.
Thanks
Thanks for this tutorial.. it does help.. and the comments as well..
Great tutorial on avoiding duplicate content slap from Google. I have a blog and i will be implementing this from now on. Thanks.
This is great… Many thanks for the tips. I just hate those irresponsible people that copy and make spam out of our articles.
Thanks for the great tip! I’m implementing those tips right away in my site!
thanks for the article. it’s very good
great tips, I’ll consider to follow your tips. thanks. great job…
Thanks for this tutorial.
Top regards to this blog ownerrr
great post, thanks really usefull
Hi
Nice pieces of code to use.
However,what I understand is that googlebot is instructed to deal with blogs’ duplicate content and to ignore it.I do not know how accurate the algorithm is.
thank^s admin , very good post , i like to your web site..!!
I liked robots.txt info i’ll use that
Nice! I had to add a bunch of random content on my pages to make every page on my site randomly different to counter this duplicate problem. Your fix is much better. Thanks!
Wordpress is difficult to seo opitimize. I can relate to your struggle. The all in one seo plug in is good to try too.
This code has been very helpful. Thanks very much!
nice idea. will try it on my site.
regards
thinkjayant
don’t know what you think but i love joomla though
it’s a really good cms – something similar to wordpress
First and foremost, let me extend my gratitude for accepting my message. This is by far the best ultimate resource for the subject matter. It is a surprise to see dedicated people who really does their homework in providing impeccable information. Im hoping to read more of your valuable advice and knowledge. Thank you and more power to you my friend!
Thanks for that. Some good ideas there, especially changing your meta data. Its basic but heaps of people forget that.
I would have to agree with the comment above, that the all in onve seo plugin is very good. Trying to please the search engines is becoming more and more difficult.
What about tag pages? I’m finding that alot of my traffic is coming from tag pages. I didn’t see any settings for true/false with regard to tag pages. Thx.
This Blog reminds me the reason I like bloging so much, the interaction is very important with readers and you guys have it right. Looks great too, will be back for more posts, David the mover. : – )
Thanks, I came across a similar code in here, but implementing yours has reduced the duplicate content issue on my blog.The robots.txt is spot on.Increase the bots speed around your site.
I am learning how to best use the all in one seo plugin and this comment thread has proved useful in that endeavour.
You get more problems if you use some of the plugins that use the htaccess file to manipulate the URLs.
Having duplicate contents on my blog was really a nuisance to me… i didn’t know how to manipulate the plugin. thanks a lot for sharing this information, it was really useful to me.
Not a bad idea you have there. Have you ever thought about turning this patch into a plugin that would make the changes automatically? Many new users would find that helpful I believe.
I would have to agree with the comment above, that the all in onve seo plugin is very good. Trying to please the search engines is becoming more and more difficult. thanks
Holy crap! This is absolutely ESSENTIAL information for anyone that uses Wordpress as their CMS system. The dupe content penalty sucks, and this is the most straight forward and efficient method to combat that! I’d always been hardcoding changes to my templates (and not efficiently) with HTML rather then PHP. Fantastic write up and thanks for this…
Great blog – just wanted to give credit where credit is due!
it worked great for me and solved a problem that seo people have to grapple with. thanks
SEO Leeds
Great minds think alike. if you click the link above this bit, you’ll see I’ve done something similar, using things like excerpts or custom fields, on posts, tag pages, categories etc.
hi,
thanks for stunning insights form blog author. i learned some technical issues related wordpress software. Great job done, will hope for same type of article in future.
I use similar code for meta descriptions (click link) but use a custom field for the actual post. Using the excerpt for meta description might be a half way house, and better than the title + standard text?
Interesting idea…. gotta try this out
Brilliant idea to eliminate duplication, thanks for the codes as well.. it really worked out well for me. Cheers
In my 3 year of blogging, this is the first post about duplicate content on a one blog or domain and I am wondering why wordpress doesn’t have such functionality pre-loaded? Anyways I still have many doubts and I am going to investigate it in further.
This is really interesting post. And code which you have given extremely good. The code is really work. Thanks for sharing. You have solve the problem for who are facing it.
Great insights. Nice Post…
yeah duplicate content can effect your SERP , search engine ranking position.
Thanks for this usefull information. I hope to read more in future.
Duplicate content had been a problem to many. One of my friend got slap in google because he run 3 sites all with the same content. I don’t want the same thing to happen to me, so everytime I post, I make sure it would be something unique.
You really seem like you know what you’re talking about. Good stuff, keep it up!
Nice, i’ll be sure to keep this in mind, thanks.
Jeff
Thanks for great post mate..
Nice post! I really like your site design. Just thought I’d drop a line, take it easy!
Hey
Nice work around there. I can use portions of this for my own situation..
Many Thanks to you
I often get problems if you using plugins that use the htaccess file to change the URLs.
Do you think google new ‘rel=canonical” tag is going to solve the duplicate content problems?
Great tip. I appreciate your time. Just wanted to say that I appreciate it.
Great job, keep up the good work.
Very nice information. Thanks for this. I stopped to say that your blog is very well written ..
This is really interesting post. And code which you have given extremely good. The code is really work. Thanks for sharing. You have solve the problem for who are facing it.
Great minds think alike. if you click the link above this bit, you’ll see I’ve done something similar, using things like excerpts or custom fields, on posts, tag pages, categories etc.
Good stuff! I’ll definately be visiting again. Nice work.
This is a really cool blog! I wish I had found it sooner. Keep up the good work.
Nice post! I really like your site design. Just thought I’d drop a line, take it easy!
This is a great wordpress tip. These will definitely help to reduce the duplicate content that wordpress creates. You should also put in a good robots.txt file as well. I believe there is also a good content control plugin that allows you to control your content and how it’s indexed.
Great tip. I appreciate your time. Just wanted to say that I appreciate it.
You will need to really think through this process. If your site has been online for awhile visitors have linked to your content and Google. Yahoo and MSN have used those links to build your page rank.
That was very useful actually. Thank you, Dejan.
Thank you for those tips. It’s really great for me, if there are posts I can easily understand even English isn’t my motherlangugae, ’cause it helps me so often, because I can say in English blogs I find much more advices and tipps than in German ones. So, thanks a lot!
Nice article, I think that many people can type with no fear, if they adjust their wordpresses. Thank you!
Duplicate content can be a real pain with wordpress, I use a couple of plugins that have helped me out, platinum SEO is one of them, the most important part is Canonical URLs this can be solved using the All in One SEO plugin. Hope this helps.
Your header.php modification is just what I’ve been looking for. Thank you! Steve
no need to worry, i think google is punishing them all the while. matters like this are really common in the blogging world.
Thanks for great post mate..
I used to publish my articles, but now I wander should I stop doing this, because the risk of duplicate content penalty. Should I stop publish my articles on article directories?
nice post n cool blog
i am already using first an 3. ones but i dont know how much necessary 2. one. maybe i try.
nice post Ilike it and very best information
nice post thanks fot tips about how to avoid duplicate content on wordpress, very usefull
Hi, I am big fan of music and a big fan of vintage music clothes.Best part people notice me and specially in concert or when I go out with my date.
Nice cool tip thanks for that I found it useful.
i know what your mean. I also have always wondered if the search engines would consider this duplicate content. I wonder if there are Search engine friendly way to have both pages be indexed.
I really hate it when somebody just scrapes out my content and publishes it on their blog. Oh… the nerve. Thanks for the tips, I really need this.
Nice advice and bookmarking it. i just know this happen in wp. thanks
that’s true, that’s why i stick with blogger. Spammers target wordpress due to lack of security.
thank you vey much
Thank you! Steve
Your tips really help to make wordpress more search engine friendly by avoiding duplicate content. Thanks for sharing this seo infos.
very good web site
thansk you
no need to worry, i think google is punishing them all the while. matters like this are really common in the blogging world.
very good article!
post a good article for the user input wordpress
Thank’s
Busana Muslim and Baju Muslim
Wow, I need to implement this for my blog. Thanks!
Hello Webmaster Very nice article! Thanks for this
thnkas i am already using first an 3. ones but i dont know how much necessary 2. one
However,what I understand is that googlebot is instructed to deal with blogs’ duplicate content and to ignore it.I do not know how accurate the algorithm is.
Wow! Thank you for this hint. Wordpress is really good for SEO (Best then Blogger or MovableType) anyway we always will need fix it. I will try it in my blogs.