Getting Out Of The Google Supplemental Index

Thanks for this information John. I haven't had this problem yet but this could come in handy later.

I had a very basic robots.txt file set up and thought I was doing ok - I had nearly 2000 supplemental links when I checked. Hopefully the changes I've made will see a reduction and who knows an increase in traffic.

I found that a lot of mine were from Forum Archives which has been active on my site for years now - quite scary to think what damage this may have been doing to my site in the past!

Here's a recent post by Matt Cutts of Google regarding the supplemental index...

Google Hell?
http://www.mattcutts.com/blog/google-hell/

This is a very interesting post. I found that I have 175 supplemental pages and 198 cached pages. I also found that my RSS feed was being returned by Google for various searches, which is obviously not preferred.

I made similar changes to my robots.txt as a result of this post that I hope will lower the supplemental number significantly. I'm especially interested in seeing how that effects my website reporting as a result.

Great post John. I've already tweaked the robots.txt file and uploaded it onto my blog's server and now eagerly await the results ;)

Oops, you're a pingback!

Translation, Sheawey?

I found this plugin for Wordpress that gets rid of duplicated content:

http://www.davidpitlyuk.com/2007/05/25/how-to-get-...

I believe it just blocks the category and archive pages, so you may still want to do some work on the robots.txt file.

Heck > hell.

Impressive. Big thanks to Nathan for sorting out the details and thanks to John for making me aware ;)

This is significant and I now have some work to do on 2/3 sites.

Yeah, I need to get my site out of supplemental heck. Thanks for the reminder.

Well, to be honest, it should do the trick. I'm just waiting for Google to update.

I found that by creating a robots.txt file with the following information and placing it in my root directory, I was able to get rid of those pesky supplementals.

Copy and paste the following:

User-agent: *
Disallow: /*/feed/
Disallow: /*/feed/rss/
Disallow: /*/trackback/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /tag/

This should do the trick for you, as it's worked pretty well for me so far. (Thanks to Tim Spangler for the hint.)

If you don't have SEO for Firefox you can find out articles that are in the supplemental index by typing this in Google:

site:www.yourdomain.com -inurl:www

Nice post as usual John, Cheers

Again a post that shows how information rich John's Blog is. Thanks for the link no so boring life where I got the full story but also thanks to the other postes her who discussed the issue more controversial. Yes you are right Joshua Watson, we should concentrate more on original quality content, but sometimes it is faster/easier to use free resources like articles, reviews etc.

Charity begins at home when it comes to beer John :grin:

so if i have wordpress in it's own directory

/wordpress

would it look like this then...

Disallow: */wordpress/feed/
Disallow: */wordpress/trackback/

even if i made it so wordpress index.php is in my main directory.

i have like 78,000 pages in the supplemental index. woohoo. :(

even the confussion, it´s a great tool

Supplemental Results and the index they are in have a bad rap however, a webmaster should not concern him/herself with trying to get their pages out of the index. It's just like PR...everyone thinks it's a big deal when in fact it is not. Web pages that are in the supplemental results can and will still rank for your targeted keywords. In fact, there are cases where a supp result will out rank a web page in the normal index.

This was talked about a lot at the search engine strategies conference in New York last month AND you can find more information over at SEOmoz. SEOmoz whiteboard

In my opinion, people should worry less about supp results and more on quality *unique* content. Just my 2 cents. :)

An eye opening article. Kudos. Time to be evil! :evil:

looks like i'll be busy for the next few months. one of my day job websites has over 200,000 pages in supplemental.

i was wondering why our traffic has been going downwards, despite us adding new content nearly every day.

thanks for the tip Mr Chow.

BTW, why all the red on your pages today? I'm seeing red really!

Kenneth

I'm going to try it out as nearly 40% of my pages are now in the Supplementals, that is truly mental!

Wow! Thanks, John for the tip! I'd heard of the supps. but didn't know I was affected!

Kenneth

Oh, never mind the previous comment, http://www.seobook.com/archives/001329.shtml shows it does work, at least for Googlebot and perhaps for Yahoo Slurp too! :)

Unfortunately you can't use the wildcard "*" in Disallow: lines of the robots.txt file. It can be used for user agents but the following won't work:
Disallow: /2006/0*

Thanks for sending me to Nathan's site, John. Looks like I've got my work cut out for me this morning.

Pretty sure your numbers are going even higher, just like it did for Nathan when he reduced his supp pages. Do keep us updated -- we thrive in your success, John LOL. :twisted:

Great article John, (and Nathan). I've read about this perhaps dozens of times before, including a discourse from Matt Cutts, and no one ever explained what the supplemental index really menat or why I cared ... just seemed like funny indetation to me. Got work to do, but thnaks for pointing it out in a way I could understand.

2000 visitors from google? Wow, that's something I'm now going to aspire too :) me like

I think thats a really good idea...I might attempt to add the text into mine *puts it on a very long to do list*

I think my server may have a melt down. This article is about to go front page on digg with just a few more diggs (http://www.notsoboringlife.com/ramblings/top-10-ca...) plus all the traffic from Johns post. Going to be an interesting day ;)

I have been working on this earlier today. I have the SEO tool installed but I never noticed that part of it ;) I did manage to clear some a while back with robots.txt but still need to do some more work.